[Koha] Issues with persistent TCP connections in Microsoft Azure cloud

[David Cook]

Hi all,

 

We recently migrated a busy Koha instance from an on-premises server to an
Azure cloud server. During that move, we also took the opportunity to switch
to using Elasticsearch (not located on the same virtual machine). 

 

Over time, we noticed that we were getting issues with SIP2 connections and
with Koha's Elasticsearch indexer worker timing out. Especially during the
least busy parts of the day. 

 

After some investigation, we learned that Azure allows TCP connections to
only be idle for a maximum of 4 minutes
(https://learn.microsoft.com/en-us/azure/firewall/long-running-sessions). 

 

One solution is to implement application-level keepalives, but that wasn't
practical with Koha's Elasticsearch client library or the SIP2 server.
Instead, I opted to utilize Linux's TCP keepalive implementation, which
operates at a lower level of the technology stack.

 

Fortunately, Koha's Elasticsearch client library is already compatible with
TCP keepalives, so I just needed to tweak the "net.ipv4.tcp_keepalive_time"
kernel parameter via "sysctl". I dropped the keepalive to under the 4 minute
threshold (from the default of 2 hours), and now no more timeouts with
Koha's Elasticsearch indexer worker!

 

Unfortunately, Koha's SIP2 server doesn't support the TCP keepalives out of
the box. However, I've written a patch at "Bug 37087 - Add support for TCP
keepalive to SIP server" which adds granular TCP keepalive support which can
actually be configured via the SIPconfig.xml configuration file (without
needing to change any kernel parameters at all). 

 

I've written up a bit about it on the wiki:

https://wiki.koha-community.org/wiki/Koha_SIP2_server_setup#Microsoft_Azure_
cloud_platform

https://wiki.koha-community.org/wiki/Elasticsearch#Microsoft_Azure_cloud_pla
tform

 

I hope that this information is useful to other people out in the world
implementing Koha. It was a very annoying problem with a surprisingly simple
solution in the end. 

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595