[Koha] F5 Attacks

Jason Sherman jsn.sherman at gmail.com
Fri Oct 28 04:09:15 NZDT 2016 

Just FYI, we run Koha behind the open source version of nginx, and it
has lots of flexibility for rate-limiting by client ip and doing
pretty sophisticated cache handling.

On Thu, Oct 27, 2016 at 7:50 AM, Pedro Amorim <pjamorim91 at gmail.com> wrote:
> Hello all,
>
> As I said before, the Koha I'm currently working on is behind a proxy
> server, namely HAProxy (http://www.haproxy.org/).
> My solution for this was adding the following lines in the frontend of my
> haproxy.cfg:
>
>     # Table definition
>     stick-table type ip size 100k expire 30s store conn_cur
>
>     # Shut the new connection as long as the client has already 10 opened
>     tcp-request connection reject if { src_conn_cur ge 10 }
>     tcp-request connection track-sc1 src
>
> I won't get into too many technical details, but what this does is check if
> the current connections is higher than the normal behaviour. More on this
> can be read at
> http://www.loadbalancer.org/uk/blog/simple-denial-of-service-dos-attack-mitigation-using-haproxy-2
>
> For me, this is working great.
>
> After some testing with several requests and stressing the server at the
> same time as keeping F5 pressed in the opac-search, only a couple processes
> reach the Koha apache server before HAproxy blocks any subsequent requests.
> And this rejection of requests is up for as long as F5 is pressed. As soon
> as F5 is released, and normal navigation behaviour ensues, the application
> works as normal and the server is healthy.
>
> I know this is solved at the HAProxy and not in Koha, so the problem
> persists in Koha (and other applications as well).
>
> Me, personally, I agree that this is a Koha problem *and* I also agree that
> this is a sys admin responsability. With that said, the Koha community and
> the application itself should be aware of this, and by aware I mean at
> least some piece of documentation in the Koha performance tuning or server
> configuration pages should mention this, even if a prevention
> implementation never comes or isn't in the scope of the application.
>
> Let me know your thoughts on the solution I implemented and if you have any
> ideas of how this particular solution could be improved, or any other
> alternative solution you might have found more adaptable for your
> environment.
>
> Cheers,
>
> Pedro Amorim
>
>
> 2016-10-26 20:32 GMT+00:00 clint.deckard <clint.deckard at frontiers.co.nz>:
>
>> Thank you to everyone who replied, I appreciate you taking the time.
>>
>> I understand that this is not a Koha specific problem, I understand the
>> solution is a sys admin responsibility and I understand that there is
>> probably not a 'one size fits all' solution given the many different system
>> configurations. I also accept my limitations when it comes to system
>> administration, though I am always learning, and I have engaged
>> professional help to assist with this issue.
>>
>> I just thought that since this would be a relatively common issue faced by
>> the Koha community and I suspect many Koha instances are run by folk
>> without in-depth sysadmin knowledge it might be helpful to have some advice
>> on possible ways to deal with the issue.
>>
>> I appreciate that the Koha community give their time and knowledge
>> generously and I really value that. I don't want to take advantage of the
>> 'free advice' and I am most appreciative of it when it is offered.
>>
>> I will be happy to share my experience when I do get a solution.
>>
>> Again, thank you for your time and assistance,
>> Clint.
>>
>>
>> Harry at DHD wrote:
>>
>>>
>>> Even though "sharing" is not obligatory, I had the impression that this
>>> is a somewhat "sharing community".
>>> If someone is willing he/she could share his solution-experience-howto or
>>> what ever.
>>>
>>> Probably this is the wrong place to ask for this kind of help ...
>>>
>>> Look into places like SHOREWALL firewall  ... etc. But don't get me wrong
>>> Firewalls are not easy to master!
>>>
>>> Regards!
>>>
>>>
>>> On 26/10/2016 08:07 μμ, Huck wrote:
>>>
>>>> Koha isn't offered 'out of the box'... there is extensive setup required
>>>> and the most basic portion(a webserver) is a requirement, and that would be
>>>> the responsibility of the server's administrator, not a function of a Koha
>>>> developer.
>>>>
>>>>
>>>> On 10/26/2016 5:52 AM, Philippe Blouin wrote:
>>>>
>>>>> I disagree.  If Koha is offered out of the box, and we take time to fix
>>>>> security issues, then it's normal for users to expect "basic" attacks to be
>>>>> taken care of.
>>>>>
>>>>> More so, blocking IP is not a possibility if genuine users are involved
>>>>> using a station from within the library.
>>>>>
>>>>> I'm not saying you're wrong that it's mostly sysadmin work and not
>>>>> Koha, but it doesn't mean nothing can be done.  From the apache's threads,
>>>>> I found nothing useful (mostly derisive comments).  But we could at least
>>>>> talk about it.
>>>>>
>>>>> What about having a javascript preventing refresh on the page withing 5
>>>>> sec of each other?  Needs to be done in a way that the refresh doesn't
>>>>> restart the timer.
>>>>>
>>>>> What about having the OPAC search be code where the refresh will
>>>>> basically send nothing ?  The checkbox are filled, the request is sent to
>>>>> the backend, but the frontend keeps nothing...  I'm just smoking here, but
>>>>> I'm trying to induce some brainstorming in this interesting topic.
>>>>>
>>>>> Philippe Blouin,
>>>>> Responsable du développement informatique
>>>>>
>>>>> Tél.  : (888) 604-2627
>>>>> philippe.blouin at inLibro.com <mailto:philippe.blouin at inLibro.com>
>>>>>
>>>>> inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com>
>>>>> On 10/26/2016 07:13 AM, Jonathan Druart wrote:
>>>>>
>>>>>> Hi,
>>>>>> I don't think this can/must be fixed on Koha side.
>>>>>> It's a sysadmin duty to take care of that.
>>>>>> I would take a look at fail2ban to parse the web server access logs.
>>>>>> But
>>>>>> make sure not to block your X librarians using the same ip ;)
>>>>>>
>>>>>> On Wed, 26 Oct 2016 at 12:28 Pedro Amorim <pjamorim91 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> I have tested this and the stress caused on the server is very severe.
>>>>>>> It
>>>>>>> seems that for every request, a new zebra process is created and the
>>>>>>> server
>>>>>>> will only respond when the last one is finished. This ofc will result
>>>>>>> in
>>>>>>> time outs and eventually a crash in the server.
>>>>>>>
>>>>>>> This is a major critical issue IMO, anyone who knows about this has
>>>>>>> the
>>>>>>> power to deny the service of any Koha online without using any
>>>>>>> additional
>>>>>>> hacking/attacking software.
>>>>>>>
>>>>>>> The Koha I'm working on right now - still in development - is accessed
>>>>>>> behind a proxy server, and I will attempt to solve the problem through
>>>>>>> that, by limiting the requests from the same origin with very little
>>>>>>> time
>>>>>>> between them. Still, even if I'm successful with this, the problem
>>>>>>> will
>>>>>>> still lie in Koha.
>>>>>>>
>>>>>>> Anyone with some sort of insight is very welcome.
>>>>>>>
>>>>>>> Pedro Amorim
>>>>>>>
>>>>>>> 2016-10-26 8:24 GMT+00:00 clint.deckard <
>>>>>>> clint.deckard at frontiers.co.nz>:
>>>>>>>
>>>>>>> I have had this issue appear today. I have attempted to set up
>>>>>>>>
>>>>>>> mod_evasive
>>>>>>>
>>>>>>>> for apache but it doesn't seem to have solved the problem.
>>>>>>>> I would really appreciate some advice.
>>>>>>>> Clint.
>>>>>>>>
>>>>>>>>
>>>>>>>> rfblanchard wrote:
>>>>>>>>
>>>>>>>> Assume a basic opac search:
>>>>>>>>> http://..../cgi-bin/koha/opac-search.pl?q=dog&branch_group_l
>>>>>>>>> imit=branch%3A349
>>>>>>>>>
>>>>>>>>> This would take about 10 seconds to return the first time.
>>>>>>>>>
>>>>>>>>> Assume the user refreshes the results using f5 and keep there finger
>>>>>>>>> there a
>>>>>>>>> moment to long (3s):
>>>>>>>>> This would kill my server for about 1 minute.
>>>>>>>>>
>>>>>>>>> Any attacker could easily make the server unresponsive indefinitely
>>>>>>>>> by
>>>>>>>>> simply holding f5 on an opac search.
>>>>>>>>>
>>>>>>>>> Any recommendations on how to deal with this problem?
>>>>>>>>>
>>>>>>>>> here is a sample from top:
>>>>>>>>>
>>>>>>>>> Tasks: 313 total,   3 running, 309 sleeping,   0 stopped,   1 zombie
>>>>>>>>> %Cpu(s): 93.7 us,  5.2 sy,  0.0 ni,  1.0 id,  0.2 wa, 0.0 hi,  0.0
>>>>>>>>> si,
>>>>>>>>> 0.0
>>>>>>>>> st
>>>>>>>>> KiB Mem:  16465036 total,  1532492 used, 14932544 free, 63180
>>>>>>>>> buffers
>>>>>>>>> KiB Swap:  8526844 total,        0 used,  8526844 free. 505124
>>>>>>>>> cached
>>>>>>>>> Mem
>>>>>>>>>
>>>>>>>>>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+
>>>>>>>>> COMMAND
>>>>>>>>>   7027 peischo+  20   0  416164 162924  12756 S  58.8 1.0   0:26.43
>>>>>>>>> /usr/share/koha
>>>>>>>>>   7009 peischo+  20   0  416800 163524  12756 S  56.5 1.0   0:33.77
>>>>>>>>> /usr/share/koha
>>>>>>>>>   7444 peischo+  20   0  129832  15216   5900 R  37.2 0.1   0:01.12
>>>>>>>>> zebrasrv
>>>>>>>>>   7445 peischo+  20   0  129832  15216   5900 R  35.6 0.1   0:01.07
>>>>>>>>> zebrasrv
>>>>>>>>>   1151 mysql     20   0  886564 181096  10808 S   8.6 1.1   1:27.57
>>>>>>>>>
>>>>>>>> mysqld
>>>>>>>
>>>>>>>>   7435 koha      20   0   25892 3272   2528 R   0.3  0.0   0:00.03
>>>>>>>>> top
>>>>>>>>>      1 root      20   0  176144   5044   3096 S   0.0 0.0   0:01.43
>>>>>>>>> systemd
>>>>>>>>>      2 root      20   0       0      0      0 S   0.0 0.0   0:00.00
>>>>>>>>> kthreadd
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> View this message in context: http://koha.1045719.n5.nabble.
>>>>>>>>> com/F5-Attacks-tp5906098.html
>>>>>>>>> Sent from the Koha-general mailing list archive at Nabble.com.
>>>>>>>>> _______________________________________________
>>>>>>>>> Koha mailing list  http://koha-community.org
>>>>>>>>> Koha at lists.katipo.co.nz
>>>>>>>>> https://lists.katipo.co.nz/mailman/listinfo/koha
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>> Koha mailing list  http://koha-community.org
>>>>>>>> Koha at lists.katipo.co.nz
>>>>>>>> https://lists.katipo.co.nz/mailman/listinfo/koha
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>> Koha mailing list  http://koha-community.org
>>>>>>> Koha at lists.katipo.co.nz
>>>>>>> https://lists.katipo.co.nz/mailman/listinfo/koha
>>>>>>>
>>>>>>> _______________________________________________
>>>>>> Koha mailing list  http://koha-community.org
>>>>>> Koha at lists.katipo.co.nz
>>>>>> https://lists.katipo.co.nz/mailman/listinfo/koha
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Koha mailing list  http://koha-community.org
>>>>> Koha at lists.katipo.co.nz
>>>>> https://lists.katipo.co.nz/mailman/listinfo/koha
>>>>>
>>>>
>>>> _______________________________________________
>>>> Koha mailing list  http://koha-community.org
>>>> Koha at lists.katipo.co.nz
>>>> https://lists.katipo.co.nz/mailman/listinfo/koha
>>>>
>>>
>>> _______________________________________________
>>> Koha mailing list  http://koha-community.org
>>> Koha at lists.katipo.co.nz
>>> https://lists.katipo.co.nz/mailman/listinfo/koha
>>>
>> _______________________________________________
>> Koha mailing list  http://koha-community.org
>> Koha at lists.katipo.co.nz
>> https://lists.katipo.co.nz/mailman/listinfo/koha
>>
> _______________________________________________
> Koha mailing list  http://koha-community.org
> Koha at lists.katipo.co.nz
> https://lists.katipo.co.nz/mailman/listinfo/koha



-- 
Jason

More information about the Koha mailing list