Just FYI, we run Koha behind the open source version of nginx, and it has lots of flexibility for rate-limiting by client ip and doing pretty sophisticated cache handling. On Thu, Oct 27, 2016 at 7:50 AM, Pedro Amorim <pjamorim91@gmail.com> wrote:
Hello all,
As I said before, the Koha I'm currently working on is behind a proxy server, namely HAProxy (http://www.haproxy.org/). My solution for this was adding the following lines in the frontend of my haproxy.cfg:
# Table definition stick-table type ip size 100k expire 30s store conn_cur
# Shut the new connection as long as the client has already 10 opened tcp-request connection reject if { src_conn_cur ge 10 } tcp-request connection track-sc1 src
I won't get into too many technical details, but what this does is check if the current connections is higher than the normal behaviour. More on this can be read at http://www.loadbalancer.org/uk/blog/simple-denial-of-service-dos-attack-miti...
For me, this is working great.
After some testing with several requests and stressing the server at the same time as keeping F5 pressed in the opac-search, only a couple processes reach the Koha apache server before HAproxy blocks any subsequent requests. And this rejection of requests is up for as long as F5 is pressed. As soon as F5 is released, and normal navigation behaviour ensues, the application works as normal and the server is healthy.
I know this is solved at the HAProxy and not in Koha, so the problem persists in Koha (and other applications as well).
Me, personally, I agree that this is a Koha problem *and* I also agree that this is a sys admin responsability. With that said, the Koha community and the application itself should be aware of this, and by aware I mean at least some piece of documentation in the Koha performance tuning or server configuration pages should mention this, even if a prevention implementation never comes or isn't in the scope of the application.
Let me know your thoughts on the solution I implemented and if you have any ideas of how this particular solution could be improved, or any other alternative solution you might have found more adaptable for your environment.
Cheers,
Pedro Amorim
2016-10-26 20:32 GMT+00:00 clint.deckard <clint.deckard@frontiers.co.nz>:
Thank you to everyone who replied, I appreciate you taking the time.
I understand that this is not a Koha specific problem, I understand the solution is a sys admin responsibility and I understand that there is probably not a 'one size fits all' solution given the many different system configurations. I also accept my limitations when it comes to system administration, though I am always learning, and I have engaged professional help to assist with this issue.
I just thought that since this would be a relatively common issue faced by the Koha community and I suspect many Koha instances are run by folk without in-depth sysadmin knowledge it might be helpful to have some advice on possible ways to deal with the issue.
I appreciate that the Koha community give their time and knowledge generously and I really value that. I don't want to take advantage of the 'free advice' and I am most appreciative of it when it is offered.
I will be happy to share my experience when I do get a solution.
Again, thank you for your time and assistance, Clint.
Harry@DHD wrote:
Even though "sharing" is not obligatory, I had the impression that this is a somewhat "sharing community". If someone is willing he/she could share his solution-experience-howto or what ever.
Probably this is the wrong place to ask for this kind of help ...
Look into places like SHOREWALL firewall ... etc. But don't get me wrong Firewalls are not easy to master!
Regards!
On 26/10/2016 08:07 μμ, Huck wrote:
Koha isn't offered 'out of the box'... there is extensive setup required and the most basic portion(a webserver) is a requirement, and that would be the responsibility of the server's administrator, not a function of a Koha developer.
On 10/26/2016 5:52 AM, Philippe Blouin wrote:
I disagree. If Koha is offered out of the box, and we take time to fix security issues, then it's normal for users to expect "basic" attacks to be taken care of.
More so, blocking IP is not a possibility if genuine users are involved using a station from within the library.
I'm not saying you're wrong that it's mostly sysadmin work and not Koha, but it doesn't mean nothing can be done. From the apache's threads, I found nothing useful (mostly derisive comments). But we could at least talk about it.
What about having a javascript preventing refresh on the page withing 5 sec of each other? Needs to be done in a way that the refresh doesn't restart the timer.
What about having the OPAC search be code where the refresh will basically send nothing ? The checkbox are filled, the request is sent to the backend, but the frontend keeps nothing... I'm just smoking here, but I'm trying to induce some brainstorming in this interesting topic.
Philippe Blouin, Responsable du développement informatique
Tél. : (888) 604-2627 philippe.blouin@inLibro.com <mailto:philippe.blouin@inLibro.com>
inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com> On 10/26/2016 07:13 AM, Jonathan Druart wrote:
Hi, I don't think this can/must be fixed on Koha side. It's a sysadmin duty to take care of that. I would take a look at fail2ban to parse the web server access logs. But make sure not to block your X librarians using the same ip ;)
On Wed, 26 Oct 2016 at 12:28 Pedro Amorim <pjamorim91@gmail.com> wrote:
I have tested this and the stress caused on the server is very severe. > It > seems that for every request, a new zebra process is created and the > server > will only respond when the last one is finished. This ofc will result > in > time outs and eventually a crash in the server. > > This is a major critical issue IMO, anyone who knows about this has > the > power to deny the service of any Koha online without using any > additional > hacking/attacking software. > > The Koha I'm working on right now - still in development - is accessed > behind a proxy server, and I will attempt to solve the problem through > that, by limiting the requests from the same origin with very little > time > between them. Still, even if I'm successful with this, the problem > will > still lie in Koha. > > Anyone with some sort of insight is very welcome. > > Pedro Amorim > > 2016-10-26 8:24 GMT+00:00 clint.deckard < > clint.deckard@frontiers.co.nz>: > > I have had this issue appear today. I have attempted to set up >> > mod_evasive > >> for apache but it doesn't seem to have solved the problem. >> I would really appreciate some advice. >> Clint. >> >> >> rfblanchard wrote: >> >> Assume a basic opac search: >>> http://..../cgi-bin/koha/opac-search.pl?q=dog&branch_group_l >>> imit=branch%3A349 >>> >>> This would take about 10 seconds to return the first time. >>> >>> Assume the user refreshes the results using f5 and keep there finger >>> there a >>> moment to long (3s): >>> This would kill my server for about 1 minute. >>> >>> Any attacker could easily make the server unresponsive indefinitely >>> by >>> simply holding f5 on an opac search. >>> >>> Any recommendations on how to deal with this problem? >>> >>> here is a sample from top: >>> >>> Tasks: 313 total, 3 running, 309 sleeping, 0 stopped, 1 zombie >>> %Cpu(s): 93.7 us, 5.2 sy, 0.0 ni, 1.0 id, 0.2 wa, 0.0 hi, 0.0 >>> si, >>> 0.0 >>> st >>> KiB Mem: 16465036 total, 1532492 used, 14932544 free, 63180 >>> buffers >>> KiB Swap: 8526844 total, 0 used, 8526844 free. 505124 >>> cached >>> Mem >>> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>> COMMAND >>> 7027 peischo+ 20 0 416164 162924 12756 S 58.8 1.0 0:26.43 >>> /usr/share/koha >>> 7009 peischo+ 20 0 416800 163524 12756 S 56.5 1.0 0:33.77 >>> /usr/share/koha >>> 7444 peischo+ 20 0 129832 15216 5900 R 37.2 0.1 0:01.12 >>> zebrasrv >>> 7445 peischo+ 20 0 129832 15216 5900 R 35.6 0.1 0:01.07 >>> zebrasrv >>> 1151 mysql 20 0 886564 181096 10808 S 8.6 1.1 1:27.57 >>> >> mysqld > >> 7435 koha 20 0 25892 3272 2528 R 0.3 0.0 0:00.03 >>> top >>> 1 root 20 0 176144 5044 3096 S 0.0 0.0 0:01.43 >>> systemd >>> 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 >>> kthreadd >>> >>> >>> >>> -- >>> View this message in context: http://koha.1045719.n5.nabble. >>> com/F5-Attacks-tp5906098.html >>> Sent from the Koha-general mailing list archive at Nabble.com. >>> _______________________________________________ >>> Koha mailing list http://koha-community.org >>> Koha@lists.katipo.co.nz >>> https://lists.katipo.co.nz/mailman/listinfo/koha >>> >>> _______________________________________________ >> Koha mailing list http://koha-community.org >> Koha@lists.katipo.co.nz >> https://lists.katipo.co.nz/mailman/listinfo/koha >> >> _______________________________________________ > Koha mailing list http://koha-community.org > Koha@lists.katipo.co.nz > https://lists.katipo.co.nz/mailman/listinfo/koha > > _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
-- Jason