Our koha server regularly grinds to a halt, preventing use of the web gui and even making it near impossible to log into via ssh. One minute load average on the server has spiked into at least the 80s, where it normally hovers around 1. If I'm patient, after about 15 minutes the system becomes usable again. Where should I begin looking for the trouble? The system is running on a VMWare ESX host. It's been given two vCPUs and 6GB of RAM. The host is not starved for resources, so I think I can exclude it from the equation. Other VMs running on the same host are not affected. Koha is version 3.01.00.140 (last beta before 3.2). The server is Ubuntu 9.10. We have ~200,000 items and ~100,000 biblios. I was able to capture a process list during the last event (this AM), and saw that I had 95 instances of apache started. At the moment, with things working smoothly, I have only 14. I've also run mysqltuner.pl in order to locate potential bottlenecks with the db, and the only glaring issues I have are: 1) Lots of joins without indexes (811 since db restart this morning) 2) Temporary tables created on disk: 45% (6K on disk / 15K total) I've set innodb_buffer_pool_size = 1G in my.cnf, but still the system keeps creating these tables on disk, which certainly can't be helping anything. OK, any ideas? Other troubleshooting info I can provide? It's gotten to the point that we're losing a decent amount of time every day to this issue. Thanks so much in advance! -- Chris Hobbs Director, Technology New Haven Unified School District -- This message was scanned by ESVA and is believed to be clean.
Le 14/03/2011 18:09, Chris Hobbs a écrit :
Our koha server regularly grinds to a halt, preventing use of the web gui and even making it near impossible to log into via ssh. One minute load average on the server has spiked into at least the 80s, where it normally hovers around 1.
If I'm patient, after about 15 minutes the system becomes usable again.
Where should I begin looking for the trouble?
The system is running on a VMWare ESX host. It's been given two vCPUs and 6GB of RAM. The host is not starved for resources, so I think I can exclude it from the equation. Other VMs running on the same host are not affected.
Koha is version 3.01.00.140 (last beta before 3.2). The server is Ubuntu 9.10. We have ~200,000 items and ~100,000 biblios.
I was able to capture a process list during the last event (this AM), and saw that I had 95 instances of apache started. At the moment, with things working smoothly, I have only 14.
I've also run mysqltuner.pl in order to locate potential bottlenecks with the db, and the only glaring issues I have are:
1) Lots of joins without indexes (811 since db restart this morning) 2) Temporary tables created on disk: 45% (6K on disk / 15K total) I've set innodb_buffer_pool_size = 1G in my.cnf, but still the system keeps creating these tables on disk, which certainly can't be helping anything.
OK, any ideas? Other troubleshooting info I can provide? It's gotten to the point that we're losing a decent amount of time every day to this issue.
Thanks so much in advance!
-- Chris Hobbs Director, Technology New Haven Unified School District Hi Innodb file management is quite important for performance. We wrote a performance blog post http://www.biblibre.com/en/blog/entry/mysql-default-config-and-very-large-ko... you might find it interesting. Also, you should check that your kernel is at least bigmem enabled. Last but not least, check that you are not sending too many queries at once. We had that problem once. Solution could be to use a firewall to block more than 10 requests of the same page from same ip in a short amount of time (say less than 3 second for instance) There also has been an interesting discussion on performance tweeking on apache on list http://www.mail-archive.com/koha-devel@lists.koha-community.org/msg00074.htm... Hope that helps -- Henri-Damien LAURENT
On 3/14/11 10:23 AM, LAURENT Henri-Damien wrote:
Le 14/03/2011 18:09, Chris Hobbs a écrit :
OK, any ideas? Other troubleshooting info I can provide? It's gotten to the point that we're losing a decent amount of time every day to this issue. Innodb file management is quite important for performance. We wrote a performance blog post http://www.biblibre.com/en/blog/entry/mysql-default-config-and-very-large-ko... you might find it interesting. Also, you should check that your kernel is at least bigmem enabled. Last but not least, check that you are not sending too many queries at once. We had that problem once. Solution could be to use a firewall to block more than 10 requests of the same page from same ip in a short amount of time (say less than 3 second for instance) There also has been an interesting discussion on performance tweeking on apache on list http://www.mail-archive.com/koha-devel@lists.koha-community.org/msg00074.htm... Hope that helps Henri,
Thanks so much for the pointers. I was able to implement all of the DB suggestions, moved the zebra indexes to another disk, and configured mod_perl. Performance seems to have improved, and I'm hoping to make it through the rest of the day without more stalls. -- Chris Hobbs Director, Technology New Haven Unified School District -- This message was scanned by ESVA and is believed to be clean.
On 3/14/11 10:23 AM, LAURENT Henri-Damien wrote:
Hi Innodb file management is quite important for performance. We wrote a performance blog post http://www.biblibre.com/en/blog/entry/mysql-default-config-and-very-large-ko... you might find it interesting. There also has been an interesting discussion on performance tweeking on apache on list http://www.mail-archive.com/koha-devel@lists.koha-community.org/msg00074.htm...
Just wanted to say thanks again to Henri, as well as to Cindy Murdock and Mason James in the linked e-mail exchange. They say a picture is worth a thousand words. Here's the performance graphs of our server over the last 30 hours. The three spikes in the first graph each represent about 15 minutes of the system being completely unusable. After the third spike I took the web server down and got the database fixes implemented, and moved the zebra indexes to another disk. At about 2:30pm Monday I turned on mod_perl and you can see the result of that in the CPU Utilization graph (it's the cliff). Today was a much better day :-) Here's the picture: http://www.clanhobbs.org/moz-screenshot-2.png -- Chris Hobbs Director, Technology New Haven Unified School District -- This message was scanned by ESVA and is believed to be clean.
participants (2)
-
Chris Hobbs -
LAURENT Henri-Damien