[Koha] Mystery system slowdown

[Agnes Rivers-Moore]

Hey Mason - and others
Are there any other "first rules" we may have missed?

Many thanks,
Agnes

On 06/03/2012 6:06 PM, Mason James wrote:
> On 2012-03-7, at 7:05 AM, Agnes Rivers-Moore wrote:
>
>> Hello Mason
>>
>> Thank you for your thoughtful comments on my problems.
>> I will pass them on to Petrus who set up the Xen stuff.
>>
>> As the list was broken for a few days we had time to discover a couple of contributing factors.
>> One was the Gnome GUI - not loading properly at start-up.
>> Another was vino - VNC which is reported in Ubuntu forums as grabbing 100% CPU under some conditions.
>>
>> Running with Gnome turned off in the virtual server where the problem seems to originate has resulted in two peaceful days. It is too soon to be sure but ... seems to be at least a major contributor.
>>
>> Agnes
>
> aaaaah, yep... ;)
>
> the first rule of running production linux servers is *dont* install a gui on it (gnome, kde etc)
> especially on your VMs!
>
> (if you dont have a gui on your server... you also have no reason to vnc to your server)
>
>
> give your sysadmin a slap! ;)
>
>
>
>> On 06/03/2012 2:48 AM, Mason James wrote:
>>> On 2012-02-29, at 1:00 PM, Agnes Rivers-Moore wrote:
>>>
>>>> Dear friends
>>>>
>>>> We have six libraries running Koha 3.00.06.
>>>> The system runs Redhat, with 6 virtual Xen servers.
>>> i also run some Koha systems on xen (debian-squeeze)
>>>
>>>> Problem #1
>>>>  From time to time, at various times of day one or more of the Koha systems will slow down and stop.
>>> install nagios, munin and monit on your systems, to help determine what is causing the slowdown
>>>
>>>> We can watch the CPU graph on the virtual machine  manager - it usually drops to less than 1% use for a few minutes.
>>>> At this point we have librarians staring at unresponsive Koha screens - connecting... connecting...
>>>> The virtual server desktop is unresponsive even at the physical server.
>>> hmmm... that sounds like a general xen problem, not a Koha problem
>>>
>>> is your system running out of resources  (RAM or swap)?
>>>
>>>
>>>> We cannot connect with a VNC client or browse in.
>>>> This is very frustrating when the CPU graph implies that the server is apparently not busy.
>>>>
>>>> Then the CPU use will ramp rapidly up to 12.4% use (which is pretty much full capacity on our virtual server setup).
>>>> It stays there - and although it may have recovered once or twice from this, usually we have to kill the server.
>>>>
>>>> Sometimes we force the VM to shut down, and then restart it, and can see zebra searches happening with Top. However we may still be unable to log in with VNC or connect from local client browsers.
>>>>
>>>> This is really difficult to track down because we have not found associated error messages, and by the time we get to them, the servers seem to be showing low load, but we may be missing some initiating event.
>>>>
>>>> That is problem #1
>>>>
>>>> Here is problem #2
>>>> We have also noticed that opac-search.pl and mysqld is causing heavy loading.
>>>> Within the virtual server we have multiple cpus and the load is shared between them,
>>>> However, often opac-search.pl is around 30%, and mysqld alone is using close to half of the virtual server's CPU capacity.
>>>> The records we have in this single site library do not have large numbers of items attached to any one biblio record.
>>>> There was a discussion in March about something like this a year ago, but I can't see a resolution.
>>>>
>>>> Does anyone have any ideas on either of these issues?
>>>> Hoping  someone from the developer group might have some ideas.
>>>>
>>> are you clearing you cookies and sessions tables periodically? they might be getting really big
>>> there is a script about for that task
>>>
>>> also, run 'SHOW PROCESSLIST' in mysql to discover what mysql is doing when it's thrashing
>>>
>>>
>>> i recommend running a single mysqld process on the dom-0,
>>> rather than a mysqld process on each of your 6 dom-u's
>>>
>>>
>>> cheers, Mason
>> -- 
>>
>> CEO/ Chief Librarian
>> Hanover Public Library.
>> 451 10th Ave,
>> Hanover, ON
>> N4N 2P1
>>
>> Tel: 519-364-1420
>> Email: arm at hanover.ca
>>
>> Please consider the environment before printing this message or attachments.
>>
>> Support your library! Please consider making a donation, joining the Friends of the Library, or remembering us in your will.
>>
>
> cheers, Mason

-- 

CEO/ Chief Librarian
Hanover Public Library.
451 10th Ave,
Hanover, ON
N4N 2P1

Tel: 519-364-1420
Email: arm at hanover.ca

Please consider the environment before printing this message or attachments.

Support your library! Please consider making a donation, joining the Friends of the Library, or remembering us in your will.