[Koha] MySQL database server, dedicated vs virtual

Sat Dec 3 09:53:36 NZDT 2016

Roger,

We also looked into the "search performance" last year. We are not a 
lending library, so cataloguing and OPAC are our primary concerns. Please 
see <http://navalmarinearchive.com/z_koha/> for the detailed tech analysis 
and conclusions.

You mention below that "Koha does not benefit a lot from multiple CPU cores 
since each CGI request is typically processed by one CPU except for the 
Zebra searches and database queries which run as separate processes."

I talked to Intel about CPU core cross-threading as it was a very obvious, 
high-load, show-stopper with Zebra. The Zebra author never replied to my 
queries. Intel were not optimistic, as kernel (and software) were not their 
bailliwick. I do not know if Plack or Elastic have worked around this.

These are hardware (perhaps software usage of core capability), not Koha, 
restrictions. Tweaking memcached can have appreciable benefits. But the 
bottom line remains that if a single CPU core hits 101-104%, search 
functions descend into the "swimming upstream in treacle" world.

We made the very conscious (but disappointing) decision to stay with Koha 
3.8.24 based on our test results. I do spend quite some time testing later 
versions at "sandbox" level, but have not been able to reproduce "older" 
search speed.

YMMV -- Paul
Production: Koha 3.8.24 on 14.04.2 LTS (GNU/Linux 3.13.0-43-generic 
x86_64), 8-core I7 processors, 64 Gigs RAM, all SSD drives.

At 06:47 PM 12/2/2016 +0100, Roger Grossmann wrote:

>Hi Tobias,
>
>a few month ago we did comparisons measuring the Koha-OPAC-Search 
>performance with plack and memcached enabled using version 3.22.
>Our tests were not scientifically organised. We wanted to gather 
>experience running Koha in different environments. We used the same MySQL 
>database and Koha version in all environments. We did two types of OPAC 
>searches on a limited collection of about 50.000 titles: an 'a*'-search of 
>the complete word index and some special searches with fixed title search 
>terms.
>We tested the following environments:
>
>1) Full Koha installation on a cloud provider hosted VM (the performance 
>of VMs of the specific hoster were high ranked in published comparisons): 
>Debian 8 on a cloud hosted system with virtual 2CPUs, 8 GB RAM, 256 SSD
>2) Full Koha installation on a physical server: Debian 8 on a well 
>equipped physical machine with 64 GB RAM, 4 processors, 250 GB SSD
>3) Full Koha installation on a kvm-VM on a physical server: Debian 8 on a 
>well equipped machine with 64 GB RAM, 4 core CPUs, kvm VM with Debian 8
>4) Koha running on a physical machine like 2) and 3) with a separate 
>physical DB Server. Koha and DB server were connected through a 1 GB network.
>
>Our findings were the following:
>No wonder, environment 2 was the fastest. Running Koha on a physical 
>server with the database on the same machine brings optimal performance.
>As a surprise, environment 3 had a performance closed to environment 2 
>with a difference of max. 5% performance loss. The kvm virtualiser seems 
>to add very little overhead. Advantages of environment 3 are beside the 
>high performance that it is possible to copy and save the VM doing updates 
>and so on. We did not test running multiple kvm-VMs on the same host. 
>Rather than running instances in separate VMs, Koha's concept of managing 
>instances makes the use of one host very efficient. Running multiple 
>instances in one machine requires less maintenance than running multiple 
>VMs each with one Koha instance.
>Environment 1 was about 60-80% slower than environment 2. We got varying 
>results probably depending on the load of the provider based environment.
>Server requests in environment 4 took double the time of environment 1. It 
>seemed that the network latency and transmission of data between DB Server 
>and Koha is a bottleneck here.
>
>We can recommend option 3. It brings an optimum of performance with the 
>flexibility of a virtualised environment. For a single Koha instance you 
>probably do not need more than 2 CPUs cores. Typically you can improve the 
>database performance with an optimised database configuration 
>(specifically the page buffer if you have enough RAM). So using >=16 GB 
>RAM should be well equipped from my perspective.
>Key parameters to speed up Koha were in our tests the hard disk speed (use 
>SSDs) and CPU speed. Koha does not benefit a lot from multiple CPU cores 
>since each CGI request is typically processed by one CPU except for the 
>Zebra searches and database queries which run as separate processes. 
>Additional CPU cores can be useful if the load is high on the server when 
>processing many parallel requests.
>
>Doing searches, the Zebra indexer impacts the speed very much. In our 
>tests Zebra took about 50% or more of the request time. I assume that the 
>planned change to Elasticsearch will result in performance improvements in 
>the future.
>
>Best regards,
>Roger
>
>--
>LMSCloud GmbH
>Roger GroÃmann - GeschÃ¤ftsfÃ¼hrer
>Konrad-Zuse-Platz 8 - D-81829 MÃ¼nchen
>e roger.grossmann at lmscloud.de
>w www.lmscloud.de
>
> > Am 01.12.2016 um 18:03 schrieb tobias carlsson 
> <tobias.carlsson at bitlabbet.se>:
> >
> >
> >
> > Hi!
> >
> > I'm currently working with a Koha implementation in a public library in
> > southern Sweden. Our installation (it's not live yet) is currently on a
> > virtual machine running Ubuntu Server 14.04 LTS and the version of Koha
> > is 3.22. The MySQL database also resides on the same virtual server.
> >
> > I've tried some at home - on a Koha test server running under Debian
> > with the MySQL database on a machine running Ubuntu Server 14.04 (with a
> > Core i3 4170 @3,7 GHz (stock), with a quite slow Western Digital Red.
> > The difference between running on "bare metal" and virtual seem rather
> > striking. Is there anyone out there who have moved from virtual running
> > databases to physical? What can you tell us about this?
> >
> > I know that a database server should run with good I/O performance and a
> > fast CPU and lots of RAM and so on, but since I'm a librarian, not an
> > engineer (that sounded familiar...), I have limited experience of
> > dealing with servers in a large scale and I'm no expert on databases.
> >
> > I'm aware of database optimizations and the use of plack and memcached,
> > but running the database on a "bare metal" server with a high clock,
> > made me start thinking of how we should deal with the database. It seems
> > to me that many libraries use virtual servers for everything, even
> > databases and that it's common to run Koha and server on the same
> > (virtual) instance.
> >
> > Please share your experiences and thoughts about this.
> >
> > Best regards,
> >
> > Tobias Carlsson
> > Systems Librarian and IT Technician
> >
> > Currently working with Koha at Vaggeryds bibliotek (Vaggeryd public
> > library) in Sweden.
> > Working as a Systems Librarian at Gislaveds bibliotek.
> >
> >
> > _______________________________________________
> > Koha mailing list  http://koha-community.org
> > Koha at lists.katipo.co.nz
> > https://lists.katipo.co.nz/mailman/listinfo/koha
>
>_______________________________________________
>Koha mailing list  http://koha-community.org
>Koha at lists.katipo.co.nz
>https://lists.katipo.co.nz/mailman/listinfo/koha

---
Maritime heritage and history, preservation and conservation,
research and education through the written word and the arts.
<http://NavalMarineArchive.com> and <http://UltraMarine.ca>