Re: [Koha] Koha resource requirements
Hi Barry, I think that it is reasonable to expect some minimum requirements to be listed, but it is easier said than done, since Koha is a large multi-component system. Personally, I tend to run many Koha instances on high powered servers with external database servers, so it's rare for me to be thinking about the requirements for just 1 Koha. Typically, I only run 1 Koha instance for development and testing, and I use the koha-testing-docker (https://gitlab.com/koha-community/koha-testing-docker) setup for that. Looking at that, I'm seeing probably around ~1.25GB of RAM used for an idle system, although that includes components like a Z3950 responder and SIP server that you won't need. Since it's a Dockerized environment, that's on top of whatever the OS and other installed software is needing. In my observations, I typically notice that a Starman "Plack" worker will take up about 200-500MB of RAM in its lifetime (mine right now are at 234 and 284 so I'd budget another 500MB). By default, you have 2 configured. If you wanted to shave off more memory, you could downgrade to just 1 Starman "Plack" worker. Looking at a server I do have with only 1 Koha instance... MySQL is using 357MB, Koha's Starman/Plack workers are 246MB and 190MB. The indexing daemon is 171MB. So that adds up to just under 1GB. Add in a bit more for Apache, Memcached, Zebra search engine, et cetera and you're probably up to 1-1.25GB. But that's just a snapshot around lunch time near Christmas holidays. Ultimately, your mileage may vary. If you add some swap space, which I know isn't included by default on AWS, you might have a smoother experience as well as that would help soak up some unexpected memory spikes. I don't have access to update the Downloads page to add some minimum requirements, but I think perhaps that someone should. Regarding CPU, I'd say minimum 4 CPU between all the various different Koha subcomponents for any serious usage, but you could get away with less. My koha-testing-docker uses 2 CPUs and it manages fine for me as a single user. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- Date: Wed, 16 Dec 2020 16:01:03 +0000 From: Barry Callahan <barryc@quicksilverdragon.com> To: koha@lists.katipo.co.nz <koha@lists.katipo.co.nz> Subject: [Koha] Koha resource requirements Message-ID: <010001766c48916c-b5c8c8db-d977-4dd9-adb1-83a6c5f2ae77-000000@email.amazonses.com> Content-Type: text/plain; charset=UTF-8 I recently started playing with Koha to see if it would help my wife and I manage our home library. Before I began, I reviewed the requirements listed on the download page ( https://koha-community.org/download-koha/ ) and the instructions for installing using the official .deb packages. One thing I thought was very conspicuously lacking was any mention of resource requirements. Naively, I assumed that meant that the requirements for running Koha would only be minimally greater than those required for a LAMPP stack serving a similar amount of traffic. Needless to say, an Amazon EC2 micro instance, which only has 1GB of RAM allocated to it is incapable of creating and enabling a single Koha library instance. Attempting to do so with no swap enabled immediately resulted in a crashed system. I resized the instance to small, with 2GB of RAM, and was able to boot the system and log in, but top showed a peak memory usage of ~1.9GB. On seeing that, I resized the server again to a medium instance with 4GB of RAM. After populating the database and adding a couple users and a few dozen items to it, the server is sitting at 1.5GB of RAM in use (not including cache) at idle. Admittedly, that includes the 500MB allocated to the local MySQL server, but still. I think I could probably be safe scaling back to a small instance if I moved the database to a remote server, which is my plan. I'd like to suggest updating the requirements section on the download page to include some information on the resources used by the koha-plack stack. At a minimum, some information about how much koha+plack would add to the memory footprint of an existing LAMP stack at idle would have saved me a fair bit of grief. Some information on how RAM usage and CPU load can be expected to scale with the number of active library instances and concurrent users would be nice, too, but less important than the base numbers, I think.
Thanks for that, David. Your comment about swap is well-taken. If the initial micro instance had had at least a Gig of swap enabled, instead of locking up hard, it would have crawled along, swapping processes in and out of RAM, and given me a chance to stop the Koha instance and recover the system. (For giggles, I tested it.) As a second point of reference, I followed the instructions at https://wiki.koha-community.org/wiki/Debian to install the current stable release (20.05) on a fresh install of Ubuntu 20.04.1. I have not made any changes to config files not otherwise indicated in those instructions. Using the smem utility reveals the following memory usage at idle (using the numbers in the PSS column): * Approximately 850MB used by applications owned by the koha instance's userid. That includes just shy of 250MB for each of the two starman plack workers. * memcache and rabbitmq (which were both installed as dependencies of the koha-common package) combine to use approx 100MB total. * mysqld is using 500MB. I'm sure I can tune it down, but as a base install, straight from the distro's repo, it's a useful datapoint. * apache's only using ~8MB. As much as I think that the Requirements section on the download page really should say something about the memory needs of the system, I agree that it could be a bit difficult to succinctly describe the memory requirements of all of the moving parts in such a way that fits into the page. On the other hand, since the Debian installation walkthrough linked above is written from the assumption that the reader is starting from a virgin system, I think adding a section to the top of that page would be a much easier mark to hit. Perhaps something along the lines of: ------------------------ Some notes on memory usage: These instructions assume you are starting from scratch, on a fresh server, with none of the prerequisites already installed. Koha is big, and it is built on top of a stack of other software that is also big. As a result, the memory requirements to run it are going to be significant. You must ensure that the target machine has adequate free memory available to run Koha before you begin. Following these instructions will result in an instance of Koha and the necessary support software which together consume between 850MB - 1GB of memory at idle. If you are installing the MySQL server on the same machine as Koha, assume it is going to need another 350MB - 500MB of RAM. Note that there are many opportunities to tune the configurations of MySQL and Koha to reduce their memory consumption, but those optimizations and their consequences are outside of the scope of this document. ------------------------ -----Original message----- From: dcook@prosentient.com.au Sent: Wednesday, December 16 2020, 8:41 pm To: Barry Callahan Cc: koha@lists.katipo.co.nz Subject: RE: [Koha] Koha resource requirements Hi Barry, I think that it is reasonable to expect some minimum requirements to be listed, but it is easier said than done, since Koha is a large multi-component system. Personally, I tend to run many Koha instances on high powered servers with external database servers, so it's rare for me to be thinking about the requirements for just 1 Koha. Typically, I only run 1 Koha instance for development and testing, and I use the koha-testing-docker (https://gitlab.com/koha-community/koha-testing-docker) setup for that. Looking at that, I'm seeing probably around ˜1.25GB of RAM used for an idle system, although that includes components like a Z3950 responder and SIP server that you won't need. Since it's a Dockerized environment, that's on top of whatever the OS and other installed software is needing. In my observations, I typically notice that a Starman "Plack" worker will take up about 200-500MB of RAM in its lifetime (mine right now are at 234 and 284 so I'd budget another 500MB). By default, you have 2 configured. If you wanted to shave off more memory, you could downgrade to just 1 Starman "Plack" worker. Looking at a server I do have with only 1 Koha instance... MySQL is using 357MB, Koha's Starman/Plack workers are 246MB and 190MB. The indexing daemon is 171MB. So that adds up to just under 1GB. Add in a bit more for Apache, Memcached, Zebra search engine, et cetera and you're probably up to 1-1.25GB. But that's just a snapshot around lunch time near Christmas holidays. Ultimately, your mileage may vary. If you add some swap space, which I know isn't included by default on AWS, you might have a smoother experience as well as that would help soak up some unexpected memory spikes. I don't have access to update the Downloads page to add some minimum requirements, but I think perhaps that someone should. Regarding CPU, I'd say minimum 4 CPU between all the various different Koha subcomponents for any serious usage, but you could get away with less. My koha-testing-docker uses 2 CPUs and it manages fine for me as a single user. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- Date: Wed, 16 Dec 2020 16:01:03 +0000 From: Barry Callahan <barryc@quicksilverdragon.com> To: koha@lists.katipo.co.nz <koha@lists.katipo.co.nz> Subject: [Koha] Koha resource requirements Message-ID: <010001766c48916c-b5c8c8db-d977-4dd9-adb1-83a6c5f2ae77-000000@email.amazonses.com> Content-Type: text/plain; charset=UTF-8 I recently started playing with Koha to see if it would help my wife and I manage our home library. Before I began, I reviewed the requirements listed on the download page ( https://koha-community.org/download-koha/ ) and the instructions for installing using the official .deb packages. One thing I thought was very conspicuously lacking was any mention of resource requirements. Naively, I assumed that meant that the requirements for running Koha would only be minimally greater than those required for a LAMPP stack serving a similar amount of traffic. Needless to say, an Amazon EC2 micro instance, which only has 1GB of RAM allocated to it is incapable of creating and enabling a single Koha library instance. Attempting to do so with no swap enabled immediately resulted in a crashed system. I resized the instance to small, with 2GB of RAM, and was able to boot the system and log in, but top showed a peak memory usage of ˜1.9GB. On seeing that, I resized the server again to a medium instance with 4GB of RAM. After populating the database and adding a couple users and a few dozen items to it, the server is sitting at 1.5GB of RAM in use (not including cache) at idle. Admittedly, that includes the 500MB allocated to the local MySQL server, but still. I think I could probably be safe scaling back to a small instance if I moved the database to a remote server, which is my plan. I'd like to suggest updating the requirements section on the download page to include some information on the resources used by the koha-plack stack. At a minimum, some information about how much koha+plack would add to the memory footprint of an existing LAMP stack at idle would have saved me a fair bit of grief. Some information on how RAM usage and CPU load can be expected to scale with the number of active library instances and concurrent users would be nice, too, but less important than the base numbers, I think.
Hi Barry, I think that all sounds reasonable. If you haven’t done so already, would you consider requesting a wiki account via https://wiki.koha-community.org/wiki/Special:RequestAccount, so that you can add those notes as you’ve suggested? It should be low effort (in comparison to all the work you’ve done so far!) and would be a valuable contribution to Koha that would likely help many others. Cheers, David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Barry Callahan <barryc@quicksilverdragon.com> Sent: Friday, 18 December 2020 7:37 AM To: dcook@prosentient.com.au Cc: koha@lists.katipo.co.nz Subject: RE: [Koha] Koha resource requirements Thanks for that, David. Your comment about swap is well-taken. If the initial micro instance had had at least a Gig of swap enabled, instead of locking up hard, it would have crawled along, swapping processes in and out of RAM, and given me a chance to stop the Koha instance and recover the system. (For giggles, I tested it.) As a second point of reference, I followed the instructions at https://wiki.koha-community.org/wiki/Debian to install the current stable release (20.05) on a fresh install of Ubuntu 20.04.1. I have not made any changes to config files not otherwise indicated in those instructions. Using the smem utility reveals the following memory usage at idle (using the numbers in the PSS column): * Approximately 850MB used by applications owned by the koha instance's userid. That includes just shy of 250MB for each of the two starman plack workers. * memcache and rabbitmq (which were both installed as dependencies of the koha-common package) combine to use approx 100MB total. * mysqld is using 500MB. I'm sure I can tune it down, but as a base install, straight from the distro's repo, it's a useful datapoint. * apache's only using ~8MB. As much as I think that the Requirements section on the download page really should say something about the memory needs of the system, I agree that it could be a bit difficult to succinctly describe the memory requirements of all of the moving parts in such a way that fits into the page. On the other hand, since the Debian installation walkthrough linked above is written from the assumption that the reader is starting from a virgin system, I think adding a section to the top of that page would be a much easier mark to hit. Perhaps something along the lines of: ------------------------ Some notes on memory usage: These instructions assume you are starting from scratch, on a fresh server, with none of the prerequisites already installed. Koha is big, and it is built on top of a stack of other software that is also big. As a result, the memory requirements to run it are going to be significant. You must ensure that the target machine has adequate free memory available to run Koha before you begin. Following these instructions will result in an instance of Koha and the necessary support software which together consume between 850MB - 1GB of memory at idle. If you are installing the MySQL server on the same machine as Koha, assume it is going to need another 350MB - 500MB of RAM. Note that there are many opportunities to tune the configurations of MySQL and Koha to reduce their memory consumption, but those optimizations and their consequences are outside of the scope of this document. ------------------------ -----Original message----- From: dcook@prosentient.com.au <mailto:dcook@prosentient.com.au> Sent: Wednesday, December 16 2020, 8:41 pm To: Barry Callahan Cc: koha@lists.katipo.co.nz <mailto:koha@lists.katipo.co.nz> Subject: RE: [Koha] Koha resource requirements Hi Barry, I think that it is reasonable to expect some minimum requirements to be listed, but it is easier said than done, since Koha is a large multi-component system. Personally, I tend to run many Koha instances on high powered servers with external database servers, so it's rare for me to be thinking about the requirements for just 1 Koha. Typically, I only run 1 Koha instance for development and testing, and I use the koha-testing-docker (https://gitlab.com/koha-community/koha-testing-docker) setup for that. Looking at that, I'm seeing probably around ˜1.25GB of RAM used for an idle system, although that includes components like a Z3950 responder and SIP server that you won't need. Since it's a Dockerized environment, that's on top of whatever the OS and other installed software is needing. In my observations, I typically notice that a Starman "Plack" worker will take up about 200-500MB of RAM in its lifetime (mine right now are at 234 and 284 so I'd budget another 500MB). By default, you have 2 configured. If you wanted to shave off more memory, you could downgrade to just 1 Starman "Plack" worker. Looking at a server I do have with only 1 Koha instance... MySQL is using 357MB, Koha's Starman/Plack workers are 246MB and 190MB. The indexing daemon is 171MB. So that adds up to just under 1GB. Add in a bit more for Apache, Memcached, Zebra search engine, et cetera and you're probably up to 1-1.25GB. But that's just a snapshot around lunch time near Christmas holidays. Ultimately, your mileage may vary. If you add some swap space, which I know isn't included by default on AWS, you might have a smoother experience as well as that would help soak up some unexpected memory spikes. I don't have access to update the Downloads page to add some minimum requirements, but I think perhaps that someone should. Regarding CPU, I'd say minimum 4 CPU between all the various different Koha subcomponents for any serious usage, but you could get away with less. My koha-testing-docker uses 2 CPUs and it manages fine for me as a single user. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- Date: Wed, 16 Dec 2020 16:01:03 +0000 From: Barry Callahan <barryc@quicksilverdragon.com <mailto:barryc@quicksilverdragon.com> > To: koha@lists.katipo.co.nz <mailto:koha@lists.katipo.co.nz> <koha@lists.katipo.co.nz <mailto:koha@lists.katipo.co.nz> > Subject: [Koha] Koha resource requirements Message-ID: <010001766c48916c-b5c8c8db-d977-4dd9-adb1-83a6c5f2ae77-000000@email.amazonses.com <mailto:010001766c48916c-b5c8c8db-d977-4dd9-adb1-83a6c5f2ae77-000000@email.amazonses.com> > Content-Type: text/plain; charset=UTF-8 I recently started playing with Koha to see if it would help my wife and I manage our home library. Before I began, I reviewed the requirements listed on the download page ( https://koha-community.org/download-koha/ ) and the instructions for installing using the official .deb packages. One thing I thought was very conspicuously lacking was any mention of resource requirements. Naively, I assumed that meant that the requirements for running Koha would only be minimally greater than those required for a LAMPP stack serving a similar amount of traffic. Needless to say, an Amazon EC2 micro instance, which only has 1GB of RAM allocated to it is incapable of creating and enabling a single Koha library instance. Attempting to do so with no swap enabled immediately resulted in a crashed system. I resized the instance to small, with 2GB of RAM, and was able to boot the system and log in, but top showed a peak memory usage of ˜1.9GB. On seeing that, I resized the server again to a medium instance with 4GB of RAM. After populating the database and adding a couple users and a few dozen items to it, the server is sitting at 1.5GB of RAM in use (not including cache) at idle. Admittedly, that includes the 500MB allocated to the local MySQL server, but still. I think I could probably be safe scaling back to a small instance if I moved the database to a remote server, which is my plan. I'd like to suggest updating the requirements section on the download page to include some information on the resources used by the koha-plack stack. At a minimum, some information about how much koha+plack would add to the memory footprint of an existing LAMP stack at idle would have saved me a fair bit of grief. Some information on how RAM usage and CPU load can be expected to scale with the number of active library instances and concurrent users would be nice, too, but less important than the base numbers, I think.
David, It took me a few tries, but I eventually succeeded in getting the wiki to accept my account request. It kept erroring out, complaining about an "Incorrect or missing confirmation code" despite verifying that the answer to the math question was right. Ended up clicking through a few links on the wiki and navigating back to the Request an account page before it finally took. Once I get notification that my account's approved, I'll go ahead and add that text. Barry -----Original message----- From: dcook@prosentient.com.au Sent: Thursday, December 17 2020, 6:24 pm To: Barry Callahan Cc: koha@lists.katipo.co.nz Subject: RE: [Koha] Koha resource requirements Hi Barry, I think that all sounds reasonable. If you haven’t done so already, would you consider requesting a wiki account via https://wiki.koha-community.org/wiki/Special:RequestAccount, so that you can add those notes as you’ve suggested? It should be low effort (in comparison to all the work you’ve done so far!) and would be a valuable contribution to Koha that would likely help many others. Cheers, David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Barry Callahan <barryc@quicksilverdragon.com> Sent: Friday, 18 December 2020 7:37 AM To: dcook@prosentient.com.au Cc: koha@lists.katipo.co.nz Subject: RE: [Koha] Koha resource requirements Thanks for that, David. Your comment about swap is well-taken. If the initial micro instance had had at least a Gig of swap enabled, instead of locking up hard, it would have crawled along, swapping processes in and out of RAM, and given me a chance to stop the Koha instance and recover the system. (For giggles, I tested it.) As a second point of reference, I followed the instructions at https://wiki.koha-community.org/wiki/Debian to install the current stable release (20.05) on a fresh install of Ubuntu 20.04.1. I have not made any changes to config files not otherwise indicated in those instructions. Using the smem utility reveals the following memory usage at idle (using the numbers in the PSS column): * Approximately 850MB used by applications owned by the koha instance's userid. That includes just shy of 250MB for each of the two starman plack workers. * memcache and rabbitmq (which were both installed as dependencies of the koha-common package) combine to use approx 100MB total. * mysqld is using 500MB. I'm sure I can tune it down, but as a base install, straight from the distro's repo, it's a useful datapoint. * apache's only using ~8MB. As much as I think that the Requirements section on the download page really should say something about the memory needs of the system, I agree that it could be a bit difficult to succinctly describe the memory requirements of all of the moving parts in such a way that fits into the page. On the other hand, since the Debian installation walkthrough linked above is written from the assumption that the reader is starting from a virgin system, I think adding a section to the top of that page would be a much easier mark to hit. Perhaps something along the lines of: ------------------------ Some notes on memory usage: These instructions assume you are starting from scratch, on a fresh server, with none of the prerequisites already installed. Koha is big, and it is built on top of a stack of other software that is also big. As a result, the memory requirements to run it are going to be significant. You must ensure that the target machine has adequate free memory available to run Koha before you begin. Following these instructions will result in an instance of Koha and the necessary support software which together consume between 850MB - 1GB of memory at idle. If you are installing the MySQL server on the same machine as Koha, assume it is going to need another 350MB - 500MB of RAM. Note that there are many opportunities to tune the configurations of MySQL and Koha to reduce their memory consumption, but those optimizations and their consequences are outside of the scope of this document. ------------------------ -----Original message----- From: dcook@prosentient.com.au Sent: Wednesday, December 16 2020, 8:41 pm To: Barry Callahan Cc: koha@lists.katipo.co.nz Subject: RE: [Koha] Koha resource requirements Hi Barry, I think that it is reasonable to expect some minimum requirements to be listed, but it is easier said than done, since Koha is a large multi-component system. Personally, I tend to run many Koha instances on high powered servers with external database servers, so it's rare for me to be thinking about the requirements for just 1 Koha. Typically, I only run 1 Koha instance for development and testing, and I use the koha-testing-docker (https://gitlab.com/koha-community/koha-testing-docker) setup for that. Looking at that, I'm seeing probably around ˜1.25GB of RAM used for an idle system, although that includes components like a Z3950 responder and SIP server that you won't need. Since it's a Dockerized environment, that's on top of whatever the OS and other installed software is needing. In my observations, I typically notice that a Starman "Plack" worker will take up about 200-500MB of RAM in its lifetime (mine right now are at 234 and 284 so I'd budget another 500MB). By default, you have 2 configured. If you wanted to shave off more memory, you could downgrade to just 1 Starman "Plack" worker. Looking at a server I do have with only 1 Koha instance... MySQL is using 357MB, Koha's Starman/Plack workers are 246MB and 190MB. The indexing daemon is 171MB. So that adds up to just under 1GB. Add in a bit more for Apache, Memcached, Zebra search engine, et cetera and you're probably up to 1-1.25GB. But that's just a snapshot around lunch time near Christmas holidays. Ultimately, your mileage may vary. If you add some swap space, which I know isn't included by default on AWS, you might have a smoother experience as well as that would help soak up some unexpected memory spikes. I don't have access to update the Downloads page to add some minimum requirements, but I think perhaps that someone should. Regarding CPU, I'd say minimum 4 CPU between all the various different Koha subcomponents for any serious usage, but you could get away with less. My koha-testing-docker uses 2 CPUs and it manages fine for me as a single user. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- Date: Wed, 16 Dec 2020 16:01:03 +0000 From: Barry Callahan <barryc@quicksilverdragon.com> To: koha@lists.katipo.co.nz <koha@lists.katipo.co.nz> Subject: [Koha] Koha resource requirements Message-ID: <010001766c48916c-b5c8c8db-d977-4dd9-adb1-83a6c5f2ae77-000000@email.amazonses.com> Content-Type: text/plain; charset=UTF-8 I recently started playing with Koha to see if it would help my wife and I manage our home library. Before I began, I reviewed the requirements listed on the download page ( https://koha-community.org/download-koha/ ) and the instructions for installing using the official .deb packages. One thing I thought was very conspicuously lacking was any mention of resource requirements. Naively, I assumed that meant that the requirements for running Koha would only be minimally greater than those required for a LAMPP stack serving a similar amount of traffic. Needless to say, an Amazon EC2 micro instance, which only has 1GB of RAM allocated to it is incapable of creating and enabling a single Koha library instance. Attempting to do so with no swap enabled immediately resulted in a crashed system. I resized the instance to small, with 2GB of RAM, and was able to boot the system and log in, but top showed a peak memory usage of ˜1.9GB. On seeing that, I resized the server again to a medium instance with 4GB of RAM. After populating the database and adding a couple users and a few dozen items to it, the server is sitting at 1.5GB of RAM in use (not including cache) at idle. Admittedly, that includes the 500MB allocated to the local MySQL server, but still. I think I could probably be safe scaling back to a small instance if I moved the database to a remote server, which is my plan. I'd like to suggest updating the requirements section on the download page to include some information on the resources used by the koha-plack stack. At a minimum, some information about how much koha+plack would add to the memory footprint of an existing LAMP stack at idle would have saved me a fair bit of grief. Some information on how RAM usage and CPU load can be expected to scale with the number of active library instances and concurrent users would be nice, too, but less important than the base numbers, I think.
participants (2)
-
Barry Callahan -
dcook＠prosentient.com.au