Major, multiple problems with our Koha 17.05 system.
VERY long sorry, but our Koha system is virtually unusable and we need help! We are experiencing dozens of outages daily (both OPAC and intranet, more often only OPAC) with our Koha 17.05.01.000 (Debian 3.2.89-2, perl 5.020002, mysql 14.14, apache 2.4.10). Searches can take 15+ seconds. While university staff provide first tier support, we don't have server access, and the second tier and hosting is via an external company. Of course we are working with that company to have them identify the cause, but I have decided to ask the community of any user on 17.05 has the problems we have. Our system is quite extensively customised, with major work done to the OPAC at https://libsearch.arts.ac.uk<https://libsearch.arts.ac.uk/>. We have some auto-renewal functionality we funded working, while we work on more improvements (we are NOT using the template toolkit, relying on workarounds to offer reasonably useful auto-renewal features). Our problems started in the summer vacation soon after the upgrade, where Koha returns functionality became unstable. It was taking 8 seconds to return one item. Our support company advised that it was the AddReturn (and, in particular, MarkIssueReturned) which was the pinch point. Seems some code to resolve the auto_increment bug works slower in 17.05 and our support company was correcting that with a view (I understand) to adding it to Master. Things improved a little, but with the start of term almost 2 weeks ago we are encountering so many problems... "We were alerted to issues at 4.22pm. and found that the plack server was no longer responding. We restarted plack and then apache and the service restored. We are trying to get to the bottom of what caused this issue. It appears to be completely separate from the high CPU problem that we are also seeing." I understand more resources have been allocated to mySQL and additional plack helpers have been allocated (whatever that means). We are heavy users of SIP (80% of transactions are via self issue). But there are so many problems with the returns unit that they are almost unusable. Front line staff are fed up! The sorters and the kiosks configured for returns (as well as issue) fail repeatedly dozens of times daily (ERR_SIP_COM_RECV). Reconnection does happen- takes about 40seconds or so, leading to queues. And the items have NOT come off the account and flow into the exceptions bin. At the same time the intranet report 500 server errors. I am pretty sure most of our problems are related to returns code, but could well be wrong/ oversimplifying things. Our support company advised: "We have added extra logging and can see that some checkin requests via the SIP2 protocol are causing the sipserver process to abort. The book is returned within Koha but as the process returns no information to the SIP2 client therefore the unit responds with an error. After waiting for the timeout period it then will reconnect to the server. We are adding further debug to the logs to work out why the sipserver process is aborting. It does seem to be a special subset of the checkin requests. I'll update you when we have more news." And more problems relating to total outages (with proxy errors 502 relating to invalid responses from an upstream server): "There appears to be a number of issues you are experiencing. Friday you had high CPU usage which is different to today's issues which we think are down to the apache webserver not restarting correctly. I will double check first thing tomorrow that the overnight apache restart has occurred properly. We are hoping the database connections change may address the high CPU issue. We will monitor closely for the errors we have seen in the apache error logs tomorrow." We are a university with 20,000 students across 6 colleges, and right now inductions and training in OPAC use is going on with new students. These continual problems are deeply embarrassing, and faith in the Koha system is dropping rapidly. Has any other Koha 17.05 user (especially one heavily reliant on SIP) experienced anything like this? Has anyone out there any suggestions? Ray Delahunty University of the Arts London This email and any attachments are intended solely for the addressee and may contain confidential information. If you are not the intended recipient of this email and/or its attachments you must not take any action based upon them and you must not copy or show them to anyone. Please send the email back to us and immediately and permanently delete it and its attachments. Where this email is unrelated to the business of University of the Arts London or of any of its group companies the opinions expressed in it are the opinions of the sender and do not necessarily constitute those of University of the Arts London (or the relevant group company). Where the sender's signature indicates that the email is sent on behalf of UAL Short Courses Limited the following also applies: UAL Short Courses Limited is a company registered in England and Wales under company number 02361261. Registered Office: University of the Arts London, 272 High Holborn, London WC1V 7EY
Very sorry to hear about your problems, Raymond. Cant say very much about it, since we do not yet use 17.05. But you should be searching on Bugzilla for the reports that got recently backported to 17.05.x. Might just include some that you need. Upgrading to the latest 17.05 would then perhaps address some issues. Just a side note: Very long delays in the past have been the result of date calculations with very far future dates like 9999 in expiry dates etc. Setting them back temporarily has been helpful in the past. (No guarantees; may be other factors involved.) Since your system is extensively customized, it could be helpful (for you) to know if your system without custom changes in a test environment would still experience the same problems.. And what about: Does your server have enough CPU and memory, etc ? Just a few remarks. Marcel -----Oorspronkelijk bericht----- Van: Koha [mailto:koha-bounces@lists.katipo.co.nz] Namens Raymund Delahunty Verzonden: donderdag 12 oktober 2017 13:29 Aan: koha@lists.katipo.co.nz Onderwerp: [Koha] Major, multiple problems with our Koha 17.05 system. VERY long sorry, but our Koha system is virtually unusable and we need help! We are experiencing dozens of outages daily (both OPAC and intranet, more often only OPAC) with our Koha 17.05.01.000 (Debian 3.2.89-2, perl 5.020002, mysql 14.14, apache 2.4.10). Searches can take 15+ seconds. While university staff provide first tier support, we don't have server access, and the second tier and hosting is via an external company. Of course we are working with that company to have them identify the cause, but I have decided to ask the community of any user on 17.05 has the problems we have. Our system is quite extensively customised, with major work done to the OPAC at https://libsearch.arts.ac.uk<https://libsearch.arts.ac.uk/>. We have some auto-renewal functionality we funded working, while we work on more improvements (we are NOT using the template toolkit, relying on workarounds to offer reasonably useful auto-renewal features). Our problems started in the summer vacation soon after the upgrade, where Koha returns functionality became unstable. It was taking 8 seconds to return one item. Our support company advised that it was the AddReturn (and, in particular, MarkIssueReturned) which was the pinch point. Seems some code to resolve the auto_increment bug works slower in 17.05 and our support company was correcting that with a view (I understand) to adding it to Master. Things improved a little, but with the start of term almost 2 weeks ago we are encountering so many problems... "We were alerted to issues at 4.22pm. and found that the plack server was no longer responding. We restarted plack and then apache and the service restored. We are trying to get to the bottom of what caused this issue. It appears to be completely separate from the high CPU problem that we are also seeing." I understand more resources have been allocated to mySQL and additional plack helpers have been allocated (whatever that means). We are heavy users of SIP (80% of transactions are via self issue). But there are so many problems with the returns unit that they are almost unusable. Front line staff are fed up! The sorters and the kiosks configured for returns (as well as issue) fail repeatedly dozens of times daily (ERR_SIP_COM_RECV). Reconnection does happen- takes about 40seconds or so, leading to queues. And the items have NOT come off the account and flow into the exceptions bin. At the same time the intranet report 500 server errors. I am pretty sure most of our problems are related to returns code, but could well be wrong/ oversimplifying things. Our support company advised: "We have added extra logging and can see that some checkin requests via the SIP2 protocol are causing the sipserver process to abort. The book is returned within Koha but as the process returns no information to the SIP2 client therefore the unit responds with an error. After waiting for the timeout period it then will reconnect to the server. We are adding further debug to the logs to work out why the sipserver process is aborting. It does seem to be a special subset of the checkin requests. I'll update you when we have more news." And more problems relating to total outages (with proxy errors 502 relating to invalid responses from an upstream server): "There appears to be a number of issues you are experiencing. Friday you had high CPU usage which is different to today's issues which we think are down to the apache webserver not restarting correctly. I will double check first thing tomorrow that the overnight apache restart has occurred properly. We are hoping the database connections change may address the high CPU issue. We will monitor closely for the errors we have seen in the apache error logs tomorrow." We are a university with 20,000 students across 6 colleges, and right now inductions and training in OPAC use is going on with new students. These continual problems are deeply embarrassing, and faith in the Koha system is dropping rapidly. Has any other Koha 17.05 user (especially one heavily reliant on SIP) experienced anything like this? Has anyone out there any suggestions? Ray Delahunty University of the Arts London This email and any attachments are intended solely for the addressee and may contain confidential information. If you are not the intended recipient of this email and/or its attachments you must not take any action based upon them and you must not copy or show them to anyone. Please send the email back to us and immediately and permanently delete it and its attachments. Where this email is unrelated to the business of University of the Arts London or of any of its group companies the opinions expressed in it are the opinions of the sender and do not necessarily constitute those of University of the Arts London (or the relevant group company). Where the sender's signature indicates that the email is sent on behalf of UAL Short Courses Limited the following also applies: UAL Short Courses Limited is a company registered in England and Wales under company number 02361261. Registered Office: University of the Arts London, 272 High Holborn, London WC1V 7EY _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
Hi Raymund, Il 12/10/2017 13:29, Raymund Delahunty ha scritto:
VERY long sorry, but our Koha system is virtually unusable and we need help!
generaly speaking, you can try to tune your MySQL setup, I find useful this tool: https://github.com/major/MySQLTuner-perl Attention that could you need more RAM on your server !! About SIP, do a search on bugzilla: https://bugs.koha-community.org/bugzilla3/ and see the open problems Bye Zeno Tajoli -- Zeno Tajoli /SVILUPPO PRODOTTI CINECA/ - Automazione Biblioteche Email: z.tajoli@cineca.it Fax: 051/6132198 *CINECA* Consorzio Interuniversitario - Sede operativa di Segrate (MI)
El jue., 12 oct. 2017 a las 8:29, Raymund Delahunty (<r.delahunty@arts.ac.uk>) escribió:
VERY long sorry, but our Koha system is virtually unusable and we need help!
We are experiencing dozens of outages daily (both OPAC and intranet, more often only OPAC) with our Koha 17.05.01.000 (Debian 3.2.89-2, perl 5.020002, mysql 14.14, apache 2.4.10). Searches can take 15+ seconds. While university staff provide first tier support, we don't have server access, and the second tier and hosting is via an external company. Of course we are working with that company to have them identify the cause, but I have decided to ask the community of any user on 17.05 has the problems we have.
It seems you haven't enabled Plack on your setup. Those numbers are really wrong. -- Tomás Cohen Arazi Theke Solutions (https://theke.io <http://theke.io/>) ✆ +54 9351 3513384 GPG: B2F3C15F
I am a bit desperate because I already told several times on this list that some versions of Koha must not be used. See https://wiki.koha-community.org/wiki/DBMS_auto_increment_fix and you will see that 17.05.01 is a buggy version My recommendation is to upgrade ASAP to the latest 17.05.x Maybe it will not change anything, but you should not use this version anyway. Regards, Jonathan On Thu, 12 Oct 2017 at 08:29 Raymund Delahunty <r.delahunty@arts.ac.uk> wrote:
VERY long sorry, but our Koha system is virtually unusable and we need help!
We are experiencing dozens of outages daily (both OPAC and intranet, more often only OPAC) with our Koha 17.05.01.000 (Debian 3.2.89-2, perl 5.020002, mysql 14.14, apache 2.4.10). Searches can take 15+ seconds. While university staff provide first tier support, we don't have server access, and the second tier and hosting is via an external company. Of course we are working with that company to have them identify the cause, but I have decided to ask the community of any user on 17.05 has the problems we have.
Our system is quite extensively customised, with major work done to the OPAC at https://libsearch.arts.ac.uk<https://libsearch.arts.ac.uk/>. We have some auto-renewal functionality we funded working, while we work on more improvements (we are NOT using the template toolkit, relying on workarounds to offer reasonably useful auto-renewal features).
Our problems started in the summer vacation soon after the upgrade, where Koha returns functionality became unstable. It was taking 8 seconds to return one item. Our support company advised that it was the AddReturn (and, in particular, MarkIssueReturned) which was the pinch point. Seems some code to resolve the auto_increment bug works slower in 17.05 and our support company was correcting that with a view (I understand) to adding it to Master.
Things improved a little, but with the start of term almost 2 weeks ago we are encountering so many problems...
"We were alerted to issues at 4.22pm. and found that the plack server was no longer responding. We restarted plack and then apache and the service restored. We are trying to get to the bottom of what caused this issue. It appears to be completely separate from the high CPU problem that we are also seeing."
I understand more resources have been allocated to mySQL and additional plack helpers have been allocated (whatever that means).
We are heavy users of SIP (80% of transactions are via self issue). But there are so many problems with the returns unit that they are almost unusable. Front line staff are fed up! The sorters and the kiosks configured for returns (as well as issue) fail repeatedly dozens of times daily (ERR_SIP_COM_RECV). Reconnection does happen- takes about 40seconds or so, leading to queues. And the items have NOT come off the account and flow into the exceptions bin. At the same time the intranet report 500 server errors. I am pretty sure most of our problems are related to returns code, but could well be wrong/ oversimplifying things.
Our support company advised: "We have added extra logging and can see that some checkin requests via the SIP2 protocol are causing the sipserver process to abort. The book is returned within Koha but as the process returns no information to the SIP2 client therefore the unit responds with an error. After waiting for the timeout period it then will reconnect to the server. We are adding further debug to the logs to work out why the sipserver process is aborting. It does seem to be a special subset of the checkin requests. I'll update you when we have more news."
And more problems relating to total outages (with proxy errors 502 relating to invalid responses from an upstream server):
"There appears to be a number of issues you are experiencing. Friday you had high CPU usage which is different to today's issues which we think are down to the apache webserver not restarting correctly. I will double check first thing tomorrow that the overnight apache restart has occurred properly. We are hoping the database connections change may address the high CPU issue. We will monitor closely for the errors we have seen in the apache error logs tomorrow."
We are a university with 20,000 students across 6 colleges, and right now inductions and training in OPAC use is going on with new students. These continual problems are deeply embarrassing, and faith in the Koha system is dropping rapidly. Has any other Koha 17.05 user (especially one heavily reliant on SIP) experienced anything like this? Has anyone out there any suggestions?
Ray Delahunty University of the Arts London
This email and any attachments are intended solely for the addressee and may contain confidential information. If you are not the intended recipient of this email and/or its attachments you must not take any action based upon them and you must not copy or show them to anyone. Please send the email back to us and immediately and permanently delete it and its attachments. Where this email is unrelated to the business of University of the Arts London or of any of its group companies the opinions expressed in it are the opinions of the sender and do not necessarily constitute those of University of the Arts London (or the relevant group company). Where the sender's signature indicates that the email is sent on behalf of UAL Short Courses Limited the following also applies: UAL Short Courses Limited is a company registered in England and Wales under company number 02361261. Registered Office: University of the Arts London, 272 High Holborn, London WC1V 7EY <https://maps.google.com/?q=272+High+Holborn,+London+WC1V+7EY&entry=gmail&source=g> _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
participants (5)
-
Jonathan Druart -
Marcel de Rooy -
Raymund Delahunty -
Tajoli Zeno -
Tomas Cohen Arazi