So I've found a really strange problem today... After starting work we realised that Koha would not give search results for any record in the database today. This includes searches in the catalogue and in the opac. I've seen similar things before so I checked that zebrasrv and mysqld are running...they are...even restarted the server a few times to prove they would start and a reboot wouldn't make a difference... I then told it to manually rebuild zebra - it seemed to do so fine but still no change. (Note: we have a cron that runs every 15 minutes and does this rebuild anyway) Looking at the log files I get the following. If anyone has an idea, please let me know - I'm out of them at the moment. koha-error_log: [Wed Sep 19 14:27:06 2012] [error] [client 172.16.255.226] [Wed Sep 19 14:27:06 2012] circulation.pl: Use of uninitialized value in string gt at /usr/share/koha/intranet/cgi-bin/circ/circulation.pl line 464., referer: http://tux-koha:8080/cgi-bin/koha/circ/circulation.pl [Wed Sep 19 14:27:06 2012] [error] [client 172.16.255.226] [Wed Sep 19 14:27:06 2012] circulation.pl: Use of uninitialized value in string gt at /usr/share/koha/intranet/cgi-bin/circ/circulation.pl line 464., referer: http://tux-koha:8080/cgi-bin/koha/circ/circulation.pl [Wed Sep 19 14:27:06 2012] [error] [client 172.16.255.226] [Wed Sep 19 14:27:06 2012] circulation.pl: Use of uninitialized value in string gt at /usr/share/koha/intranet/cgi-bin/circ/circulation.pl line 464., referer: http://tux-koha:8080/cgi-bin/koha/circ/circulation.pl [Wed Sep 19 14:27:06 2012] [error] [client 172.16.255.226] [Wed Sep 19 14:27:06 2012] circulation.pl: Use of uninitialized value in concatenation (.) or string at /usr/share/koha/intranet/cgi-bin/circ/circulation.pl line 644., referer: http://tux-koha:8080/cgi-bin/koha/circ/circulation.pl [Wed Sep 19 14:27:23 2012] [error] [client 172.16.255.226] [Wed Sep 19 14:27:23 2012] circulation.pl: Reference found where even-sized list expected at /usr/share/koha/intranet/cgi-bin/circ/circulation.pl line 97., referer: http://tux-koha:8080/cgi-bin/koha/circ/circulation.pl [Wed Sep 19 14:27:23 2012] [error] [client 172.16.255.226] [Wed Sep 19 14:27:23 2012] circulation.pl: Use of uninitialized value $print in string eq at /usr/share/koha/intranet/cgi-bin/circ/circulation.pl line 163., referer: http://tux-koha:8080/cgi-bin/koha/circ/circulation.pl [Wed Sep 19 14:27:23 2012] [error] [client 172.16.255.226] [Wed Sep 19 14:27:23 2012] circulation.pl: Use of uninitialized value in string eq at /usr/share/koha/intranet/cgi-bin/circ/circulation.pl line 168., referer: http://tux-koha:8080/cgi-bin/koha/circ/circulation.pl [Wed Sep 19 14:27:23 2012] [error] [client 172.16.255.226] [Wed Sep 19 14:27:23 2012] circulation.pl: Use of uninitialized value $print in string eq at /usr/share/koha/intranet/cgi-bin/circ/circulation.pl line 175., referer: http://tux-koha:8080/cgi-bin/koha/circ/circulation.pl [Wed Sep 19 14:27:23 2012] [error] [client 172.16.255.226] [Wed Sep 19 14:27:23 2012] circulation.pl: Use of uninitialized value in hash element at /usr/share/koha/intranet/cgi-bin/circ/circulation.pl line 644., referer: http://tux-koha:8080/cgi-bin/koha/circ/circulation.pl [Wed Sep 19 14:27:23 2012] [error] [client 172.16.255.226] [Wed Sep 19 14:27:23 2012] circulation.pl: Use of uninitialized value in concatenation (.) or string at /usr/share/koha/intranet/cgi-bin/circ/circulation.pl line 644., referer: http://tux-koha:8080/cgi-bin/koha/circ/circulation.pl koha-opac_log: [Wed Sep 19 13:32:53 2012] [error] [client 172.16.255.175] File does not exist: /usr/share/koha/opac/htdocs/NoImage.png, referer: http://tux-koha.stpeters.nsw.edu.au/ [Wed Sep 19 13:33:29 2012] [error] [client 172.16.255.226] File does not exist: /usr/share/koha/opac/htdocs/NoImage.png, referer: http://tux-koha.stpeters.nsw.edu.au/ [Wed Sep 19 13:35:30 2012] [error] [client 172.16.1.12] File does not exist: /usr/share/koha/opac/htdocs/NoImage.png, referer: http://tux-koha/ [Wed Sep 19 13:35:30 2012] [error] [client 172.16.1.12] File does not exist: /usr/share/koha/opac/htdocs/NoImage.png, referer: http://tux-koha/ [Wed Sep 19 13:35:33 2012] [error] [client 172.16.254.94] File does not exist: /usr/share/koha/opac/htdocs/NoImage.png, referer: http://tux-koha/ [Wed Sep 19 13:51:05 2012] [error] [client 172.16.4.222] File does not exist: /usr/share/koha/opac/htdocs/NoImage.png, referer: http://tux-koha.stpeters.nsw.edu.au/ [Wed Sep 19 14:07:30 2012] [error] [client 172.16.255.175] File does not exist: /usr/share/koha/opac/htdocs/NoImage.png, referer: http://tux-koha.stpeters.nsw.edu.au/ [Wed Sep 19 14:44:29 2012] [error] [client 172.16.254.230] File does not exist: /usr/share/koha/opac/htdocs/NoImage.png, referer: http://tux-koha.stpeters.nsw.edu.au/ [Wed Sep 19 23:03:43 2012] [error] [client 172.16.1.1] File does not exist: /usr/share/koha/opac/htdocs/NoImage.png, referer: http://tux-koha/ [Wed Sep 19 23:03:51 2012] [error] [client 172.16.1.1] File does not exist: /usr/share/koha/opac/htdocs/NoImage.png, referer: http://tux-koha/cgi-bin/koha/opac-search.pl?q=101 koha-zebradaemon.err: 20120703 17:38:53 koha-zebra-ctl.kohadb: client (pid 850) killed by signal 15, stopping 20120703 17:39:41 koha-zebra-ctl.kohadb: client (pid 848) killed by signal 15, stopping 20120712 11:27:24 koha-zebra-ctl.kohadb: client (pid 884) killed by signal 15, stopping 20120806 08:42:55 koha-zebra-ctl.kohadb: client (pid 866) killed by signal 15, stopping 20120806 09:21:50 koha-zebra-ctl.kohadb: client (pid 876) killed by signal 15, stopping 20120918 15:16:03 koha-zebra-ctl.kohadb: client (pid 864) killed by signal 15, stopping 20120918 15:42:42 koha-zebra-ctl.kohadb: client (pid 837) killed by signal 15, stopping 20120918 16:21:31 koha-zebra-ctl.kohadb: client (pid 856) killed by signal 15, stopping 20120919 11:55:14 koha-zebra-ctl.kohadb: client (pid 889) killed by signal 15, stopping 20120919 22:51:23 koha-zebra-ctl.kohadb: client (pid 899) killed by signal 15, stopping koha-zebradaemon-output.log: 22:07:15-19/09 zebrasrv(14) [warn] ir_session (exception) 22:07:20-19/09 zebrasrv(15) [warn] zebra_lock_create fail fname=/var/lock/koha/zebradb/biblios/norm..LCK [Permission denied] 22:07:20-19/09 zebrasrv(15) [warn] zebra_lock_create fail fname=/var/lock/koha/zebradb/biblios/shadow..LCK [Permission denied] 22:07:20-19/09 zebrasrv(15) [warn] ir_session (exception) 22:07:25-19/09 zebrasrv(16) [warn] zebra_lock_create fail fname=/var/lock/koha/zebradb/biblios/norm..LCK [Permission denied] 22:07:25-19/09 zebrasrv(16) [warn] zebra_lock_create fail fname=/var/lock/koha/zebradb/biblios/shadow..LCK [Permission denied] 22:07:25-19/09 zebrasrv(16) [warn] ir_session (exception) 23:03:50-19/09 zebrasrv(1) [warn] zebra_lock_create fail fname=/var/lock/koha/zebradb/biblios/norm..LCK [Permission denied] 23:03:50-19/09 zebrasrv(1) [warn] zebra_lock_create fail fname=/var/lock/koha/zebradb/biblios/shadow..LCK [Permission denied] 23:03:50-19/09 zebrasrv(1) [warn] ir_session (exception) koha-zebraqueue.err: 20120703 17:38:53 koha-zebraqueue-ctl-kohadb: client (pid 857) killed by signal 15, stopping 20120703 17:39:41 koha-zebraqueue-ctl-kohadb: client (pid 868) killed by signal 15, stopping 20120712 11:27:24 koha-zebraqueue-ctl-kohadb: client (pid 911) killed by signal 15, stopping 20120806 08:42:55 koha-zebraqueue-ctl-kohadb: client (pid 875) killed by signal 15, stopping 20120806 09:21:50 koha-zebraqueue-ctl-kohadb: client (pid 883) killed by signal 15, stopping 20120918 15:16:03 koha-zebraqueue-ctl-kohadb: client (pid 876) killed by signal 15, stopping 20120918 15:42:42 koha-zebraqueue-ctl-kohadb: client (pid 852) killed by signal 15, stopping 20120918 16:21:31 koha-zebraqueue-ctl-kohadb: client (pid 881) killed by signal 15, stopping 20120919 11:55:14 koha-zebraqueue-ctl-kohadb: client (pid 899) killed by signal 15, stopping 20120919 22:51:23 koha-zebraqueue-ctl-kohadb: client (pid 905) killed by signal 15, stopping koha-zebraqueue-output.log: DBD::mysql::db commit failed: MySQL server has gone away at /usr/share/koha/bin/zebraqueue_daemon.pl line 135. DBI connect('dbname=kohadb;host=localhost;port=3306','koha',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/share/koha/lib/C4/Context.pm line 758 DBI connect('dbname=kohadb;host=localhost;port=3306','koha',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/share/koha/lib/C4/Context.pm line 758 DBI connect('dbname=kohadb;host=localhost;port=3306','koha',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/share/koha/lib/C4/Context.pm line 758 DBD::mysql::st execute failed: MySQL server has gone away at /usr/share/koha/bin/zebraqueue_daemon.pl line 124. DBD::mysql::st fetchrow_hashref failed: fetch() without execute() at /usr/share/koha/bin/zebraqueue_daemon.pl line 125. DBD::mysql::db commit failed: MySQL server has gone away at /usr/share/koha/bin/zebraqueue_daemon.pl line 135. DBI connect('dbname=kohadb;host=localhost;port=3306','koha',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/share/koha/lib/C4/Context.pm line 758 DBI connect('dbname=kohadb;host=localhost;port=3306','koha',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/share/koha/lib/C4/Context.pm line 758 DBI connect('dbname=kohadb;host=localhost;port=3306','koha',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/share/koha/lib/C4/Context.pm line 758 Thanks, Seth Mayo IT Technical Manager Email: smayo@stpeters.nsw.edu.au ------------------------------------------------------- St Peter's Anglican Primary School Howe Street Campbelltown NSW 2560 PH: 02 4627 2990 DID: 02 8203 1368 Website: http://www.stpeters.nsw.edu.au
On 2012-09-20, at 1:36 AM, Seth Mayo wrote:
So I've found a really strange problem today...
I then told it to manually rebuild zebra - it seemed to do so fine but still no change. (Note: we have a cron that runs every 15 minutes and does this rebuild anyway)
koha-zebradaemon-output.log: 22:07:15-19/09 zebrasrv(14) [warn] ir_session (exception) 22:07:20-19/09 zebrasrv(15) [warn] zebra_lock_create fail fname=/var/lock/koha/zebradb/biblios/norm..LCK [Permission denied]
your zebra files have incorrect permissions… (you ran rebuild-zebra.pl accidentally as root?)
koha-zebraqueue-output.log: DBD::mysql::db commit failed: MySQL server has gone away at /usr/share/koha/bin/zebraqueue_daemon.pl line 135. DBI connect('dbname=kohadb;host=localhost;port=3306','koha',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/share/koha/lib/C4/Context.pm line 758
...and your database hasn't started properly
Greetings,
koha-zebradaemon-output.log: 22:07:15-19/09 zebrasrv(14) [warn] ir_session (exception) 22:07:20-19/09 zebrasrv(15) [warn] zebra_lock_create fail fname=/var/lock/koha/zebradb/biblios/norm..LCK [Permission denied]
Mason James said:
your zebra files have incorrect permissions… (you ran rebuild-zebra.pl accidentally as root?)
If your /etc/cron.d/koha file says "koha" as the user, then you need to: chown -R -v koha.koha /var/lock/koha and perhaps other directories too. The -v shows you what it is doing. The -R is recursive, so subdirectories and files get fixed too. $ sudo su - koha $ <now run the reindex, and make sure the environment variables KOHA_CONF and PERL5LIB are set properly too>
koha-zebraqueue-output.log: DBD::mysql::db commit failed: MySQL server has gone away at /usr/share/koha/bin/zebraqueue_daemon.pl line 135.
Oooo. I had this, it may be a low memory problem. If your system thinks a process is going to hog too much memory, it kills it. What does "free -m" say? If you have less than 55MB free and no swap, you are seriously up a creek. You need more memory. We were running a machine with only 512MB of RAM with no swap. Needless to say, things were mysteriously failing. If you can't give Koha a machine with at least (and I'm saying this is still not really enough) 1GB, then you can expect these sorts of problems. GPML, Mark Tompsett
Thanks for all the pointers to potential problems :-) Here's what I've found... The mysqld database is running fine - the error in the log was an old entry. Memory is at 200MB+ free out of 1.5GB and the swap is barely touched. Our rebuild-zebra.pl has always run as root - both in the cron job and manually. It has been this way for more than a year without problems. However, I chown'ed the path mentioned and then changed my cron jobs to run as koha instead of root and this did somehow bring back the zebra index. rebuild-zebra.pl now has the following errors though...which ones should I worry about and has anyone seen them before? 10:06:40-20/09 zebraidx(9341) [warn] Missing attribute 'type' in attribute info 10:09:18-20/09 zebraidx(9341) [warn] Unknown register type: 10:09:34-20/09 zebraidx(9351) [warn] Failed to open /var/lib/koha/zebradb/biblios/register/reci-0.mf [Permission denied] 10:09:34-20/09 zebraidx(9351) [fatal] mf_write: reci error (1) 10:09:34-20/09 zebraidx(9351) [fatal] write commit block 10:09:34-20/09 zebraidx(9351) [warn] zebra_commit: failed BTW the first 2 lines 'missing attribute' and 'unknown register' have always been there as long as I can remember so I'm guessing they aren't important. However, I would have expected zebra_commit: failed to mean I wouldn't have a working index but it seems that isn't the case. Do I need to change more permissions to make every 'koha' - if so, what paths? Otherwise, anything else I'm missing? Thanks for all your help! I really appreciate it! Seth Mayo IT Technical Manager Email: smayo@stpeters.nsw.edu.au ------------------------------------------------------- St Peter's Anglican Primary School Howe Street Campbelltown NSW 2560 PH: 02 4627 2990 DID: 02 8203 1368 Website: http://www.stpeters.nsw.edu.au On 20 September 2012 00:47, Mark Tompsett <mtompset@hotmail.com> wrote:
Greetings,
koha-zebradaemon-output.log:
22:07:15-19/09 zebrasrv(14) [warn] ir_session (exception) 22:07:20-19/09 zebrasrv(15) [warn] zebra_lock_create fail fname=/var/lock/koha/zebradb/**biblios/norm..LCK [Permission denied]
Mason James said:
your zebra files have incorrect permissions… (you ran rebuild-zebra.placcidentally as root?)
If your /etc/cron.d/koha file says "koha" as the user, then you need to: chown -R -v koha.koha /var/lock/koha and perhaps other directories too. The -v shows you what it is doing. The -R is recursive, so subdirectories and files get fixed too.
$ sudo su - koha $ <now run the reindex, and make sure the environment variables KOHA_CONF and PERL5LIB are set properly too>
koha-zebraqueue-output.log:
DBD::mysql::db commit failed: MySQL server has gone away at /usr/share/koha/bin/zebraqueue**_daemon.pl <http://zebraqueue_daemon.pl>line 135.
Oooo. I had this, it may be a low memory problem. If your system thinks a process is going to hog too much memory, it kills it. What does "free -m" say? If you have less than 55MB free and no swap, you are seriously up a creek. You need more memory.
We were running a machine with only 512MB of RAM with no swap. Needless to say, things were mysteriously failing. If you can't give Koha a machine with at least (and I'm saying this is still not really enough) 1GB, then you can expect these sorts of problems.
GPML, Mark Tompsett
Ok so chown koha.koha -R on /var/lib/koha/zebradb/biblios got rid of the following...so now it is down to the two that were always there. 10:09:34-20/09 zebraidx(9351) [warn] Failed to open /var/lib/koha/zebradb/biblios/ register/reci-0.mf [Permission denied] 10:09:34-20/09 zebraidx(9351) [fatal] mf_write: reci error (1) 10:09:34-20/09 zebraidx(9351) [fatal] write commit block 10:09:34-20/09 zebraidx(9351) [warn] zebra_commit: failed I'll keep running everything as the koha user now as that looks like what it should have been from the beginning. Thanks for the tips and help everyone - all seems back to normal now. Seth Mayo IT Technical Manager Email: smayo@stpeters.nsw.edu.au ------------------------------------------------------- St Peter's Anglican Primary School Howe Street Campbelltown NSW 2560 PH: 02 4627 2990 DID: 02 8203 1368 Website: http://www.stpeters.nsw.edu.au On 20 September 2012 10:24, Seth Mayo <smayo@stpeters.nsw.edu.au> wrote:
10:09:34-20/09 zebraidx(9351) [warn] Failed to open /var/lib/koha/zebradb/biblios/register/reci-0.mf [Permission denied] 10:09:34-20/09 zebraidx(9351) [fatal] mf_write: reci error (1) 10:09:34-20/09 zebraidx(9351) [fatal] write commit block 10:09:34-20/09 zebraidx(9351) [warn] zebra_commit: failed
On Wed, Sep 19, 2012 at 9:47 PM, Seth Mayo <smayo@stpeters.nsw.edu.au> wrote:
Ok so chown koha.koha -R on /var/lib/koha/zebradb/biblios got rid of the following...so now it is down to the two that were always there.
10:09:34-20/09 zebraidx(9351) [warn] Failed to open /var/lib/koha/zebradb/biblios/
$ mkdir -p /var/lib/koha/zebradb/biblios/ ; chown -R koha:koha /var/lib/koha/zebradb/biblios/ Maybe? Regards To+
Seth, I then told it to manually rebuild zebra - it seemed to do so fine but
still no change. (Note: we have a cron that runs every 15 minutes and does this rebuild anyway)
koha-zebraqueue.err:
[etc.] I think the problem is just the Zebra permissions, as Mason pointed out. However, make sure you're not using zebraqueue_daemon. It is deprecated and causes problems. Regards Jared -- Jared Camins-Esakov Bibliographer, C & P Bibliography Services, LLC (phone) +1 (917) 727-3445 (e-mail) jcamins@cpbibliography.com (web) http://www.cpbibliography.com/
participants (5)
-
Jared Camins-Esakov -
Mark Tompsett -
Mason James -
Seth Mayo -
Tomas Cohen Arazi