Zebra indexing question
I am noticing some errors in the koha-zebradaemon-output.log file. When new records are added it takes a bit longer to index than we think they should. Running rebuild zebra is often faster. The log file shows the following: 14:31:44-18/11 zebrasrv(55) [warn] Failed to open /var/lib/bbmkoha/zebradb/biblios/register/reci-0.mf [Permission denied] 14:31:44-18/11 zebrasrv(55) [fatal] mf_write: reci error (1) 14:31:44-18/11 zebrasrv(55) [fatal] write commit block 14:31:44-18/11 zebrasrv(55) [warn] zebra_commit: failed Zebra ends up indexing and search works, but I am concerned about the errors. Any ideas? Thanks, John +----------------------------------------------------------------------- -----+ John Chadwick, Ed.D. Information Technology Manager New Mexico State Library 1209 Camino Carlos Rey Santa Fe, NM 87507 Phone: 505-476-9740 Cell: 505-629-8116 Fax: 505-476-9761 john.chadwick@state.nm.us http://www.nmstatelibrary.org Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System. ______________________________________________________________________ Confidentiality Notice: This e-mail,including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review,use,disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. This email has been scanned using Webroot Email Security. ______________________________________________________________________
The recommended configuration is to run rebuild_zebra out of crontab (See the example crontab file) but I'm not sure if this error is related. 2009/11/18 Chadwick, John, DCA <john.chadwick@state.nm.us>
I am noticing some errors in the koha-zebradaemon-output.log file. When new records are added it takes a bit longer to index than we think they should. Running rebuild zebra is often faster.
The log file shows the following:
14:31:44-18/11 zebrasrv(55) [warn] Failed to open /var/lib/bbmkoha/zebradb/biblios/register/reci-0.mf [Permission denied]
14:31:44-18/11 zebrasrv(55) [fatal] mf_write: reci error (1)
14:31:44-18/11 zebrasrv(55) [fatal] write commit block
14:31:44-18/11 zebrasrv(55) [warn] zebra_commit: failed
Zebra ends up indexing and search works, but I am concerned about the errors. Any ideas?
OK, baseline question: does /var/lib/bbmkoha/zebradb/biblios/register/reci-0.mf exist? What are the owner:group and permissions? --joe
The first batch of records (500) went in with no errors. The second batch of 500 went in and generated the errors. The permissions and ownership did not change. The ownership WAS root and permissions were read/write. I changed the permissions on the files to 766 and owner/group to Koha. John +----------------------------------------------------------------------- -----+ John Chadwick, Ed.D. Information Technology Manager New Mexico State Library 1209 Camino Carlos Rey Santa Fe, NM 87507 Phone: 505-476-9740 Cell: 505-629-8116 Fax: 505-476-9761 john.chadwick@state.nm.us http://www.nmstatelibrary.org ________________________________ From: Joe Atzberger [mailto:ohiocore@gmail.com] Sent: Wednesday, November 18, 2009 3:11 PM To: Chadwick, John, DCA Cc: Koha@lists.katipo.co.nz Subject: Re: [Koha] Zebra indexing question The recommended configuration is to run rebuild_zebra out of crontab (See the example crontab file) but I'm not sure if this error is related. 2009/11/18 Chadwick, John, DCA <john.chadwick@state.nm.us> I am noticing some errors in the koha-zebradaemon-output.log file. When new records are added it takes a bit longer to index than we think they should. Running rebuild zebra is often faster. The log file shows the following: 14:31:44-18/11 zebrasrv(55) [warn] Failed to open /var/lib/bbmkoha/zebradb/biblios/register/reci-0.mf [Permission denied] 14:31:44-18/11 zebrasrv(55) [fatal] mf_write: reci error (1) 14:31:44-18/11 zebrasrv(55) [fatal] write commit block 14:31:44-18/11 zebrasrv(55) [warn] zebra_commit: failed Zebra ends up indexing and search works, but I am concerned about the errors. Any ideas? OK, baseline question: does /var/lib/bbmkoha/zebradb/biblios/register/reci-0.mf exist? What are the owner:group and permissions? --joe ________________________________ This inbound email has been scanned for malicious software and transmitted safely to you using Webroot Email Security. ________________________________ This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. ______________________________________________________________________ Confidentiality Notice: This e-mail,including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review,use,disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. This email has been scanned using Webroot Email Security. ______________________________________________________________________
Looks like the issue was ownership, not permissions. As a test I deleted the log files and restarted the server. The files in the directory remained with the koha user and group, but the permissions got set back down. -rw-r--r-- 1 koha koha 8192 2009-11-18 15:40 reci-0.mf John +----------------------------------------------------------------------- -----+ John Chadwick, Ed.D. Information Technology Manager New Mexico State Library 1209 Camino Carlos Rey Santa Fe, NM 87507 Phone: 505-476-9740 Cell: 505-629-8116 Fax: 505-476-9761 john.chadwick@state.nm.us http://www.nmstatelibrary.org ________________________________ From: Joe Atzberger [mailto:ohiocore@gmail.com] Sent: Wednesday, November 18, 2009 3:11 PM To: Chadwick, John, DCA Cc: Koha@lists.katipo.co.nz Subject: Re: [Koha] Zebra indexing question The recommended configuration is to run rebuild_zebra out of crontab (See the example crontab file) but I'm not sure if this error is related. 2009/11/18 Chadwick, John, DCA <john.chadwick@state.nm.us> I am noticing some errors in the koha-zebradaemon-output.log file. When new records are added it takes a bit longer to index than we think they should. Running rebuild zebra is often faster. The log file shows the following: 14:31:44-18/11 zebrasrv(55) [warn] Failed to open /var/lib/bbmkoha/zebradb/biblios/register/reci-0.mf [Permission denied] 14:31:44-18/11 zebrasrv(55) [fatal] mf_write: reci error (1) 14:31:44-18/11 zebrasrv(55) [fatal] write commit block 14:31:44-18/11 zebrasrv(55) [warn] zebra_commit: failed Zebra ends up indexing and search works, but I am concerned about the errors. Any ideas? OK, baseline question: does /var/lib/bbmkoha/zebradb/biblios/register/reci-0.mf exist? What are the owner:group and permissions? --joe ________________________________ This inbound email has been scanned for malicious software and transmitted safely to you using Webroot Email Security. ________________________________ Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System. ______________________________________________________________________ Confidentiality Notice: This e-mail,including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review,use,disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. This email has been scanned using Webroot Email Security. ______________________________________________________________________
It seems like a lot of folks have had similar problems, and we have as well, and maybe it would help to explain why these problems occur. Rebuild_zebra.pl -r deletes all of the files in the Zebra db directories (such as reci-0.mf) and then recreates them. Thus, permissions will be lost, and the files will be owned by the user who ran rebuild_zebra.pl. If one rebuilds the zebra indexes as root, the daemons, which typically run under the user koha, will not be able to update the indexes. Thus, it's important then that the zebra rebuilds are put in the cronjob file of the user koha, and not root. Also important is that other users, such as root, don't manually execute rebuilds. If one desires that another user be able to execute rebuild_zebra.pl, he should be given the permission to execute 'sudo -u Koha .../rebuild_zebra.pl,' (if you want to do this, you also have to edit the sudoers file to pass the PERL5LIB variable with the env_keep option as by default sudo strips away almost all environment variables). Or, as root user, one can use a simple 'su koha' and then the rebuild_zebra.pl command. I've also tried to set the sticky bit on rebuild_zebra.pl, but for whatever reason it didn't seem to work due to some problem with the PERL5LIB variable that I wasn't able to figure. That seems to me the easiest thing to do, if anybody has any idea how to make it work. If it worked and were the default, I think it would help folks to avoid a great deal of the problems that come up with zebra. Best regards to all, Patrick Redmon 2009/11/18 Chadwick, John, DCA <john.chadwick@state.nm.us>
Looks like the issue was ownership, not permissions.
As a test I deleted the log files and restarted the server. The files in the directory remained with the koha user and group, but the permissions got set back down.
-rw-r--r-- 1 koha koha 8192 2009-11-18 15:40 reci-0.mf
John
+----------------------------------------------------------------------------+
John Chadwick, Ed.D. Information Technology Manager
New Mexico State Library
1209 Camino Carlos Rey
Santa Fe, NM 87507
Phone: 505-476-9740 Cell: 505-629-8116 Fax: 505-476-9761
john.chadwick@state.nm.us
http://www.nmstatelibrary.org ------------------------------
*From:* Joe Atzberger [mailto:ohiocore@gmail.com] *Sent:* Wednesday, November 18, 2009 3:11 PM *To:* Chadwick, John, DCA *Cc:* Koha@lists.katipo.co.nz *Subject:* Re: [Koha] Zebra indexing question
The recommended configuration is to run rebuild_zebra out of crontab (See the example crontab file) but I'm not sure if this error is related.
2009/11/18 Chadwick, John, DCA <john.chadwick@state.nm.us>
I am noticing some errors in the koha-zebradaemon-output.log file. When new records are added it takes a bit longer to index than we think they should. Running rebuild zebra is often faster.
The log file shows the following:
14:31:44-18/11 zebrasrv(55) [warn] Failed to open /var/lib/bbmkoha/zebradb/biblios/register/reci-0.mf [Permission denied]
14:31:44-18/11 zebrasrv(55) [fatal] mf_write: reci error (1)
14:31:44-18/11 zebrasrv(55) [fatal] write commit block
14:31:44-18/11 zebrasrv(55) [warn] zebra_commit: failed
Zebra ends up indexing and search works, but I am concerned about the errors. Any ideas?
OK, baseline question: does /var/lib/bbmkoha/zebradb/biblios/register/reci-0.mf exist? What are the owner:group and permissions?
--joe ------------------------------
This inbound email has been scanned for malicious software and transmitted safely to you using Webroot Email Security. ------------------------------
Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System.
------------------------------ Confidentiality Notice: This e-mail,including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review,use,disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. This email has been scanned using Webroot Email Security. ------------------------------
_______________________________________________ Koha mailing list Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
Thanks for the advice. When we run rebuild zebra manually we had been running it as root through Webmin. I can set up the manual job to run under Koha user. Now that things are working with no errors we will make the changes necessary so that we don't lose ownership again. John ________________________________ From: Biblioteca Casa Balthasar [mailto:biblioteca@casabalthasar.org] Sent: Thursday, November 19, 2009 2:43 AM To: Chadwick, John, DCA; koha koha Subject: Re: [Koha] Zebra indexing question It seems like a lot of folks have had similar problems, and we have as well, and maybe it would help to explain why these problems occur. Rebuild_zebra.pl -r deletes all of the files in the Zebra db directories (such as reci-0.mf) and then recreates them. Thus, permissions will be lost, and the files will be owned by the user who ran rebuild_zebra.pl. If one rebuilds the zebra indexes as root, the daemons, which typically run under the user koha, will not be able to update the indexes. Thus, it's important then that the zebra rebuilds are put in the cronjob file of the user koha, and not root. Also important is that other users, such as root, don't manually execute rebuilds. If one desires that another user be able to execute rebuild_zebra.pl, he should be given the permission to execute 'sudo -u Koha .../rebuild_zebra.pl,' (if you want to do this, you also have to edit the sudoers file to pass the PERL5LIB variable with the env_keep option as by default sudo strips away almost all environment variables). Or, as root user, one can use a simple 'su koha' and then the rebuild_zebra.pl command. I've also tried to set the sticky bit on rebuild_zebra.pl, but for whatever reason it didn't seem to work due to some problem with the PERL5LIB variable that I wasn't able to figure. That seems to me the easiest thing to do, if anybody has any idea how to make it work. If it worked and were the default, I think it would help folks to avoid a great deal of the problems that come up with zebra. Best regards to all, Patrick Redmon 2009/11/18 Chadwick, John, DCA <john.chadwick@state.nm.us> Looks like the issue was ownership, not permissions. As a test I deleted the log files and restarted the server. The files in the directory remained with the koha user and group, but the permissions got set back down. -rw-r--r-- 1 koha koha 8192 2009-11-18 15:40 reci-0.mf John +----------------------------------------------------------------------- -----+ John Chadwick, Ed.D. Information Technology Manager New Mexico State Library 1209 Camino Carlos Rey Santa Fe, NM 87507 Phone: 505-476-9740 Cell: 505-629-8116 Fax: 505-476-9761 john.chadwick@state.nm.us http://www.nmstatelibrary.org ________________________________ From: Joe Atzberger [mailto:ohiocore@gmail.com] Sent: Wednesday, November 18, 2009 3:11 PM To: Chadwick, John, DCA Cc: Koha@lists.katipo.co.nz Subject: Re: [Koha] Zebra indexing question The recommended configuration is to run rebuild_zebra out of crontab (See the example crontab file) but I'm not sure if this error is related. 2009/11/18 Chadwick, John, DCA <john.chadwick@state.nm.us> I am noticing some errors in the koha-zebradaemon-output.log file. When new records are added it takes a bit longer to index than we think they should. Running rebuild zebra is often faster. The log file shows the following: 14:31:44-18/11 zebrasrv(55) [warn] Failed to open /var/lib/bbmkoha/zebradb/biblios/register/reci-0.mf [Permission denied] 14:31:44-18/11 zebrasrv(55) [fatal] mf_write: reci error (1) 14:31:44-18/11 zebrasrv(55) [fatal] write commit block 14:31:44-18/11 zebrasrv(55) [warn] zebra_commit: failed Zebra ends up indexing and search works, but I am concerned about the errors. Any ideas? OK, baseline question: does /var/lib/bbmkoha/zebradb/biblios/register/reci-0.mf exist? What are the owner:group and permissions? --joe ________________________________ This inbound email has been scanned for malicious software and transmitted safely to you using Webroot Email Security. ________________________________ Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System. ________________________________ Confidentiality Notice: This e-mail,including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review,use,disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. This email has been scanned using Webroot Email Security. ________________________________ _______________________________________________ Koha mailing list Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System. ______________________________________________________________________ Confidentiality Notice: This e-mail,including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review,use,disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. This email has been scanned using Webroot Email Security. ______________________________________________________________________
Just don't run rebuild_zebra as root. The simplest thing would be to have the script check to see if it was being run as root and fail loudly in that case. --joe
We have modified our script to run from Webmin and I added two lines to the script so that file and group ownership is reset to koha. Here is the shell script we are running now. I did change the cron job to run as Koha, but the extra lines are a bit of insurance. #!/bin/bash export PERL5LIB=/koha/bbmkoha/lib/ perl /koha/bbmkoha/bin/migration_tools/rebuild_zebra.pl -b -r -v echo "changing owner and group in /var/lib/bbmkoha/zebradb/biblios/register to koha chown koha /var/lib/bbmkoha/zebradb/biblios/register/* chgrp koha /var/lib/bbmkoha/zebradb/biblios/register/* ls -al /var/lib/bbmkoha/zebradb/biblios/register/ John +----------------------------------------------------------------------- -----+ John Chadwick, Ed.D. Information Technology Manager New Mexico State Library 1209 Camino Carlos Rey Santa Fe, NM 87507 Phone: 505-476-9740 Cell: 505-629-8116 Fax: 505-476-9761 john.chadwick@state.nm.us http://www.nmstatelibrary.org ________________________________ From: koha-bounces@lists.katipo.co.nz [mailto:koha-bounces@lists.katipo.co.nz] On Behalf Of Joe Atzberger Sent: Thursday, November 19, 2009 10:42 AM To: Biblioteca Casa Balthasar Cc: koha koha Subject: Re: [Koha] Zebra indexing question Just don't run rebuild_zebra as root. The simplest thing would be to have the script check to see if it was being run as root and fail loudly in that case. --joe ________________________________ This inbound email has been scanned for malicious software and transmitted safely to you using Webroot Email Security. ________________________________ Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System. ______________________________________________________________________ Confidentiality Notice: This e-mail,including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review,use,disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. This email has been scanned using Webroot Email Security. ______________________________________________________________________
This data has been added to the new manual: http://git.biblibre.com/cgi-bin/gitweb.cgi?p=kohadocs;a=summary Thanks Nicole C. Engard Doc Manager 2009/11/19 Biblioteca Casa Balthasar <biblioteca@casabalthasar.org>:
It seems like a lot of folks have had similar problems, and we have as well, and maybe it would help to explain why these problems occur. Rebuild_zebra.pl -r deletes all of the files in the Zebra db directories (such as reci-0.mf) and then recreates them. Thus, permissions will be lost, and the files will be owned by the user who ran rebuild_zebra.pl. If one rebuilds the zebra indexes as root, the daemons, which typically run under the user koha, will not be able to update the indexes. Thus, it's important then that the zebra rebuilds are put in the cronjob file of the user koha, and not root. Also important is that other users, such as root, don't manually execute rebuilds. If one desires that another user be able to execute rebuild_zebra.pl, he should be given the permission to execute 'sudo -u Koha .../rebuild_zebra.pl,' (if you want to do this, you also have to edit the sudoers file to pass the PERL5LIB variable with the env_keep option as by default sudo strips away almost all environment variables). Or, as root user, one can use a simple 'su koha' and then the rebuild_zebra.pl command. I've also tried to set the sticky bit on rebuild_zebra.pl, but for whatever reason it didn't seem to work due to some problem with the PERL5LIB variable that I wasn't able to figure. That seems to me the easiest thing to do, if anybody has any idea how to make it work. If it worked and were the default, I think it would help folks to avoid a great deal of the problems that come up with zebra. Best regards to all, Patrick Redmon
2009/11/18 Chadwick, John, DCA <john.chadwick@state.nm.us>
Looks like the issue was ownership, not permissions.
As a test I deleted the log files and restarted the server. The files in the directory remained with the koha user and group, but the permissions got set back down.
-rw-r--r-- 1 koha koha 8192 2009-11-18 15:40 reci-0.mf
John
+----------------------------------------------------------------------------+
John Chadwick, Ed.D. Information Technology Manager
New Mexico State Library
1209 Camino Carlos Rey
Santa Fe, NM 87507
Phone: 505-476-9740 Cell: 505-629-8116 Fax: 505-476-9761
john.chadwick@state.nm.us
________________________________
From: Joe Atzberger [mailto:ohiocore@gmail.com] Sent: Wednesday, November 18, 2009 3:11 PM To: Chadwick, John, DCA Cc: Koha@lists.katipo.co.nz Subject: Re: [Koha] Zebra indexing question
The recommended configuration is to run rebuild_zebra out of crontab (See the example crontab file) but I'm not sure if this error is related.
2009/11/18 Chadwick, John, DCA <john.chadwick@state.nm.us>
I am noticing some errors in the koha-zebradaemon-output.log file. When new records are added it takes a bit longer to index than we think they should. Running rebuild zebra is often faster.
The log file shows the following:
14:31:44-18/11 zebrasrv(55) [warn] Failed to open /var/lib/bbmkoha/zebradb/biblios/register/reci-0.mf [Permission denied]
14:31:44-18/11 zebrasrv(55) [fatal] mf_write: reci error (1)
14:31:44-18/11 zebrasrv(55) [fatal] write commit block
14:31:44-18/11 zebrasrv(55) [warn] zebra_commit: failed
Zebra ends up indexing and search works, but I am concerned about the errors. Any ideas?
OK, baseline question: does /var/lib/bbmkoha/zebradb/biblios/register/reci-0.mf exist? What are the owner:group and permissions?
--joe
________________________________
This inbound email has been scanned for malicious software and transmitted safely to you using Webroot Email Security.
________________________________
Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System.
________________________________ Confidentiality Notice: This e-mail,including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review,use,disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. This email has been scanned using Webroot Email Security. ________________________________ _______________________________________________ Koha mailing list Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________ Koha mailing list Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
participants (4)
-
Biblioteca Casa Balthasar -
Chadwick, John, DCA -
Joe Atzberger -
Nicole Engard