Re: [Koha] Zebra rebuild frequency
At 04:20 PM 11/14/2014 +0000, Tomas Cohen Arazi wrote:
Paul, I'm sorry you suffered that on your Koha, but I've never seen it. I'm only aware of a situation that could trigger that on NORMARC (recently fixed) using DOM indexing. [snip]
3.8.5 (production) and from my test notes 3.12.various and 3.14.2 required a complete biblio re-index after deleting a biblio record. Has this been fixed?
I don't think that was ever true. [snip]
It is true on our production box (and its backup; sandbox in other use) Koha: 3.08.05.000 on Linux 3.2.0-31-generic (12.04LTS), Perl version: 5.014002, MySQL version: mysql 5.5.24, Apache version: 2.2.22, Zebra: 2.0.44 Pls see <http://opac.navalmarinearchive.com/cgi-bin/koha/opac-search.pl?q=kent+beyond+reef>, two biblios for Kent, "Beyond the reef": one is good, the other has been deleted, gives a 404 on the OPAC page, but will not go away from Zebra by cron job alone (it will go away next time I CLI re-index.) I have a feeling that when zebra says "incremental" it means just that -- adding new data, but not deleting what has gone from MySQL. Slightly different case, <http://opac.navalmarinearchive.com/cgi-bin/koha/opac-search.pl?q=steele+worcester> -- this was produced by "overwriting" an existing biblio (old, incomplete) with a new Z39.50, and adding an item -- it's the same biblio, but the "dead" instance will not disappear until I re-index. This is a further example of the (for us) need to CLI re-index as the cronjob does not handle it. Best -- Paul --- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>
Greetings, And what cronjobs do you currently have? Perhaps you don't even have your partial reindex cronjob installed? GPML, Mark Tompsett
On 2014-11-15, at 9:49 AM, Paul A wrote:
At 04:20 PM 11/14/2014 +0000, Tomas Cohen Arazi wrote:
Paul, I'm sorry you suffered that on your Koha, but I've never seen it. I'm only aware of a situation that could trigger that on NORMARC (recently fixed) using DOM indexing. [snip]
3.8.5 (production) and from my test notes 3.12.various and 3.14.2 required a complete biblio re-index after deleting a biblio record. Has this been fixed?
I don't think that was ever true. [snip]
It is true on our production box (and its backup; sandbox in other use) Koha: 3.08.05.000 on Linux 3.2.0-31-generic (12.04LTS), Perl version: 5.014002, MySQL version: mysql 5.5.24, Apache version: 2.2.22, Zebra: 2.0.44
Pls see <http://opac.navalmarinearchive.com/cgi-bin/koha/opac-search.pl?q=kent+beyond+reef>, two biblios for Kent, "Beyond the reef": one is good, the other has been deleted, gives a 404 on the OPAC page, but will not go away from Zebra by cron job alone (it will go away next time I CLI re-index.) I have a feeling that when zebra says "incremental" it means just that -- adding new data, but not deleting what has gone from MySQL.
Slightly different case, <http://opac.navalmarinearchive.com/cgi-bin/koha/opac-search.pl?q=steele+worcester> -- this was produced by "overwriting" an existing biblio (old, incomplete) with a new Z39.50, and adding an item -- it's the same biblio, but the "dead" instance will not disappear until I re-index.
This is a further example of the (for us) need to CLI re-index as the cronjob does not handle it.
you should show an example of your cron file - its probably just misconfigured?
At 12:53 PM 11/15/2014 +1300, Mason James wrote:
On 2014-11-15, at 9:49 AM, Paul A wrote:
At 04:20 PM 11/14/2014 +0000, Tomas Cohen Arazi wrote:
Paul, I'm sorry you suffered that on your Koha, but I've never seen it. I'm only aware of a situation that could trigger that on NORMARC (recently fixed) using DOM indexing. [snip]
3.8.5 (production) and from my test notes 3.12.various and 3.14.2 required a complete biblio re-index after deleting a biblio record. Has this been fixed?
I don't think that was ever true. [snip]
It is true on our production box (and its backup; sandbox in other use) Koha: 3.08.05.000 on Linux 3.2.0-31-generic (12.04LTS), Perl version: 5.014002, MySQL version: mysql 5.5.24, Apache version: 2.2.22, Zebra: 2.0.44
Pls see <http://opac.navalmarinearchive.com/cgi-bin/koha/opac-search.pl?q=kent+beyond+reef>, two biblios for Kent, "Beyond the reef": one is good, the other has been deleted, gives a 404 on the OPAC page, but will not go away from Zebra by cron job alone (it will go away next time I CLI re-index.) I have a feeling that when zebra says "incremental" it means just that -- adding new data, but not deleting what has gone from MySQL.
Slightly different case, <http://opac.navalmarinearchive.com/cgi-bin/koha/opac-search.pl?q=steele+worcester> -- this was produced by "overwriting" an existing biblio (old, incomplete) with a new Z39.50, and adding an item -- it's the same biblio, but the "dead" instance will not disappear until I re-index.
This is a further example of the (for us) need to CLI re-index as the cronjob does not handle it.
you should show an example of your cron file - its probably just misconfigured?
It's been running every minute for two years ;=} using KOHA_CONF=/etc/koha/koha-conf.xml KOHAPATH=/usr/share/koha PERL5LIB=$KOHAPATH/lib */1 * * * * koha $KOHAPATH/bin/migration_tools/rebuild_zebra.pl -a -b -z > /dev/null 2>&1 paul@nelson:/$ grep CRON /var/log/syslog [snip]Nov 14 21:22:01 nelson CRON[23795]: (koha) CMD (/usr/share/koha/bin/migration_tools/rebuild_zebra.pl -z -b -a >> /tmp/cron_koha.log 2>&1) Nov 14 21:23:01 nelson CRON[23842]: (koha) CMD (/usr/share/koha/bin/migration_tools/rebuild_zebra.pl -z -b -a >> /tmp/cron_koha.log 2>&1) Nov 14 21:24:01 nelson CRON[23930]: (koha) CMD (/usr/share/koha/bin/migration_tools/rebuild_zebra.pl -z -b -a >> /tmp/cron_koha.log 2>&1) Nov 14 21:25:01 nelson CRON[24054]: (koha) CMD (/usr/share/koha/bin/migration_tools/rebuild_zebra.pl -z -b -a >> /tmp/cron_koha.log 2>&1) Nov 14 21:26:01 nelson CRON[24139]: (koha) CMD (/usr/share/koha/bin/migration_tools/rebuild_zebra.pl -z -b -a >> /tmp/cron_koha.log 2>&1) Nov 14 21:27:01 nelson CRON[24182]: (koha) CMD (/usr/share/koha/bin/migration_tools/rebuild_zebra.pl -z -b -a >> /tmp/cron_koha.log 2>&1) Nov 14 21:28:02 nelson CRON[24310]: (koha) CMD (/usr/share/koha/bin/migration_tools/rebuild_zebra.pl -z -b -a >> /tmp/cron_koha.log 2>&1) Best -- Paul --- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>
On 2014-11-15, at 1:53 PM, Paul A wrote:
At 12:53 PM 11/15/2014 +1300, Mason James wrote:
On 2014-11-15, at 9:49 AM, Paul A wrote:
At 04:20 PM 11/14/2014 +0000, Tomas Cohen Arazi wrote:
Paul, I'm sorry you suffered that on your Koha, but I've never seen it. I'm only aware of a situation that could trigger that on NORMARC (recently fixed) using DOM indexing. [snip]
> 3.8.5 (production) and from my test notes 3.12.various and 3.14.2 required a > complete biblio re-index after deleting a biblio record. Has this been > fixed?
I don't think that was ever true. [snip]
It is true on our production box (and its backup; sandbox in other use) Koha: 3.08.05.000 on Linux 3.2.0-31-generic (12.04LTS), Perl version: 5.014002, MySQL version: mysql 5.5.24, Apache version: 2.2.22, Zebra: 2.0.44
Pls see <http://opac.navalmarinearchive.com/cgi-bin/koha/opac-search.pl?q=kent+beyond+reef>, two biblios for Kent, "Beyond the reef": one is good, the other has been deleted, gives a 404 on the OPAC page, but will not go away from Zebra by cron job alone (it will go away next time I CLI re-index.) I have a feeling that when zebra says "incremental" it means just that -- adding new data, but not deleting what has gone from MySQL.
Slightly different case, <http://opac.navalmarinearchive.com/cgi-bin/koha/opac-search.pl?q=steele+worcester> -- this was produced by "overwriting" an existing biblio (old, incomplete) with a new Z39.50, and adding an item -- it's the same biblio, but the "dead" instance will not disappear until I re-index.
This is a further example of the (for us) need to CLI re-index as the cronjob does not handle it.
you should show an example of your cron file - its probably just misconfigured?
It's been running every minute for two years ;=} using
KOHA_CONF=/etc/koha/koha-conf.xml KOHAPATH=/usr/share/koha PERL5LIB=$KOHAPATH/lib */1 * * * * koha $KOHAPATH/bin/migration_tools/rebuild_zebra.pl -a -b -z > /dev/null 2>&1
paul@nelson:/$ grep CRON /var/log/syslog
[snip]Nov 14 21:22:01 nelson CRON[23795]: (koha) CMD (/usr/share/koha/bin/migration_tools/rebuild_zebra.pl -z -b -a >> /tmp/cron_koha.log 2>&1) Nov 14 21:23:01 nelson CRON[23842]: (koha) CMD (/usr/share/koha/bin/migration_tools/rebuild_zebra.pl -z -b -a >> /tmp/cron_koha.log 2>&1) Nov 14 21:24:01 nelson CRON[23930]: (koha) CMD (/usr/share/koha/bin/migration_tools/rebuild_zebra.pl -z -b -a >> /tmp/cron_koha.log 2>&1) Nov 14 21:25:01 nelson CRON[24054]: (koha) CMD (/usr/share/koha/bin/migration_tools/rebuild_zebra.pl -z -b -a >> /tmp/cron_koha.log 2>&1) Nov 14 21:26:01 nelson CRON[24139]: (koha) CMD (/usr/share/koha/bin/migration_tools/rebuild_zebra.pl -z -b -a >> /tmp/cron_koha.log 2>&1) Nov 14 21:27:01 nelson CRON[24182]: (koha) CMD (/usr/share/koha/bin/migration_tools/rebuild_zebra.pl -z -b -a >> /tmp/cron_koha.log 2>&1) Nov 14 21:28:02 nelson CRON[24310]: (koha) CMD (/usr/share/koha/bin/migration_tools/rebuild_zebra.pl -z -b -a >> /tmp/cron_koha.log 2>&1)
Best -- Paul
try using the -v arg, it will show you what it is actually doing $ /usr/share/koha/bin/migration_tools/rebuild_zebra.pl -a -b -z -v Zebra configuration information ================================ Zebra biblio directory = /var/lib/koha/mastcaro/biblios Zebra authorities directory = /var/lib/koha/mastcaro/authorities Koha directory = /usr/share/koha Lockfile = /var/lock/zebra_koha_mastcaro/rebuild/rebuild..LCK BIBLIONUMBER in : 999$c BIBLIOITEMNUMBER in : 999$d ================================ ==================== exporting authority ==================== Records exported: 0 Records exported: 0 ==================== REINDEXING zebra ==================== ==================== exporting biblio ==================== 1. Records exported: 1 <<<<< DELETED BIB, REMOVED FROM ZEBRA Records exported: 0 ==================== REINDEXING zebra ==================== ==================== CLEANING ==================== $
Paul A schreef op vr 14-11-2014 om 19:53 [-0500]:
It's been running every minute for two years ;=} using
Under older versions, there was a chance of two zebra reindexes crashing into each other, and I think that if this happened, it could be possible that a change would get lost. Running every minute would increase the chances of that happening. Newer versions (3.16 I think) lock the reindexing process so two runs can't collide. -- Robin Sheat Catalyst IT Ltd. ✆ +64 4 803 2204 GPG: 5FA7 4B49 1E4D CAA4 4C38 8505 77F5 B724 F871 3BDF
Paul, could you confirm that you use Zebra in DOM mode? It seems Zebra isn't configured properly, and so doesn't know where to pick up unique ID for biblio records (biblionumber field). Without an ID to identify biblio records, Zebra can't neither delete nor update a biblio record. Check your Zebra index definition file: biblio-koha-indexdefs.xml. At the beginning of the file, you should have a line like this one: <id>marc:datafield[@tag='999']/marc:subfield[@code='c']</id> My assumption is that you haven't this. This could happen if the upgrade isn't done properly, for a reason or another.
I have a feeling that when zebra says "incremental" it means just that -- adding new data, but not deleting what has gone from MySQL.
It works for the vast majority of the Koha crowd. Zebra is obviously designed to be able to add/edit/delete records. It's a minimum :-) You have to figure out what's wrong with your installation. Kind regards, --
Paul's XSLT was using 001 for the id. El sáb, nov 15, 2014 05:39 AM, Frédéric Demians <frederic@tamil.fr> escribió:
Paul, could you confirm that you use Zebra in DOM mode? It seems Zebra isn't configured properly, and so doesn't know where to pick up unique ID for biblio records (biblionumber field). Without an ID to identify biblio records, Zebra can't neither delete nor update a biblio record.
Check your Zebra index definition file: biblio-koha-indexdefs.xml.
At the beginning of the file, you should have a line like this one:
<id>marc:datafield[@tag='999']/marc:subfield[@code='c']</id>
My assumption is that you haven't this. This could happen if the upgrade isn't done properly, for a reason or another.
I have a feeling that when zebra says "incremental" it means just that -- adding new data, but not deleting what has gone from MySQL.
It works for the vast majority of the Koha crowd. Zebra is obviously designed to be able to add/edit/delete records. It's a minimum :-) You have to figure out what's wrong with your installation.
Kind regards, -- _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
participants (6)
-
Frédéric Demians -
Mark Tompsett -
Mason James -
Paul A -
Robin Sheat -
Tomas Cohen Arazi