We are currently setting up a server and have a question about the location of the zebra indexes. Is /var/lib/koha/instancename/biblios/ the only place zebra keeps the index files it writes to during the indexing process? We are considering mounting the index files to a separate high-performance disk array in hopes of improving performance during the indexing process -- has anyone experimented with something like this? Thanks! -- Chad Roseburg Automation Dept. North Central Regional Library
There is also the temporary directory. It usually is on /tmp, but you can explictly set it using -d (for rebuild_zebra.pl). If you are using 3.18, keep in mind the availability of the indexing daemon on packages seutp. Regards Tomas On Thu, Feb 19, 2015 at 8:14 PM, Chad Roseburg <croseburg@ncrl.org> wrote:
We are currently setting up a server and have a question about the location of the zebra indexes. Is /var/lib/koha/instancename/biblios/ the only place zebra keeps the index files it writes to during the indexing process?
We are considering mounting the index files to a separate high-performance disk array in hopes of improving performance during the indexing process -- has anyone experimented with something like this?
Thanks!
-- Chad Roseburg Automation Dept. North Central Regional Library _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
-- Tomás Cohen Arazi Prosecretaría de Informática Universidad Nacional de Córdoba ✆ +54 351 5353750 ext 13168 GPG: B76C 6E7C 2D80 551A C765 E225 0A27 2EA1 B2F3 C15F
Chad Roseburg schreef op do 19-02-2015 om 15:14 [-0800]:
We are currently setting up a server and have a question about the location of the zebra indexes. Is /var/lib/koha/instancename/biblios/ the only place zebra keeps the index files it writes to during the indexing process?
It puts temporary files in /tmp during the indexing process, and authority indices in /var/lib/koha/instancename/authorities/
We are considering mounting the index files to a separate high-performance disk array in hopes of improving performance during the indexing process -- has anyone experimented with something like this?
It'd be worth seeing where the slowdown in the indexing actually is: is the exporting the slow part, or the merging into the index? Though, I've never tried with zebra indicies on a high speed disk, I'd be interested to see some real benchmarks. Also, more RAM might be useful for disk caching, and you could experiment with disk fsync() commit options (i.e. by making it not require data to be written back before returning.) This has risk though, but if you're careful, it could well be acceptable, for example just needing to a full reindex in case of a power outage. -- Robin Sheat Catalyst IT Ltd. ✆ +64 4 803 2204 GPG: 5FA7 4B49 1E4D CAA4 4C38 8505 77F5 B724 F871 3BDF
At 01:26 PM 2/20/2015 +1300, Robin Sheat wrote:
Chad Roseburg schreef op do 19-02-2015 om 15:14 [-0800]:
We are considering mounting the index files to a separate high-performance disk array in hopes of improving performance during the indexing process -- has anyone experimented with something like this?
It'd be worth seeing where the slowdown in the indexing actually is: is the exporting the slow part, or the merging into the index?
Though, I've never tried with zebra indicies on a high speed disk, I'd be interested to see some real benchmarks.
Not absolute, but... our latest server uses SSD (raided 240 Gig Kingstons) rather than HDD (raided 500 Gig Seagates) with very slightly faster 8-core CPU, same 18 Gigs RAM; and the "read speed" (counting by 100's) of a total re-index is about twice as fast, with authorities faster than biblios. However, the "Cleaning" stage is disappointingly about the same speed, maybe a tad faster but barely perceptible. Memcached enabled on both boxes, same version of Zebra. I get the *impression* that Zebra sorts (or does some other process) during the final write stage, otherwise we'd have seen an improvement. However, we rarely use the full re-index, having got the Cron job fully sorted every minute (including minor tweaking of biblio-zebra-indexdefs.xsl) -- and that is as near instantaneous as anyone could wish. Best -- Paul
Also, more RAM might be useful for disk caching, and you could experiment with disk fsync() commit options (i.e. by making it not require data to be written back before returning.) This has risk though, but if you're careful, it could well be acceptable, for example just needing to a full reindex in case of a power outage.
-- Robin Sheat Catalyst IT Ltd. â +64 4 803 2204 GPG: 5FA7 4B49 1E4D CAA4 4C38 8505 77F5 B724 F871 3BDF
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>
Paul, you can use the indexing daemon with the packages (a.k.a. rebuild_zebra.pl in daemon mode) and set it to 5 seconds if you wish! Please read /etc/default/koha-common :-D On Thu, Feb 19, 2015 at 9:48 PM, Paul A <paul.a@navalmarinearchive.com> wrote:
At 01:26 PM 2/20/2015 +1300, Robin Sheat wrote:
We are considering mounting the index files to a separate high-performance disk array in hopes of improving performance during the indexing
Chad Roseburg schreef op do 19-02-2015 om 15:14 [-0800]: process --
has anyone experimented with something like this?
It'd be worth seeing where the slowdown in the indexing actually is: is the exporting the slow part, or the merging into the index?
Though, I've never tried with zebra indicies on a high speed disk, I'd be interested to see some real benchmarks.
Not absolute, but... our latest server uses SSD (raided 240 Gig Kingstons) rather than HDD (raided 500 Gig Seagates) with very slightly faster 8-core CPU, same 18 Gigs RAM; and the "read speed" (counting by 100's) of a total re-index is about twice as fast, with authorities faster than biblios. However, the "Cleaning" stage is disappointingly about the same speed, maybe a tad faster but barely perceptible. Memcached enabled on both boxes, same version of Zebra. I get the *impression* that Zebra sorts (or does some other process) during the final write stage, otherwise we'd have seen an improvement.
However, we rarely use the full re-index, having got the Cron job fully sorted every minute (including minor tweaking of biblio-zebra-indexdefs.xsl) -- and that is as near instantaneous as anyone could wish.
Best -- Paul
Also, more RAM might be
useful for disk caching, and you could experiment with disk fsync() commit options (i.e. by making it not require data to be written back before returning.) This has risk though, but if you're careful, it could well be acceptable, for example just needing to a full reindex in case of a power outage.
-- Robin Sheat Catalyst IT Ltd. ✆ +64 4 803 2204 GPG: 5FA7 4B49 1E4D CAA4 4C38 8505 77F5 B724 F871 3BDF
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
-- Tomás Cohen Arazi Prosecretaría de Informática Universidad Nacional de Córdoba ✆ +54 351 5353750 ext 13168 GPG: B76C 6E7C 2D80 551A C765 E225 0A27 2EA1 B2F3 C15F
Thanks for the info! Our servers have SSDs and plenty of RAM. So it sounds like possibly putting the index files in tmp on a separate disk might add some performance improvement ...though the SSDs by themselves will probably be the biggest improvement. On Thu, Feb 19, 2015 at 4:48 PM, Paul A <paul.a@navalmarinearchive.com> wrote:
At 01:26 PM 2/20/2015 +1300, Robin Sheat wrote:
We are considering mounting the index files to a separate high-performance disk array in hopes of improving performance during the indexing
Chad Roseburg schreef op do 19-02-2015 om 15:14 [-0800]: process --
has anyone experimented with something like this?
It'd be worth seeing where the slowdown in the indexing actually is: is the exporting the slow part, or the merging into the index?
Though, I've never tried with zebra indicies on a high speed disk, I'd be interested to see some real benchmarks.
Not absolute, but... our latest server uses SSD (raided 240 Gig Kingstons) rather than HDD (raided 500 Gig Seagates) with very slightly faster 8-core CPU, same 18 Gigs RAM; and the "read speed" (counting by 100's) of a total re-index is about twice as fast, with authorities faster than biblios. However, the "Cleaning" stage is disappointingly about the same speed, maybe a tad faster but barely perceptible. Memcached enabled on both boxes, same version of Zebra. I get the *impression* that Zebra sorts (or does some other process) during the final write stage, otherwise we'd have seen an improvement.
However, we rarely use the full re-index, having got the Cron job fully sorted every minute (including minor tweaking of biblio-zebra-indexdefs.xsl) -- and that is as near instantaneous as anyone could wish.
Best -- Paul
Also, more RAM might be
useful for disk caching, and you could experiment with disk fsync() commit options (i.e. by making it not require data to be written back before returning.) This has risk though, but if you're careful, it could well be acceptable, for example just needing to a full reindex in case of a power outage.
-- Robin Sheat Catalyst IT Ltd. ✆ +64 4 803 2204 GPG: 5FA7 4B49 1E4D CAA4 4C38 8505 77F5 B724 F871 3BDF
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
-- Chad Roseburg Automation Dept. North Central Regional Library
participants (4)
-
Chad Roseburg -
Paul A -
Robin Sheat -
Tomas Cohen Arazi