Import staged MARC records with ElasticSearch engine
Hello all, - Importing MARC records into catalog form (Manage staged MARC records) in staff client GUI using ElasticSearch engine is very slow and accidentally stops after importing some hundreds of records, That we may spend 3 days to import about 26000 record only, Is that normal? or there are any other configurations should be done first with ElasticSearch engine? - We are using Koha 16.11.01. Thanks in advance.
Hi Yousef, How much RAM is there on your server and how many CPU cores available to you? While the speed of import and indexing does depend on the server config, 3 days is way too much. Regards Indranil Das Gupta L2C2 Technologies On Apr 17, 2017 1:35 PM, "Abdulsalam Yousef" <ayousef@kwareict.com> wrote:
Hello all,
- Importing MARC records into catalog form (Manage staged MARC records) in staff client GUI using ElasticSearch engine is very slow and accidentally stops after importing some hundreds of records, That we may spend 3 days to import about 26000 record only, Is that normal? or there are any other configurations should be done first with ElasticSearch engine?
- We are using Koha 16.11.01.
Thanks in advance. _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
Excerpts from Abdulsalam Yousef's message of 2017-04-17 11:04:58 +0300:
- Importing MARC records into catalog form (Manage staged MARC records) in staff client GUI using ElasticSearch engine is very slow and accidentally stops after importing some hundreds of records, That we may spend 3 days to import about 26000 record only, Is that normal? or there are any other configurations should be done first with ElasticSearch engine?
- We are using Koha 16.11.01.
We're not using Elastic Search on our installation, but I have found that attempting to import our entire catalog of 24000 MARC records failed due to Koha running out of memory (our system has 2 GB of RAM). I was forced to split the MARC file into smaller chunks of 2000 records, and import each chunk separately. A larger chunk might have worked but I didn't want to take any chances. I wrote a Ruby script to do the splitting of the MARC records, but perhaps other tools can do the job (maybe Marcedit, which I haven't used). The Ruby script is here: https://github.com/bloovis/marc/blob/master/splitmarc.rb
participants (3)
-
Abdulsalam Yousef -
Indranil Das Gupta -
Mark Alexander