[Koha] Elasticsearch error stops bulkmarcimport

[Bales (US), Tasha R]

I've occasionally seen this error when loading MARC record on the command line using bulkmarcimport.pl:
Bad response received when submitting request to Elasticsearch.

When it occurs, the bulkmarcimport ceases, interrupting the load.

I've found nothing else in any other logs on the Koha server that help explain the error.  I'm not sure what Elasticsearch has to do with my bulkmarcimport, unless it is Elasticsearch trying to index records on-the-fly after they are loaded?  Is there an easy way to prevent this error from happening?  Perhaps I should stop the Elasticsearch process while doing a bulkmarcimport? My Elasticsearch servers are external to the Koha environment, and I don't see any obvious indexing processes running on my local machine, so I'm not sure how best to intervene to prevent the above issue.

I'm currently working my way through loading 1.3 million records, and I've had several redo's already as a result of loads being interrupted partway through (running bulkmarcimport via a cronjob helped *a lot*, but now I've had two ES errors/interruptions in the last day, which has put me a day behind).  I am currently working with batches of 200K records or less.

Thanks for any tips you may have!  In the meantime, I will modify my latest MARC file to try to remove records already loaded and then resubmit the job (in order to save additional hours spent reloading 98K records).


Tasha Bales
Enterprise Services
http://isesi.web.boeing.com/