[Koha] Background job / Staging MARC import stuck at 0%
Cindy Murdock Ames
cmurdock at ccfls.org
Thu Jun 1 06:47:43 NZST 2023
Hi Anna,
Are you able to check your server for zombie processes? (Hint: run "ps aux
|grep Z" from the command line or else run "top" and look at the end of the
Tasks line for the number of zombies.)
Since upgrading to 22.11 from 22.05 I've had an issue where the
background_jobs_worker.pl tasks would turn into zombie processes. If all
the background job worker tasks become zombies then imports won't get
processed and it remains stuck at the "import queued" stage. I've been
unable to identify the cause or a proper fix, but for the time being I've
made a hacky cron script that runs every minute checking for these zombies
and killing their parent process, which seems to restart the queue, but you
have to retry the upload (the original job remains in the queue with a
progress of "null"). Sorry I don't have a proper solution, I was just
curious if others were seeing this symptom.
Cindy
-----------------------------------------------------------
Cindy Murdock Ames
IT Services Director
Meadville Public Library | CCFLS
https://meadvillelibrary.org | https://ccfls.org
On Mon, May 29, 2023 at 11:06 PM Anna Shields <apcshields at gmail.com> wrote:
> Michael,
>
> Did you ever have any luck resolving this?
>
> I'm having a similar issue in which my imports get stuck at 0% on the
> staging step. I, too, seem to have a functioning RabbitMQ, and none of the
> suggestions I saw in this thread seemed to make a difference.
>
> I'm now on v22.11.06-3, and updating hasn't seemed to make any difference,
> either.
>
> Anna
>
> On Tue, May 16, 2023 at 7:01 PM <koha-request at lists.katipo.co.nz> wrote:
>
> > Hi Michael,
> >
> > Did you also restart some services after changing the script including
> > memcache?
> >
> > Sometimes it is necessary but not remember for this case.
> >
> > You should also look
> > https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33412
> >
> > Mengü Yazıcıoğlu
> > Devinim Yazılım Eğitim Danışmanlık
> > Reşit Galip Cad. No: 29/6
> > Çankaya/ANKARA
> >
> > Tel: (312) 446 86 68
> > Cep: (532) 701 90 27
> >
> > On 13.05.2023 16:21, Michael Brown wrote:
> > > I tried Mengü's suggestion of editing manage-marc-import.pl, to no
> > avail.
> > > Still hangs at enqueuement; no messages, no errors, simply hangs and
> > spins
> > > my CPU into a frenzy. Responses to George's message below:
> > >
> > > On Wed, May 10, 2023 at 8:34 AM George Veranis <gveranis at dataly.gr>
> > wrote:
> > >
> > >> Hi Michael,
> > >>
> > >> Could you check the following things:
> > >>
> > >> 1) if your rabbitmq-server is running ?
> > >>
> > > Yes.
> > >
> > >
> > >> 1.1) if yes, then check if the cluster of nodes in rabbit exist , the
> > >> command is:
> > >> rabbitmqctl cluster_status
> > >>
> > > Cluster status of node rabbit at kiko ...
> > > Basics
> > >
> > > Cluster name: rabbit at kiko
> > >
> > > Disk Nodes
> > > rabbit at kiko
> > >
> > > Running Nodes
> > > rabbit at kiko
> > >
> > > Versions
> > > rabbit at kiko: RabbitMQ 3.9.21 on Erlang 24.3.4.2
> > >
> > > Maintenance status
> > >
> > > Node: rabbit at kiko, status: not under maintenance
> > >
> > > Alarms
> > >
> > > (none)
> > >
> > > Network Partitions
> > >
> > > (none)
> > >
> > > Listeners
> > >
> > > Node: rabbit at kiko, interface: [::], port: 61613, protocol: stomp,
> > purpose:
> > > STOMP
> > > Node: rabbit at kiko, interface: [::], port: 25672, protocol: clustering,
> > > purpose: inter-node and CLI tool communication
> > > Node: rabbit at kiko, interface: [::], port: 5672, protocol: amqp,
> purpose:
> > > AMQP 0-9-1 and AMQP 1.0
> > >
> > > Feature flags
> > >
> > > Flag: implicit_default_bindings, state: enabled
> > > Flag: maintenance_mode_status, state: enabled
> > > Flag: quorum_queue, state: enabled
> > > Flag: stream_queue, state: enabled
> > > Flag: user_limits, state: enabled
> > > Flag: virtual_host_metadata, state: enabled
> > >
> > >
> > >
> > >> 1.1.1) if not shows you something in json format you can try to start
> it
> > >> with command
> > >> rabbitmqctl start_app
> > >>
> > >> N/A, rabbitmq is already running.
> > >
> > >> 1.2 ) If no , start it and check the logs from your rabbitmq-server at
> > >> /var/log/rabbitmq/ and check if the service is start it normally .
> > >>
> > >> N/A, rabbitmq is already running.
> > >
> > >> 2) check the worker-output.log on your koha if koha write there
> > something
> > >> and have koha connection with rabbit server .
> > >>
> > >> I cannot find a file called "worker-output.log" anywhere in my Alma
> > > installation.
> > >
> > >
> > >> Best Regards,
> > >> George
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Koha <koha-bounces at lists.katipo.co.nz> On Behalf Of Michael
> Brown
> > >> Sent: Wednesday, May 10, 2023 3:50 PM
> > >> To: Cindy Murdock Ames <cmurdock at ccfls.org>
> > >> Cc: koha at lists.katipo.co.nz
> > >> Subject: Re: [Koha] Background job / Staging MARC import stuck at 0%
> > >>
> > >> Hi Cindy, apologies for the double-response: I had unsubscribed from
> > the
> > >> koha list so my response didn't get added to the thread. I wanted to
> > remedy
> > >> that, and also say that, because I had unsubscribed, I missed Mengü's
> > >> suggestion of updating manage-marc-import.pl. I am eager to try that
> > fix
> > >> and will report back later today or tomorrow with my findings. Here's
> my
> > >> original response to Cindy before re-subscribing to to the koha list:
> > >>
> > >> Yes, my koha-conf.xml does contain that block because I added it
> during
> > >> the initial setup. (You're right, it wasn't there originally.)
> > >>
> > >> However, it doesn't help; I still cannot import batch bib records in
> > Koha
> > >> 22.11 the way I could in Koha 22.05. The enqueuement hangs and sends
> my
> > >> cpu through the roof until I navigate away from that page. I've gotten
> > one
> > >> off-list suggestion that upgrading to the newest Koha fixes the
> problem.
> > >> Unfortunately for me, I don't really have time for that right now, so
> I
> > >> have (at least temporarily) abandoned my efforts with Koha.
> > >>
> > >> Thanks for your suggestion!
> > >>
> > >> Bests,
> > >> Michael
> > >>
> > >> On Tue, May 9, 2023 at 4:40 PM Michael Brown <michael8rown at gmail.com>
> > >> wrote:
> > >>
> > >>> Hi Cindy,
> > >>>
> > >>> Yes, my koha-conf.xml does contain that block because I added it
> > >>> during the initial setup. (You're right, it wasn't there originally.)
> > >>>
> > >>> However, it doesn't help; I still cannot import batch bib records in
> > >>> Koha
> > >>> 22.11 the way I could in Koha 22.05. The enqueuement hangs and sends
> > >>> my cpu through the roof until I navigate away from that page. I've
> > >>> gotten one off-list suggestion that upgrading to the newest Koha
> fixes
> > >> the problem.
> > >>> Unfortunately for me, I don't really have time for that right now, so
> > >>> I have (at least temporarily) abandoned my efforts with Koha.
> > >>>
> > >>> Thanks for your suggestion!
> > >>>
> > >>> Bests,
> > >>> Michael
> > >>>
> > >>> On Tue, May 9, 2023 at 1:37 PM Cindy Murdock Ames <
> cmurdock at ccfls.org>
> > >>> wrote:
> > >>>
> > >>>> Hi Michael,
> > >>>>
> > >>>> Can you confirm your koha-conf.xml contains a message broker
> section,
> > >>>> like this?
> > >>>>
> > >>>> <message_broker>
> > >>>> <hostname>localhost</hostname>
> > >>>> <port>61613</port>
> > >>>> <username>guest</username>
> > >>>> <password>guest</password>
> > >>>> <vhost></vhost>
> > >>>> </message_broker>
> > >>>>
> > >>>> I had the same problem after upgrading from 22.05 to 22.11; I'm
> > >>>> guessing
> > >>>> 22.05 must have had some defaults set in the code that allowed the
> > >>>> message broker to start automatically without this section whereas
> > >> 22.11 must not.
> > >>>> I was missing this section and added it and was able to complete
> > >>>> imports after restarting Koha.
> > >>>>
> > >>>> I'm still having the issue with zombie background jobs (I'm afraid
> to
> > >>>> test rolling back the code on a production server) but I made a
> hacky
> > >>>> bash script to run as a cron job to check for them and kill them off
> > >>>> so at least our staff can do batch imports without issue, but that's
> > >> another matter.
> > >>>> HTH!
> > >>>> Cindy
> > >>>>
> > >>>>
> > >>>> -----------------------------------------------------------
> > >>>> Cindy Murdock Ames
> > >>>> IT Services Director
> > >>>> Meadville Public Library | CCFLS
> > >>>> https://meadvillelibrary.org | https://ccfls.org
> > >>>>
> > >>>> Please report tech support issues in Mantis:
> > >>>> https://mantis.ccfls.org
> > >>>>
> > >>>>
> > >>>> On Thu, Mar 2, 2023 at 8:51 AM Michael Brown <
> michael8rown at gmail.com>
> > >>>> wrote:
> > >>>>
> > >>>>> Greetings:
> > >>>>>
> > >>>>> My name is Michael Brown and I am a professional cataloger and
> > >>>>> SirsiDynix System Admin at the Texas State Library & Archives in
> > >>>>> Austin (20+ years now). I have been using Koha on Arch Linux in my
> > >>>>> home library for about a year now. I am migrating my home server to
> > >>>>> AlmaLinux and I'm having a problem.
> > >>>>>
> > >>>>> I am running Koha 22.11.03.000 Rosalie on AlmaLinux 9.1. Staging a
> > >>>>> MARC file for import gets stuck at 0%. On screen, I am able to
> > >>>>> select the file for import (bib.mrc), review the profile options
> > >>>>> (but I don't change any defaults), and then click on "Stage for
> > >>>>> import" at the bottom. Next screen
> > >>>>> reads:
> > >>>>>
> > >>>>> The job has been enqueued! It will be processed as soon as
> > >> possible.
> > >>>>> 0%
> > >>>>> View detail of the enqueued job
> > >>>>>
> > >>>>> After a few seconds, it changes to (and then hangs at):
> > >>>>>
> > >>>>> The job has been enqueued! It will be processed as soon as
> > >> possible.
> > >>>>> 0% Not started
> > >>>>> View detail of the enqueued job
> > >>>>>
> > >>>>> Clicking on "View detail of the enqueued job" I see:
> > >>>>>
> > >>>>> Details of job #22
> > >>>>>
> > >>>>> Job ID: 22
> > >>>>> Status: New
> > >>>>> Progress: 0 / 0
> > >>>>> Type: Staged MARC records for import
> > >>>>> Queued: 03/02/2023 05:42
> > >>>>> Started:
> > >>>>> Ended:
> > >>>>>
> > >>>>> Report
> > >>>>> Detailed messages
> > >>>>> Return to the job list
> > >>>>>
> > >>>>> The corresponding entry in mariadb is:
> > >>>>>
> > >>>>> id 22
> > >>>>> status new
> > >>>>> progress NULL
> > >>>>> size 0
> > >>>>> borrowernumber 1
> > >>>>> type stage_marc_for_import
> > >>>>> queue Name of the queue the job is sent to long_tasks data
> > >>>>> {"encoding":"UTF-8","comments":"","basket_id":null...
> > >>>>> context JSON-serialized context information for the job
> > >>>>> {"flags":1,"branch":"ALMA","interface":"intranet",...
> > >>>>> enqueued_on 2023-03-02 05:42:33
> > >>>>> started_on NULL
> > >>>>> ended_on NULL
> > >>>>>
> > >>>>> (If you need to see the full entries for "data" and "context",
> > >>>>> please let me know.)
> > >>>>>
> > >>>>> tmp, koha_upload, and lock directories have been tweaked and fine
> > >> tuned.
> > >>>>> I
> > >>>>> was getting early warnings about them not being set in
> koha-conf.xml
> > >>>>> so I created them (and set correct permissions) and I can see the
> > >>>>> uploaded file (for job 22 the name is
> > >>>>> 2287629673fb980ad4102f62ebeaa1b9_bib.mrc), so the actual upload
> > >>>>> function appears to be working.
> > >>>>>
> > >>>>> I am getting no apache errors and no other on-screen diagnostics.
> > >>>>>
> > >>>>> I have Koha 22.05.02.000 running on Arch Linux that imports this
> > >>>>> file just fine. Similarly, I have Koha latest running on a Debian
> VM
> > >>>>> that can import this file just fine, too.
> > >>>>>
> > >>>>> What am I missing?
> > >>>>>
> > >>>>> Details of my system:
> > >>>>>
> > >>>>> Koha version: 22.11.03.000 Rosalie
> > >>>>> OS version ('uname -a'): Linux alma 5.14.0-162.12.1.el9_1.x86_64 #1
> > >>>>> SMP PREEMPT_DYNAMIC Mon Jan 23 14:51:52 EST 2023 x86_64 Perl
> > >>>>> interpreter: /usr/bin/perl Perl version: 5.032001 Perl @INC:
> > >>>>> /usr/share/koha/lib
> > >>>>> /usr/local/lib64/perl5/5.32
> > >>>>> /usr/local/share/perl5/5.32
> > >>>>> /usr/lib64/perl5/vendor_perl
> > >>>>> /usr/share/perl5/vendor_perl
> > >>>>> /usr/lib64/perl5
> > >>>>> /usr/share/perl5
> > >>>>> /var/lib/koha/plugins
> > >>>>> MySQL version: mysql Ver 15.1 Distrib 10.5.16-MariaDB, for Linux
> > >>>>> (x86_64) using EditLine wrapper Apache version: Server version:
> > >>>>> Apache/2.4.53 (AlmaLinux) Server built:
> > >>>>> Jul
> > >>>>> 20 2022 00:00:00
> > >>>>> Memcached: Servers: 127.0.0.1:11211 | Namespace: KOHA | Status:
> > >>>>> running. |
> > >>>>> Config read from: koha-conf.xml
> > >>>>> Zebra version: Zebra 2.2.7 (C) 1994-2023, Index Data Zebra is free
> > >>>>> software, covered by the GNU General Public License, and you are
> > >>>>> welcome to change it and/or distribute copies of it under certain
> > >>>>> conditions. SHA1
> > >>>>> ID:
> > >>>>> ac40f289672405a299436d73c1532f9906774cc6 Using ICU Zebra status:
> > >>>>> Running Message broker: Using RabbitMQ Date and time: 03/01/2023
> > >>>>> 16:20 Time zone: Used: America/Chicago | Config: Undefined |
> > >>>>> Environment (TZ):
> > >>>>> Undefined
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Michael
> > >>>>> _______________________________________________
> > >>>>>
> > >>>>> Koha mailing list http://koha-community.org
> Koha at lists.katipo.co.nz
> > >>>>> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
> > >>>>>
> > >> _______________________________________________
> > >>
> > >> Koha mailing list http://koha-community.org Koha at lists.katipo.co.nz
> > >> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
> > >>
> > >>
> > > _______________________________________________
> > >
> > > Koha mailing list http://koha-community.org
> > > Koha at lists.katipo.co.nz
> > > Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
> >
> >
> > ------------------------------
> >
> > Subject: Digest Footer
> >
> > _______________________________________________
> > Koha mailing list
> > Koha at lists.katipo.co.nz
> > https://lists.katipo.co.nz/mailman/listinfo/koha
> >
> >
> > ------------------------------
> >
> > End of Koha Digest, Vol 211, Issue 15
> > *************************************
> >
> _______________________________________________
>
> Koha mailing list http://koha-community.org
> Koha at lists.katipo.co.nz
> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
>
More information about the Koha
mailing list