[Koha] Background job / Staging MARC import stuck at 0%

Michael Brown michael8rown at gmail.com
Sat Mar 4 12:14:31 NZDT 2023 

Sorry, one more quick detail: the koha-conf.xml file that was generated
during installation set the following <lockdir> parameter:

     <lockdir>__LOCK_DIR__</lockdir>

This threw a warning that read:

     The configured <lockdir> entry in
     your koha-conf.xml file points
     to a non-writable directory (__LOCK_DIR__).

I took a wild guess and created a lockdir in the same folder with the
zebradb lockdir and with the same permissions:

     <lockdir>/var/lock/koha/alma</lockdir>

The warning is gone now, but I don't see any activity in that directory.
Maybe it's used for other purposes, but in the event that I have
misconfigured the lockdir setting, I wanted to mention it.

On Fri, Mar 3, 2023 at 4:48 PM Michael Brown <michael8rown at gmail.com> wrote:

> Thanks, Eric. Still no luck even though I downgraded several packages to
> the Koha required versions:
>
>      JSON::Validator => 5.08
>      Mojolicious => 8.12
>      Mojolicious::Plugin::OAuth2 => 2.02
>      Mojolicious::Plugin::OpenAPI => 5.05
>      Mojolicious::Plugin::RenderFile => 0.12
>
> background_jobs.data =
>
> {"parse_items":"1","marc_modification_template":null,"nomatch_action":"create_new","encoding":"UTF-8","record_type":"biblio","matcher_id":"","vendor_id":null,"basket_id":null,"filepath":"/var/lib/koha/alma/tmp/koha_upload/b63453b06ba1fa2ca63a76e64e4909bb_l.mrc","comments":"","filename":"l.mrc","item_action":"always_add","format":"ISO2709","overlay_action":"replace"}
>
> Still nothing being logged, but app.pl is really lighting up the CPU:
>
> top - 16:38:47 up 5 min,  1 user,  load average: 1.32, 1.04, 0.50
> Tasks: 237 total,   2 running, 235 sleeping,   0 stopped,   0 zombie
> %Cpu(s): 26.3 us,  1.9 sy,  0.0 ni, 71.5 id,  0.0 wa,  0.2 hi,  0.1 si,
>  0.0 st
> MiB Mem :   5827.8 total,   2597.2 free,   2334.9 used,   1492.9 buff/cache
> MiB Swap:   1331.0 total,   1331.0 free,      0.0 used.   3492.9 avail Mem
>
>     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
> COMMAND
>    3936 apache    20   0  127888 121296  19480 R  99.3   2.0   0:04.01
> app.pl
>    1592 michael   20   0 4325576 206032 112492 S   3.3   3.5   0:15.64
> gnome-shell
>    1179 michael   20   0  521988  92176  57912 S   3.0   1.5   0:13.63
> Xorg
>    2217 michael   20   0  691664  50664  38408 S   1.3   0.8   0:03.49
> gnome-terminal-
>    3599 michael   20   0  226528   4204   3332 R   0.7   0.1   0:00.48 top
>
>     783 memcach+  20   0  418036  11400   4360 S   0.3   0.2   0:00.12
> memcached
>     786 rabbitmq  20   0 2848520 127696  65352 S   0.3   2.1   0:09.08
> beam.smp
>
> It will remain at about that level until I close the window.
>
> Michael
>
> On Fri, Mar 3, 2023 at 12:56 PM Eric Phetteplace <ephetteplace at cca.edu>
> wrote:
>
>> I would absolutely think that the wrong version of JSON::Validator could
>> cause a problem here. Try to match Koha's versions in the cpanfile,
>> probably `cpanm --installdeps .` from the root but maybe that's not wise
>> with a package installation.
>>
>> Best,
>>
>> ERIC PHETTEPLACE Systems Librarian, Libraries (he/him)
>>
>> ephetteplace at cca.edu
>>
>>
>> *CCA is situated on the traditional unceded lands of the **Chochenyo and
>> Ramaytush Ohlone** peoples.*
>>
>> Black-owned bookstores in Oakland: Ashay by the Bay
>> <https://ashaybythebay.com/>, Marcus Books
>> <https://www.facebook.com/marcus.books/>
>>
>> :(){ :|: & };:
>>
>>
>> On Fri, Mar 3, 2023 at 5:58 AM Michael Brown <michael8rown at gmail.com>
>> wrote:
>>
>>> Hi Jonathan,
>>>
>>> Thanks for the suggestion; nothing much to report, though.
>>>
>>>     tail -f /var/log/koha/*log
>>>
>>> showed nothing at all when I staged "l.mrc" this morning.
>>>
>>> As for the background_job.data suggestion, it took me a minute to
>>> understand what you were referring to and by the time I figured it out I
>>> had already left home, so I cannot access my server right now. However, I
>>> did copy the background_job.data entry for one of the uploads I had
>>> performed yesterday. Here it is:
>>>
>>>
>>> {"encoding":"UTF-8","comments":"","basket_id":null,"record_type":"biblio","vendor_id":null,"matcher_id":"","parse_items":"1","item_action":"always_add","format":"ISO2709","marc_modification_template":null,"filepath":"/var/lib/koha/alma/tmp/koha_upload/2287629673fb980ad4102f62ebeaa1b9_bib.mrc","nomatch_action":"create_new","filename":"bib.mrc","overlay_action":"replace"}
>>>
>>> Is that helpful? If not, I will pull the entry from the test I made this
>>> morning while watching the logs.
>>>
>>> When I was first starting out with Koha (on Arch Linux about a year ago)
>>> I
>>> was having issues that I eventually traced to JSON Validator and
>>> Mojolicious perl modules. They were both much newer than those required
>>> by
>>> Koha, and when I downgraded those two modules, things started working.
>>> But
>>> my problem back then was not with uploads; it was with the display of
>>> borrowers. Patrons were not loading on-screen and an Inspect > Console
>>> message gave me the idea to downgrade. It worked after that.
>>>
>>> Could this possibly be a similar problem? Should I try force-downgrading
>>> JSON::Validator and/or Mojolicious? Is there any debugging I can do of
>>> the
>>> Koha::REST::V1 module? As I reported yesterday, app.pl shows
>>> continual-constant CPU usage as long as that enqueuement page is open.
>>> Once
>>> I navigate away from that enqueuement page, the app.pl process quits,
>>> CPU
>>> usage returns to normal.
>>>
>>> Michael
>>>
>>> On Fri, Mar 3, 2023 at 12:55 AM Jonathan Druart <
>>> jonathan.druart at bugs.koha-community.org> wrote:
>>>
>>> > Hello Michael,
>>> > I have just tried on 22.11.03 and it seems to work for me.
>>> > Can you try to open a console, watch the logs: tail -f
>>> > /var/log/koha/$KOHA_INSTANCE/*.log
>>> > Import a file.
>>> > Do you see something in the logs?
>>> >
>>> > Otherwise look at the background_jobs.data JSON, you may see an error
>>> in
>>> > it. But if the status is still "new" I am not expecting an error there.
>>> >
>>> >
>>> > Le jeu. 2 mars 2023 à 14:50, Michael Brown <michael8rown at gmail.com> a
>>> > écrit :
>>> >
>>> > > Greetings:
>>> > >
>>> > > My name is Michael Brown and I am a professional cataloger and
>>> SirsiDynix
>>> > > System Admin at the Texas State Library & Archives in Austin (20+
>>> years
>>> > > now). I have been using Koha on Arch Linux in my home library for
>>> about a
>>> > > year now. I am migrating my home server to AlmaLinux and I'm having a
>>> > > problem.
>>> > >
>>> > > I am running Koha 22.11.03.000 Rosalie on AlmaLinux 9.1. Staging a
>>> MARC
>>> > > file for import gets stuck at 0%. On screen, I am able to select the
>>> file
>>> > > for import (bib.mrc), review the profile options (but I don't change
>>> any
>>> > > defaults), and then click on "Stage for import" at the bottom. Next
>>> > screen
>>> > > reads:
>>> > >
>>> > >     The job has been enqueued! It will be processed as soon as
>>> possible.
>>> > >     0%
>>> > >     View detail of the enqueued job
>>> > >
>>> > > After a few seconds, it changes to (and then hangs at):
>>> > >
>>> > >     The job has been enqueued! It will be processed as soon as
>>> possible.
>>> > >     0% Not started
>>> > >     View detail of the enqueued job
>>> > >
>>> > > Clicking on "View detail of the enqueued job" I see:
>>> > >
>>> > > Details of job #22
>>> > >
>>> > >     Job ID: 22
>>> > >     Status:    New
>>> > >     Progress:    0 / 0
>>> > >     Type:    Staged MARC records for import
>>> > >     Queued:    03/02/2023 05:42
>>> > >     Started:
>>> > >     Ended:
>>> > >
>>> > > Report
>>> > > Detailed messages
>>> > > Return to the job list
>>> > >
>>> > > The corresponding entry in mariadb is:
>>> > >
>>> > > id 22
>>> > > status new
>>> > > progress NULL
>>> > > size 0
>>> > > borrowernumber 1
>>> > > type stage_marc_for_import
>>> > > queue Name of the queue the job is sent to long_tasks
>>> > > data {"encoding":"UTF-8","comments":"","basket_id":null...
>>> > > context JSON-serialized context information for the job
>>> > > {"flags":1,"branch":"ALMA","interface":"intranet",...
>>> > > enqueued_on 2023-03-02 05:42:33
>>> > > started_on NULL
>>> > > ended_on NULL
>>> > >
>>> > > (If you need to see the full entries for "data" and "context",
>>> please let
>>> > > me know.)
>>> > >
>>> > > tmp, koha_upload, and lock directories have been tweaked and fine
>>> tuned.
>>> > I
>>> > > was getting early warnings about them not being set in koha-conf.xml
>>> so I
>>> > > created them (and set correct permissions) and I can see the uploaded
>>> > file
>>> > > (for job 22 the name is 2287629673fb980ad4102f62ebeaa1b9_bib.mrc),
>>> so the
>>> > > actual upload function appears to be working.
>>> > >
>>> > > I am getting no apache errors and no other on-screen diagnostics.
>>> > >
>>> > > I have Koha 22.05.02.000 running on Arch Linux that imports this file
>>> > just
>>> > > fine. Similarly, I have Koha latest running on a Debian VM that can
>>> > import
>>> > > this file just fine, too.
>>> > >
>>> > > What am I missing?
>>> > >
>>> > > Details of my system:
>>> > >
>>> > > Koha version: 22.11.03.000 Rosalie
>>> > > OS version ('uname -a'): Linux alma 5.14.0-162.12.1.el9_1.x86_64 #1
>>> SMP
>>> > > PREEMPT_DYNAMIC Mon Jan 23 14:51:52 EST 2023 x86_64
>>> > > Perl interpreter: /usr/bin/perl
>>> > > Perl version: 5.032001
>>> > > Perl @INC: /usr/share/koha/lib
>>> > > /usr/local/lib64/perl5/5.32
>>> > > /usr/local/share/perl5/5.32
>>> > > /usr/lib64/perl5/vendor_perl
>>> > > /usr/share/perl5/vendor_perl
>>> > > /usr/lib64/perl5
>>> > > /usr/share/perl5
>>> > > /var/lib/koha/plugins
>>> > > MySQL version: mysql Ver 15.1 Distrib 10.5.16-MariaDB, for Linux
>>> (x86_64)
>>> > > using EditLine wrapper
>>> > > Apache version: Server version: Apache/2.4.53 (AlmaLinux) Server
>>> built:
>>> > Jul
>>> > > 20 2022 00:00:00
>>> > > Memcached: Servers: 127.0.0.1:11211 | Namespace: KOHA | Status:
>>> > running. |
>>> > > Config read from: koha-conf.xml
>>> > > Zebra version: Zebra 2.2.7 (C) 1994-2023, Index Data Zebra is free
>>> > > software, covered by the GNU General Public License, and you are
>>> welcome
>>> > to
>>> > > change it and/or distribute copies of it under certain conditions.
>>> SHA1
>>> > ID:
>>> > > ac40f289672405a299436d73c1532f9906774cc6 Using ICU
>>> > > Zebra status: Running
>>> > > Message broker: Using RabbitMQ
>>> > > Date and time: 03/01/2023 16:20
>>> > > Time zone: Used: America/Chicago | Config: Undefined | Environment
>>> (TZ):
>>> > > Undefined
>>> > >
>>> > > Thanks,
>>> > > Michael
>>> > > _______________________________________________
>>> > >
>>> > > Koha mailing list  http://koha-community.org
>>> > > Koha at lists.katipo.co.nz
>>> > > Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
>>> > >
>>> > _______________________________________________
>>> >
>>> > Koha mailing list  http://koha-community.org
>>> > Koha at lists.katipo.co.nz
>>> > Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
>>> >
>>> _______________________________________________
>>>
>>> Koha mailing list  http://koha-community.org
>>> Koha at lists.katipo.co.nz
>>> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
>>>
>>

More information about the Koha mailing list