Help testing marc staging performance issues
Hi, All! At ByWater, we've had a few reports of "Home › Tools › Stage MARC records for import" timing out on large import batches, with the message "there was an error." When this happens, there are typically two batches under Home › Tools › Manage staged MARC records The first batch will show a number of records, and the second batch will show 0 records. I believe this typically happens under plack. I'm having a *very* difficult time re-creating the error, so I don't have any plack error logs, I don't have any good numbers on how many records cause the problem. One of our libraries told me that the problem seems to be worse when using matching rules. If you've seen issues like this, I would love to know 1) What version of Koha you're using 2) How many records you were importing 3) If you were doing any sort of record matching 4) Any other relevant details 5) If you have access to the server error logs, I'd like to know what messages you see there. I'd like to file a bug, but without being able to replicate the problem, I can't get any traction there. Thanks, --Barton
Hi Barton,
I believe this typically happens under plack.
These scripts should not run under plack, check your apache config file (search for "ProxyPass" in apache-shared-intranet-plack.conf) Cheers, Jonathan On Wed, 19 Sep 2018 at 19:17 Barton Chittenden <barton@bywatersolutions.com> wrote:
Hi, All!
At ByWater, we've had a few reports of
"Home › Tools › Stage MARC records for import" timing out on large import batches, with the message "there was an error."
When this happens, there are typically two batches under
Home › Tools › Manage staged MARC records
The first batch will show a number of records, and the second batch will show 0 records.
I believe this typically happens under plack. I'm having a *very* difficult time re-creating the error, so I don't have any plack error logs, I don't have any good numbers on how many records cause the problem.
One of our libraries told me that the problem seems to be worse when using matching rules.
If you've seen issues like this, I would love to know
1) What version of Koha you're using 2) How many records you were importing 3) If you were doing any sort of record matching 4) Any other relevant details 5) If you have access to the server error logs, I'd like to know what messages you see there.
I'd like to file a bug, but without being able to replicate the problem, I can't get any traction there.
Thanks,
--Barton _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
On Wed, Sep 19, 2018 at 6:29 PM, Jonathan Druart < jonathan.druart@bugs.koha-community.org> wrote:
Hi Barton,
I believe this typically happens under plack.
These scripts should not run under plack, check your apache config file (search for "ProxyPass" in apache-shared-intranet-plack.conf)
You're exactly right, those are excluded: ProxyPass "/cgi-bin/koha/tools/stage-marc-import.pl" "!" ProxyPass "/cgi-bin/koha/tools/upload-cover-image.pl" "!" All, please disregard what I said about plack, apparently my brain was playing tricks with me when I wrote that. The rest of the message stands: If you've seen errors importing large batches of Marc records and gotten the message "there was an error.", I'd like more information so that I can file a decent bug report. Thanks, --Barton
We should really do a two-pass processing, so the first thing is calculating the batch size. That way Ajax will not timeout. El mié., 19 de sep. de 2018 20:04, Barton Chittenden < barton@bywatersolutions.com> escribió:
On Wed, Sep 19, 2018 at 6:29 PM, Jonathan Druart < jonathan.druart@bugs.koha-community.org> wrote:
Hi Barton,
I believe this typically happens under plack.
These scripts should not run under plack, check your apache config file (search for "ProxyPass" in apache-shared-intranet-plack.conf)
You're exactly right, those are excluded:
ProxyPass "/cgi-bin/koha/tools/stage-marc-import.pl" "!" ProxyPass "/cgi-bin/koha/tools/upload-cover-image.pl" "!"
All, please disregard what I said about plack, apparently my brain was playing tricks with me when I wrote that. The rest of the message stands: If you've seen errors importing large batches of Marc records and gotten the message "there was an error.", I'd like more information so that I can file a decent bug report.
Thanks,
--Barton _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
-- Tomás Cohen Arazi Theke Solutions (https://theke.io <http://theke.io/>) ✆ +54 9351 3513384 GPG: B2F3C15F
Tomas, Do you know if there's a bug open for this? If not, would you mind filing one? Thanks, --Barton On Wed, Sep 19, 2018 at 7:57 PM, Tomas Cohen Arazi <tomascohen@gmail.com> wrote:
We should really do a two-pass processing, so the first thing is calculating the batch size. That way Ajax will not timeout.
El mié., 19 de sep. de 2018 20:04, Barton Chittenden < barton@bywatersolutions.com> escribió:
On Wed, Sep 19, 2018 at 6:29 PM, Jonathan Druart < jonathan.druart@bugs.koha-community.org> wrote:
Hi Barton,
I believe this typically happens under plack.
These scripts should not run under plack, check your apache config file (search for "ProxyPass" in apache-shared-intranet-plack.conf)
You're exactly right, those are excluded:
ProxyPass "/cgi-bin/koha/tools/stage-marc-import.pl" "!" ProxyPass "/cgi-bin/koha/tools/upload-cover-image.pl" "!"
All, please disregard what I said about plack, apparently my brain was playing tricks with me when I wrote that. The rest of the message stands: If you've seen errors importing large batches of Marc records and gotten the message "there was an error.", I'd like more information so that I can file a decent bug report.
Thanks,
--Barton _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
-- Tomás Cohen Arazi Theke Solutions (https://theke.io <http://theke.io/>) ✆ +54 9351 3513384 GPG: B2F3C15F
So, looking around, I see some bugs that seem to cover the same ground: Bug 17240 - "Allow processes that rely on background jobs run in Plack mode" Bug 15032 - [Plack] Scripts that fork (like stage-marc-import.pl) don't work as expected Bug 20342 - Fix table locking issue, progress bar not updating, and allow use of plack for Stage MARC Import On Thu, Sep 20, 2018 at 5:02 PM Barton Chittenden < barton@bywatersolutions.com> wrote:
Tomas,
Do you know if there's a bug open for this? If not, would you mind filing one?
Thanks,
--Barton
On Wed, Sep 19, 2018 at 7:57 PM, Tomas Cohen Arazi <tomascohen@gmail.com> wrote:
We should really do a two-pass processing, so the first thing is calculating the batch size. That way Ajax will not timeout.
El mié., 19 de sep. de 2018 20:04, Barton Chittenden < barton@bywatersolutions.com> escribió:
On Wed, Sep 19, 2018 at 6:29 PM, Jonathan Druart < jonathan.druart@bugs.koha-community.org> wrote:
Hi Barton,
I believe this typically happens under plack.
These scripts should not run under plack, check your apache config file (search for "ProxyPass" in apache-shared-intranet-plack.conf)
You're exactly right, those are excluded:
ProxyPass "/cgi-bin/koha/tools/stage-marc-import.pl" "!" ProxyPass "/cgi-bin/koha/tools/upload-cover-image.pl" "!"
All, please disregard what I said about plack, apparently my brain was playing tricks with me when I wrote that. The rest of the message stands: If you've seen errors importing large batches of Marc records and gotten the message "there was an error.", I'd like more information so that I can file a decent bug report.
Thanks,
--Barton _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
-- Tomás Cohen Arazi Theke Solutions (https://theke.io <http://theke.io/>) ✆ +54 9351 3513384 GPG: B2F3C15F
participants (3)
-
Barton Chittenden -
Jonathan Druart -
Tomas Cohen Arazi