Introduction / Problem with bulk import
Dear Koha-community! - First i want to apologize that my English is a little bit rusty ... - Second, i want to introduce myself: My name ist Markus Becker, living in Duisburg, Germany. On the one hand i am head librarian of the Ministry of Home Affairs Northrine-Westfalia in Düsseldorf, on other hand i am a student of ,,Bibliotheksinformatik'' (Library Computer Science) at the Wildau Institute of Technology. For my Master Thesis i want to take a deeper look on Koha, trying to proof, that Koha is an possible LMS for the governmental libraries in Northrine-Westfalia. - Third, i have a problem with the bulk-import of MARC21XML-Records to Koha and i hope, someone can give me a little hint (technical details of the Koha-installation below): When i try to import the MARC21XML-File with the bulkmarcimport.pl via console (following http://kohageek.blogspot.de/2016/05/bulk-marc-import-using-command-line.html), i get the following error:
..Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 397, <GEN46> chunk 5.
2 MARC records done in 0.16786003112793 seconds
The 2 MARC records were *not* imported. The error is occurring with MARC21XML-files created with my own conversion-script and with files exported from Koha itself. Any hints for me? Thank You very much in advance, Markus Technical details of my Koha-installation: (Running in a Virtual Box) Koha-Version: 17.11.00.000 OS Version ('uname -a'): Linux debian 4.9.0-4-amd64 #1 SMP Debian 4.9.65-3+deb9u1 (2017-12-23) x86_64 Perl Interpreter: /usr/bin/perl Perl-Version: 5.024001 Perl @INC: /usr/share/koha/lib /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base MySQL-Version: mysql Ver 15.1 Distrib 10.1.26-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2 Apache-Version: Server version: Apache/2.4.25 (Debian) Zebra 2.0.59 (C) 1994-2014, Index Data Zebra is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. SHA1 ID: c00bfddbf0f3608340d61298acc61dafb167f9b2 Using ICU
Willkommen, Markus! On Fri, May 25, 2018 at 09:04:56AM +0200, Markus Becker wrote:
- Third, i have a problem with the bulk-import of MARC21XML-Records to Koha and i hope, someone can give me a little hint (technical details of the Koha-installation below):
When i try to import the MARC21XML-File with the bulkmarcimport.pl via console (following http://kohageek.blogspot.de/2016/05/bulk-marc-import-using-command-line.html), i get the following error:
..Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 397, <GEN46> chunk 5.
2 MARC records done in 0.16786003112793 seconds
The 2 MARC records were *not* imported.
The error is occurring with MARC21XML-files created with my own conversion-script and with files exported from Koha itself.
Any hints for me?
It sounds as though the file you are trying to import is not a well-formed XML document; it may be truncated in the middle of a record. Can you post it for us to look at? What XML library does your conversion script use? Alternatively, if you have installed libexpat you can use the xmlwf command to check that the file is well-formed. Or try xmllint if you have installed libxml. For example: xmlwf FILE xmlwf -n FILE xmllint --noout FILE Paul. -- Paul Hoffman <paul@flo.org> Systems Librarian Fenway Libraries Online c/o Wentworth Institute of Technology 550 Huntington Ave. Boston, MA 02115 (617) 442-2384 (FLO main number)
Hello Paul, 2018-05-25 17:30 GMT+02:00 Paul Hoffman <paul@flo.org>: [...]
Any hints for me?
It sounds as though the file you are trying to import is not a well-formed XML document; it may be truncated in the middle of a record. Can you post it for us to look at? What XML library does your conversion script use?
It uses XML::Writer.
Alternatively, if you have installed libexpat you can use the xmlwf command to check that the file is well-formed. Or try xmllint if you have installed libxml. For example:
xmlwf FILE xmlwf -n FILE xmllint --noout FILE
I will try this out! Thank You very much! Yours, Markus
Markus, First, welcome to the Koha community. There's also a German mailing list, Katrin or Mirko can point you in the right direction if you're more comfortable _mit Deutsch_ although your English seems quite good. In terms of the error message that you're seeing, MARC::File::XML isn't actually part of the Koha distribution, which makes it difficult to tell what's happening just from the error message itself. I can't remember if this mailing list allows attachments -- you can try attaching the marcxml files as long as they're not too large, otherwise upload them somewhere public and send us a link. We'll also need the command line arguments that you used. I would also suggest running bulkmarcimport.pl with the -v2 flag enabled -- this will give you verbose output, which might be useful for trouble-shooting. Cheers, Gruesse, usw. --Barton On Fri, May 25, 2018 at 3:04 AM, Markus Becker <beckmarkus@gmail.com> wrote:
Dear Koha-community!
- First i want to apologize that my English is a little bit rusty ...
- Second, i want to introduce myself:
My name ist Markus Becker, living in Duisburg, Germany. On the one hand i am head librarian of the Ministry of Home Affairs Northrine-Westfalia in Düsseldorf, on other hand i am a student of ,,Bibliotheksinformatik'' (Library Computer Science) at the Wildau Institute of Technology.
For my Master Thesis i want to take a deeper look on Koha, trying to proof, that Koha is an possible LMS for the governmental libraries in Northrine-Westfalia.
- Third, i have a problem with the bulk-import of MARC21XML-Records to Koha and i hope, someone can give me a little hint (technical details of the Koha-installation below):
When i try to import the MARC21XML-File with the bulkmarcimport.pl via console (following http://kohageek.blogspot.de/2016/05/bulk-marc-import- using-command-line.html), i get the following error:
..Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 397, <GEN46> chunk 5.
2 MARC records done in 0.16786003112793 seconds
The 2 MARC records were *not* imported.
The error is occurring with MARC21XML-files created with my own conversion-script and with files exported from Koha itself.
Any hints for me?
Thank You very much in advance, Markus
Technical details of my Koha-installation: (Running in a Virtual Box) Koha-Version: 17.11.00.000 OS Version ('uname -a'): Linux debian 4.9.0-4-amd64 #1 SMP Debian 4.9.65-3+deb9u1 (2017-12-23) x86_64 Perl Interpreter: /usr/bin/perl Perl-Version: 5.024001 Perl @INC: /usr/share/koha/lib /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base MySQL-Version: mysql Ver 15.1 Distrib 10.1.26-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2 Apache-Version: Server version: Apache/2.4.25 (Debian) Zebra 2.0.59 (C) 1994-2014, Index Data Zebra is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. SHA1 ID: c00bfddbf0f3608340d61298acc61dafb167f9b2 Using ICU _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
On Fri, May 25, 2018 at 11:36 AM, Barton Chittenden < barton@bywatersolutions.com> wrote:
In terms of the error message that you're seeing, MARC::File::XML isn't actually part of the Koha distribution, which makes it difficult to tell what's happening just from the error message itself.
To be clear, this is maintained by Galen Charlton -- he's a member of both the Koha and Evergreen communities; you can see the documentation for the module here: https://metacpan.org/pod/MARC::File::XML
Hi! Uploaded my MARC21XML-Files in my Dropbox. Here is the URL: https://www.dropbox.com/sh/0kr0hgebies4val/AADdKfPeOr-reITDq8q9UhZfa?dl=0 The file ,,atlas-roh.marcxml'' is dowloaded from the DNB, ,,hotten.marcxml'' ist exportet by Koha itself and ,,output.xml'' is created by my conversion-script. All these files cause the mentioned error - it seems to me, that the error is caused by some encoding-trouble ... Greetings from the humid Ruhr Valley, Markus 2018-05-25 17:36 GMT+02:00 Barton Chittenden <barton@bywatersolutions.com>:
Markus,
First, welcome to the Koha community. There's also a German mailing list, Katrin or Mirko can point you in the right direction if you're more comfortable _mit Deutsch_ although your English seems quite good.
In terms of the error message that you're seeing, MARC::File::XML isn't actually part of the Koha distribution, which makes it difficult to tell what's happening just from the error message itself.
I can't remember if this mailing list allows attachments -- you can try attaching the marcxml files as long as they're not too large, otherwise upload them somewhere public and send us a link. We'll also need the command line arguments that you used.
I would also suggest running bulkmarcimport.pl with the -v2 flag enabled -- this will give you verbose output, which might be useful for trouble-shooting.
Cheers, Gruesse, usw.
--Barton
On Fri, May 25, 2018 at 3:04 AM, Markus Becker <beckmarkus@gmail.com> wrote:
Dear Koha-community!
- First i want to apologize that my English is a little bit rusty ...
- Second, i want to introduce myself:
My name ist Markus Becker, living in Duisburg, Germany. On the one hand i am head librarian of the Ministry of Home Affairs Northrine-Westfalia in Düsseldorf, on other hand i am a student of ,,Bibliotheksinformatik'' (Library Computer Science) at the Wildau Institute of Technology.
For my Master Thesis i want to take a deeper look on Koha, trying to proof, that Koha is an possible LMS for the governmental libraries in Northrine-Westfalia.
- Third, i have a problem with the bulk-import of MARC21XML-Records to Koha and i hope, someone can give me a little hint (technical details of the Koha-installation below):
When i try to import the MARC21XML-File with the bulkmarcimport.pl via console (following
http://kohageek.blogspot.de/2016/05/bulk-marc-import-using-command-line.html), i get the following error:
..Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 397, <GEN46> chunk 5.
2 MARC records done in 0.16786003112793 seconds
The 2 MARC records were *not* imported.
The error is occurring with MARC21XML-files created with my own conversion-script and with files exported from Koha itself.
Any hints for me?
Thank You very much in advance, Markus
Technical details of my Koha-installation: (Running in a Virtual Box) Koha-Version: 17.11.00.000 OS Version ('uname -a'): Linux debian 4.9.0-4-amd64 #1 SMP Debian 4.9.65-3+deb9u1 (2017-12-23) x86_64 Perl Interpreter: /usr/bin/perl Perl-Version: 5.024001 Perl @INC: /usr/share/koha/lib /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base MySQL-Version: mysql Ver 15.1 Distrib 10.1.26-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2 Apache-Version: Server version: Apache/2.4.25 (Debian) Zebra 2.0.59 (C) 1994-2014, Index Data Zebra is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. SHA1 ID: c00bfddbf0f3608340d61298acc61dafb167f9b2 Using ICU _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
Hi Markus On May 25 you worte
The 2 MARC records were *not* imported.
Today you wrote:
Uploaded my MARC21XML-Files in my Dropbox. Here is the URL: https://www.dropbox.com/sh/0kr0hgebies4val/AADdKfPeOr-reITDq8q9UhZfa?dl=0
The file ,,atlas-roh.marcxml'' is dowloaded from the DNB, ,,hotten.marcxml'' ist exportet by Koha itself and ,,output.xml'' is created by my conversion-script.
All these files cause the mentioned error - it seems to me, that the error is caused by some encoding-trouble ...
I have downloaded your three files and loaded them into my Koha 18.05 demo installation, using "bulkmarcimport.pl" (see the screen output at the end). However, all records imported correctly (as far as I can see) and can be found in the demo installation (but only until tomorrow about 5 AM when my demo installation is reset): http://koha.adminkuhn.ch:8080/cgi-bin/koha/catalogue/detail.pl?biblionumber=... http://koha.adminkuhn.ch:8080/cgi-bin/koha/catalogue/detail.pl?biblionumber=... http://koha.adminkuhn.ch:8080/cgi-bin/koha/catalogue/detail.pl?biblionumber=... http://koha.adminkuhn.ch:8080/cgi-bin/koha/catalogue/detail.pl?biblionumber=... Of course there was an error: Item not added (bib 542, item tag #1, barcode 00802006481): invalid homebranch MAS To prevent this you need to create that missing homebranch MAS (which i didn't know about) in Koha menu "Administration > Libraries" before loading the MARC records. Indeed there was also the error you mentioned, but to me it seems it's just a warning that doesn't mean much: Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 397, <GEN46> chunk 5. This message is also mentioned here: * https://github.com/LibreCat/Catmandu-MARC/issues/22 * https://rt.cpan.org/Public/Bug/Display.html?id=111473 Hope this helps. Best wishes: Michael -- Geschäftsführer · Diplombibliothekar BBS, Informatiker eidg. Fachausweis Admin Kuhn GmbH · Pappelstrasse 20 · 4123 Allschwil · Schweiz T 0041 (0)61 261 55 61 · E mik@adminkuhn.ch · W www.adminkuhn.ch # INSTANZ=instanzname # export KOHA_CONF=/etc/koha/sites/$INSTANZ/koha-conf.xml # /usr/share/koha/bin/migration_tools/bulkmarcimport.pl -b -m marcxml -file /tmp/becker/atlas_roh.marcxml -v 2 Characteristic MARC flavour: MARC21 LDR 00416nam0a2200121 i 4500 999 _c539 _d539 001 0000401 003 OSt 005 20180305195910.0 008 17 628 |||||||||||||||||||||||||||| || 040 _cOSt 245 10 _aAtlas : _bÖstliches Ruhrgebiet; Karte + Luftbild. Maßstab 1 : 20 000 / _cHrsg. vom Kommunalverband Ruhrgebie 250 _aAusg. 199 260 _aEssen _bKommunalverband Ruhrgebiet _c1999 650 _aLuftbild _xKartenwerk _zRuhrgebiet _zNordrhein-Westfale 942 _2z _cBK Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 397, <GEN46> chunk 2. 1 MARC records done in 0.157258033752441 seconds # /usr/share/koha/bin/migration_tools/bulkmarcimport.pl -b -m marcxml -file /tmp/becker/hotten.marcxml -v 2 Characteristic MARC flavour: MARC21 LDR 02178pam a2200553 c 4500 999 _c540 _d540 001 1061000028 003 DE-101 005 20180524142526.0 007 tu 008 141117s2015 gw ||||| |||| 00||||ger 015 _a15,A25 _z14,N48 _2dnb 016 7 _2DE-101 _a1061000028 020 _a9783498064358 _cPp. : EUR 19.95 (DE), EUR 20.60 (AT), sfr 28.50 (freier Pr.) 020 _a3498064355 024 3 _a9783498064358 035 _a(DE-599)DNB1061000028 035 _a(OCoLC)897128560 040 _a1245 _bger _cDE-101 _d9999 041 _ager 044 _cXA-DE-SH 082 04 _81\x _a070.92 _qDE-101 _222/ger 083 7 _a070 _a791 _qDE-101 _223sdnb 084 _a070 _a791 _qDE-101 _2sdnb 085 _81\x _b070.92 090 _ab 100 1 _0(DE-588)121165574 _0http://d-nb.info/gnd/121165574 _0(DE-101)121165574 _aSchneider, Wolf _d1925- _4aut 245 00 _aHottentottenstottertrottel _bmein langes, wunderliches Leben _cWolf Schneider 250 _a1. Aufl. 264 1 _aReinbek bei Hamburg _bRowohlt _c2015 300 _a447 S. _bIll. _c22 cm 500 _aLiteraturangaben 600 17 _0(DE-588)121165574 _0http://d-nb.info/gnd/121165574 _0(DE-101)121165574 _aSchneider, Wolf _d1925- _2gnd 653 _a(Produktform)Hardback 653 _aAutobiographie 653 _aDeutsche Zeitgeschichte 653 _aJournalismus 653 _aJournalistenschule 653 _aNDR Talkshow 653 _a(VLB-WN)1951: Hardcover, Softcover / Sachbücher/Kunst, Literatur/Biographien, Autobiographien 655 7 _aAutobiographie _2gnd 776 08 _iOnline-Ausg. _aSchneider, Wolf, 1925- _tHottentottenstottertrottel _dReinbek : Rowohlt E-Book, 2015 _hOnline-Ressource _w(DE-101)1073868982 850 _aDE-101a _aDE-101b 856 42 _mB:DE-101 _qapplication/pdf _uhttp://d-nb.info/1061000028/04 _3Inhaltsverzeichnis 856 42 _mX:MVB _qtext/html _uhttp://deposit.d-nb.de/cgi-bin/dokserv?id=4826460&prov=M&dok_var=1&dok_ext=htm _3Inhaltstext 942 _2ddc _cMONO Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 397, <GEN46> chunk 2. 1 MARC records done in 0.226579189300537 seconds # /usr/share/koha/bin/migration_tools/bulkmarcimport.pl -b -m marcxml -file /tmp/becker/output.xml -v 2 Characteristic MARC flavour: MARC21 Item not added (bib 541, item tag #1, barcode ): invalid homebranch MAS LDR 00000pam a2200000 c 4500 003 MAS 001 0000194 245 _aFlug über das Ruhrgebiet _bein Bilderbuch _cFotos : Corneel Voigt; Texte : Wolfgang Schulze ; Hrsg. : Kommunalverband Ruhrgebiet 250 _a4. Aufl. 260 _c1993 700 _aVoigt, Corneel 700 _aSchulze, Wolfgang 700 _aMustermann, Max 952 _aMAS _i94/269 _oBe 009 / 4 _t0000194001 999 _c541 _d541 Item not added (bib 542, item tag #1, barcode 00802006481 ): invalid homebranch MAS LDR 00000pam a2200000 c 4500 003 MAS 001 9052982 100 _aKnapp, Radek 245 _aGebrauchsanweisung für Polen _cRadek Knapp 250 _a3. Aufl. 260 _c2006 952 _aMAS _iP2006/005 _oP2006/005 _p00802006481 _t9052982001 999 _c542 _d542 Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 397, <GEN46> chunk 5. 2 MARC records done in 0.612886905670166 seconds
Hi Michael! 2018-05-27 23:05 GMT+02:00 Michael Kuhn <mik@adminkuhn.ch>: [...]
Of course there was an error:
Item not added (bib 542, item tag #1, barcode 00802006481): invalid homebranch MAS
To prevent this you need to create that missing homebranch MAS (which i didn't know about) in Koha menu "Administration > Libraries" before loading the MARC records.
I thougt that "homebranch" is our good old ,,Zweigstelle'' and it is in the local-copy-record in my source-file. So my converting-script puts the "MAS" in MARC21-Field 952a. For testing purpose i deleted 952a in "my" MARC21XML-File and now the records are imported! I am very happy now!! Thank You very much!! Greetings, Markus
participants (5)
-
Barton Chittenden -
Markus Becker -
Markus Becker -
Michael Kuhn -
Paul Hoffman