Re: [Koha] Something new with bulkmarcimport.pl
Stephen, At 07:53 PM 1/5/06 -0500, you wrote:
Andro, I've forgotten what Koha version you are using.
I'm not at the museum as I write this. If I remember correctly, I'm using ver. 2.2.2.
The Biblio.pm line numbers you list don't match my (2.2.4) Biblio.pm.
In any case, I would guess that you have some MARC records in the file that have errors.
That's what I was thinking at first, of course. It looked like I was getting a steady stream of messages saying "DBD::mysql::db do failed: Access denied for user: 'myuserid@localhost' to database 'libcat' at /usr/local/koha/intranet/modules/C4/Biblio.pm line 331. [or 426.]" on the screen. It looked like there were hundreds of lines of these messages--so many that I could not scroll back to the first one. But when I searched for records, I found that there were quite a few records in the database. And this was the first time I had attempted importing MARC records. To confuse matters more, the last line of the script's output, "1572 MARC record done in 731.838590145111 seconds," made me wonder whether the script was telling me 1,572 records had been imported successfully or just telling me that 1,572 records had been processed, giving me no information about successful imports. As this first import of records was done without mapping item information in the MARC records to the Koha database's item field, I could not get a report of the number of items in the OPAC. I'm hoping I can use bulkmarcimport.pl to delete all of the current records in the OPAC database so I can properly map the item information tag(s) and re-import the records. If I remember correctly, bulkmarcimport.pl has a switch that will allow me to delete the records in the OPAC. If you have any advice for me on this point, I'd be very pleased to hear it.
There is another script in the intranet/scripts/misc directory called dumpmarc.pl that will simply list out your MARC file. But with 1500+ records, it wouldn't be much fun looking for stray errors!
I am familiar with it, as I used it to look over a number of records to make sure they looked like they were in good shape before trying to import them. However, as you can imagine, I did not examine all 1,572 of them. To resolve my issue with bulkmarcimport.pl, what I may need to do is to get a batch of only 10 records in a file, examine them in dumpmarc.pl, and then import them with bulkmarcimport.pl. If I get the same kind of screen output from bulkmarcimport.pl with those records, I will have a much more manageable situation to deal with.
You could also use this script to find the title of the last record in the file. Do a search on that title and the Biblionumber on the screen should equal the number of records in your catalog -- assuming you haven't yet been doing any deleting of records.
That's a very good suggestion, as I haven't deleted any of the imported records. Thanks. Best regards, Andro P. Gagné P.O. Box 715 Canandaigua, NY 14424
Andro et al, I get almost the same thing when running bulkmarcimport: DBD::mysql::db do failed: Access denied for user: 'kohaadmin@localhost' to database 'Koha' at /usr/local/koha/intranet/modules/C4/Biblio.pm line 333. DBD::mysql::db do failed: Access denied for user: 'kohaadmin@localhost' to database 'Koha' at /usr/local/koha/intranet/modules/C4/Biblio.pm line 382. This is with 2.2.4. I just assumed that it's normal, since it happens every time, and all of my 97,500+ records seem to be importing. (Phpmyadmin is handy for examining your database, BTW--if the number of records in the Biblio table is the same as your number of marc records, then AFAIK that means that they all were imported.) Should I be concerned with these errors? I already have item types defined. I think if you use the -d switch when running the script, it will delete all your old records. If you look at the script in an editor, or run it without any options (including '-file') it will tell you what all the options are. Cindy _________________________ Cindy Murdock Network Administrator Meadville Public Library | Crawford County Federated Library System meadvillelibrary.org | ccfls.org Andro Gagne wrote:
Stephen,
At 07:53 PM 1/5/06 -0500, you wrote:
Andro, I've forgotten what Koha version you are using.
I'm not at the museum as I write this. If I remember correctly, I'm using ver. 2.2.2.
The Biblio.pm line numbers you list don't match my (2.2.4) Biblio.pm.
In any case, I would guess that you have some MARC records in the file that have errors.
That's what I was thinking at first, of course. It looked like I was getting a steady stream of messages saying "DBD::mysql::db do failed: Access denied for user: 'myuserid@localhost' to database 'libcat' at /usr/local/koha/intranet/modules/C4/Biblio.pm line 331. [or 426.]" on the screen. It looked like there were hundreds of lines of these messages--so many that I could not scroll back to the first one. But when I searched for records, I found that there were quite a few records in the database. And this was the first time I had attempted importing MARC records. To confuse matters more, the last line of the script's output, "1572 MARC record done in 731.838590145111 seconds," made me wonder whether the script was telling me 1,572 records had been imported successfully or just telling me that 1,572 records had been processed, giving me no information about successful imports. As this first import of records was done without mapping item information in the MARC records to the Koha database's item field, I could not get a report of the number of items in the OPAC. I'm hoping I can use bulkmarcimport.pl to delete all of the current records in the OPAC database so I can properly map the item information tag(s) and re-import the records. If I remember correctly, bulkmarcimport.pl has a switch that will allow me to delete the records in the OPAC. If you have any advice for me on this point, I'd be very pleased to hear it.
There is another script in the intranet/scripts/misc directory called dumpmarc.pl that will simply list out your MARC file. But with 1500+ records, it wouldn't be much fun looking for stray errors!
I am familiar with it, as I used it to look over a number of records to make sure they looked like they were in good shape before trying to import them. However, as you can imagine, I did not examine all 1,572 of them. To resolve my issue with bulkmarcimport.pl, what I may need to do is to get a batch of only 10 records in a file, examine them in dumpmarc.pl, and then import them with bulkmarcimport.pl. If I get the same kind of screen output from bulkmarcimport.pl with those records, I will have a much more manageable situation to deal with.
You could also use this script to find the title of the last record in the file. Do a search on that title and the Biblionumber on the screen should equal the number of records in your catalog -- assuming you haven't yet been doing any deleting of records.
That's a very good suggestion, as I haven't deleted any of the imported records. Thanks.
Best regards,
Andro P. Gagné P.O. Box 715 Canandaigua, NY 14424
_______________________________________________ Koha mailing list Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
--
Cindy et al., At 02:06 PM 1/6/06 -0500, Cindy Murdock wrote:
Andro et al,
I get almost the same thing when running bulkmarcimport:
DBD::mysql::db do failed: Access denied for user: 'kohaadmin@localhost' to database 'Koha' at /usr/local/koha/intranet/modules/C4/Biblio.pm line 333. DBD::mysql::db do failed: Access denied for user: 'kohaadmin@localhost' to database 'Koha' at /usr/local/koha/intranet/modules/C4/Biblio.pm line 382.
This is with 2.2.4.
I just assumed that it's normal, since it happens every time, and all of my 97,500+ records seem to be importing. (Phpmyadmin is handy for examining your database, BTW--if the number of records in the Biblio table is the same as your number of marc records, then AFAIK that means that they all were imported.)
I'm actually using 2.2.2, and, as Stephen indicated, that's probably the reason the lines where the "failures" occurred for you were different from mine. I finally had a chance today to check the results of my first bulkmarcimport.pl import by using SQL to directly examine the biblio table, and, sure enough, all 1,572 records were there, and they appeared to be in perfect shape. Perhaps I should have done the SQL work before bothering the list with my questions, but I didn't have time to do this on the same day I did the import. Also, I found the bulkmarcimport.pl screen output particularly unhelpful and confusing, and I was wondering what others' experiences with this script were. If the "DBD::mysql::db do failed" messages do not indicate a malfunction of the import process, they really should not be displayed, it seems to me. Furthermore, when the screen output ends with something like "1572 MARC record done in 731.838590145111 seconds," it really should indicate that 1572 MARC records were _successfully_ imported. It would be even better if it gave the number of records successfully imported and the number that could not be imported, if possible.
Should I be concerned with these errors? I already have item types defined.
It looks to me like these messages have very little purpose and can be ignored. Can someone tell us otherwise?
I think if you use the -d switch when running the script, it will delete all your old records.
I was pretty sure that this was the case when I wrote my last message, but I wasn't at the museum--where my Koha machine is located--at the time I wrote it. Thanks for the confirmation. Thanks to everyone for helping this new user of Koha. Best regards, Andy (Andro P. Gagné) P.O. Box 715 Canandaigua, NY 14424
Cindy Murdock said:
I get almost the same thing when running bulkmarcimport:
DBD::mysql::db do failed: Access denied for user: 'kohaadmin@localhost' to database 'Koha' at /usr/local/koha/intranet/modules/C4/Biblio.pm line 333. DBD::mysql::db do failed: Access denied for user: 'kohaadmin@localhost' to database 'Koha' at /usr/local/koha/intranet/modules/C4/Biblio.pm line 382.
This is with 2.2.4.
Well, I have a partial answer to this problem, but am not smart enough to figure it all out. The lines that are throwing the error perform a "lock" on the MySQL tables so bulkmarcimport can have exclusive access while loading the records. Apparently, the username that gets passed from the script to the module does not have lock priviliges. The result is that MySQL throws an error that get's reported (as above), the records get loaded anyway, and if you have exclusive access to the Koha server, there won't be any MySQL conflicts with other users. I'll submit this to kohabugs (possibly related to bug 1002). -- Stephen Hedges Skemotah Solutions, USA www.skemotah.com -- shedges@skemotah.com
participants (3)
-
Andro Gagne -
Cindy Murdock -
Stephen Hedges