Re: [Koha] Best practices for koha - journal migration using marcedit
At 09:39 PM 4/28/2016 +0000, Karl Holten wrote:
The MARC format has an individual record size limit of 99999 bytes, but big serials in Koha can eclipse that size. The result is that really big serials don't compile as valid MARC if you have a ton of items attached to them.
Thanks for the comment. I think that Indranil Das Gupta was proposing a direct intervention at MySQL level -- that's certainly what I am looking for. I'm uncertain as to _where_ the limit of 99999 bytes applies; it's the formal definition and MarcEdit probably respects that (I only have a very old version, and know very little about it, so cannot properly verify) but I don't think MySQL has any hard limits, nor has Koha unless I'm mistaken. Not at all sure about Zebra... I've just looked at one rather large record <http://opac.navalmarinearchive.com/cgi-bin/koha/opac-detail.pl?biblionumber=23866> but it has only 94 items (and MarcEdit swallows it, but it's only 31242 bytes, so nowhere near "the limit." We're looking at quite a few serial runs of over 100 years of monthly issues! already in ODS format so a direct import into MySQL might be the fastest way. I keep pushing this project to the back of the queue, but will have to face it sooner or later. Paul
You could try compiling to MARCXML, but I don't think the file importer for Koha can handle MARCXML.
The closest I got to creating a massive MRC serials record was to use YAZ-MARCDUMP (http://www.indexdata.com/yaz/doc/yaz-marcdump.html) to convert the big MARCXML record into MARC21. It seemed to compile but it was hard to tell. It wasn't valid MARC after all, and I couldn't exactly look at it in MARCEDIT.
Regards, Karl Holten Systems Integration Specialist SWITCH Inc 414-382-6711
-----Original Message----- From: Koha [mailto:koha-bounces@lists.katipo.co.nz] On Behalf Of Paul A Sent: Thursday, April 28, 2016 2:39 PM To: koha <koha@lists.katipo.co.nz> Subject: Re: [Koha] Best practices for koha - journal migration using marcedit
At 08:24 PM 4/28/2016 +0530, Indranil Das Gupta wrote: [snip
I would like to know how to covert excel data of journals especially with large number of issues(having more than 600 issues) for each journal. Tried using marcedit and it is giving error - 501 when trying to covert mrk to mrc format. [snip] Is there any other way how I can covert this data to mrc format.
Yes, if you do programming or have access to a programmer adept at handling MARC21 data, you can.
If you have details, please make them available. We looked into this (again after MarcEdit attempts), but ran into problems (from memory, more_subfields_xml) ... and we've got about .25 million items to add ;=}
Thanks -- Paul
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>
As far as I know, Koha converts the MRC file into MARCXML format and populates the MySQL database with the information in it. So perhaps he wouldn't need to compile to MRC at all if he went that route. If I'm reading schemaspy right, MySQL has a hard limit of 2147483647 bytes, which is good enough for anyone's needs, I would hope. Zebra has a lower ceiling. I don't know what that is, but we ran into it after migration when we discovered that large serials were not showing up in search results. We deleted some legacy data, which pushed us below that size limit. We're moving to ElasticSearch soon enough and I'm hoping it's not going to be an issue there. (Knock on wood) Regards, Karl Holten Systems Integration Specialist SWITCH Inc 414-382-6711 -----Original Message----- From: Koha [mailto:koha-bounces@lists.katipo.co.nz] On Behalf Of Paul A Sent: Thursday, April 28, 2016 5:44 PM To: koha <koha@lists.katipo.co.nz> Subject: Re: [Koha] Best practices for koha - journal migration using marcedit At 09:39 PM 4/28/2016 +0000, Karl Holten wrote:
The MARC format has an individual record size limit of 99999 bytes, but big serials in Koha can eclipse that size. The result is that really big serials don't compile as valid MARC if you have a ton of items attached to them.
Thanks for the comment. I think that Indranil Das Gupta was proposing a direct intervention at MySQL level -- that's certainly what I am looking for. I'm uncertain as to _where_ the limit of 99999 bytes applies; it's the formal definition and MarcEdit probably respects that (I only have a very old version, and know very little about it, so cannot properly verify) but I don't think MySQL has any hard limits, nor has Koha unless I'm mistaken. Not at all sure about Zebra... I've just looked at one rather large record <http://opac.navalmarinearchive.com/cgi-bin/koha/opac-detail.pl?biblionumber=23866> but it has only 94 items (and MarcEdit swallows it, but it's only 31242 bytes, so nowhere near "the limit." We're looking at quite a few serial runs of over 100 years of monthly issues! already in ODS format so a direct import into MySQL might be the fastest way. I keep pushing this project to the back of the queue, but will have to face it sooner or later. Paul
You could try compiling to MARCXML, but I don't think the file importer for Koha can handle MARCXML.
The closest I got to creating a massive MRC serials record was to use YAZ-MARCDUMP (http://www.indexdata.com/yaz/doc/yaz-marcdump.html) to convert the big MARCXML record into MARC21. It seemed to compile but it was hard to tell. It wasn't valid MARC after all, and I couldn't exactly look at it in MARCEDIT.
Regards, Karl Holten Systems Integration Specialist SWITCH Inc 414-382-6711
-----Original Message----- From: Koha [mailto:koha-bounces@lists.katipo.co.nz] On Behalf Of Paul A Sent: Thursday, April 28, 2016 2:39 PM To: koha <koha@lists.katipo.co.nz> Subject: Re: [Koha] Best practices for koha - journal migration using marcedit
At 08:24 PM 4/28/2016 +0530, Indranil Das Gupta wrote: [snip
I would like to know how to covert excel data of journals especially with large number of issues(having more than 600 issues) for each journal. Tried using marcedit and it is giving error - 501 when trying to covert mrk to mrc format. [snip] Is there any other way how I can covert this data to mrc format.
Yes, if you do programming or have access to a programmer adept at handling MARC21 data, you can.
If you have details, please make them available. We looked into this (again after MarcEdit attempts), but ran into problems (from memory, more_subfields_xml) ... and we've got about .25 million items to add ;=}
Thanks -- Paul
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca> _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
I found the bug that describes the Zebra size limit. To be searchable in Koha, a MARCXML record has to be below 1 MB in size: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15399 Regards, Karl Holten Systems Integration Specialist SWITCH Inc 414-382-6711 -----Original Message----- From: Koha [mailto:koha-bounces@lists.katipo.co.nz] On Behalf Of Karl Holten Sent: Friday, April 29, 2016 1:07 PM To: Paul A <paul.a@navalmarinearchive.com>; koha <koha@lists.katipo.co.nz> Subject: Re: [Koha] Best practices for koha - journal migration using marcedit As far as I know, Koha converts the MRC file into MARCXML format and populates the MySQL database with the information in it. So perhaps he wouldn't need to compile to MRC at all if he went that route. If I'm reading schemaspy right, MySQL has a hard limit of 2147483647 bytes, which is good enough for anyone's needs, I would hope. Zebra has a lower ceiling. I don't know what that is, but we ran into it after migration when we discovered that large serials were not showing up in search results. We deleted some legacy data, which pushed us below that size limit. We're moving to ElasticSearch soon enough and I'm hoping it's not going to be an issue there. (Knock on wood) Regards, Karl Holten Systems Integration Specialist SWITCH Inc 414-382-6711 -----Original Message----- From: Koha [mailto:koha-bounces@lists.katipo.co.nz] On Behalf Of Paul A Sent: Thursday, April 28, 2016 5:44 PM To: koha <koha@lists.katipo.co.nz> Subject: Re: [Koha] Best practices for koha - journal migration using marcedit At 09:39 PM 4/28/2016 +0000, Karl Holten wrote:
The MARC format has an individual record size limit of 99999 bytes, but big serials in Koha can eclipse that size. The result is that really big serials don't compile as valid MARC if you have a ton of items attached to them.
Thanks for the comment. I think that Indranil Das Gupta was proposing a direct intervention at MySQL level -- that's certainly what I am looking for. I'm uncertain as to _where_ the limit of 99999 bytes applies; it's the formal definition and MarcEdit probably respects that (I only have a very old version, and know very little about it, so cannot properly verify) but I don't think MySQL has any hard limits, nor has Koha unless I'm mistaken. Not at all sure about Zebra... I've just looked at one rather large record <http://opac.navalmarinearchive.com/cgi-bin/koha/opac-detail.pl?biblionumber=23866> but it has only 94 items (and MarcEdit swallows it, but it's only 31242 bytes, so nowhere near "the limit." We're looking at quite a few serial runs of over 100 years of monthly issues! already in ODS format so a direct import into MySQL might be the fastest way. I keep pushing this project to the back of the queue, but will have to face it sooner or later. Paul
You could try compiling to MARCXML, but I don't think the file importer for Koha can handle MARCXML.
The closest I got to creating a massive MRC serials record was to use YAZ-MARCDUMP (http://www.indexdata.com/yaz/doc/yaz-marcdump.html) to convert the big MARCXML record into MARC21. It seemed to compile but it was hard to tell. It wasn't valid MARC after all, and I couldn't exactly look at it in MARCEDIT.
Regards, Karl Holten Systems Integration Specialist SWITCH Inc 414-382-6711
-----Original Message----- From: Koha [mailto:koha-bounces@lists.katipo.co.nz] On Behalf Of Paul A Sent: Thursday, April 28, 2016 2:39 PM To: koha <koha@lists.katipo.co.nz> Subject: Re: [Koha] Best practices for koha - journal migration using marcedit
At 08:24 PM 4/28/2016 +0530, Indranil Das Gupta wrote: [snip
I would like to know how to covert excel data of journals especially with large number of issues(having more than 600 issues) for each journal. Tried using marcedit and it is giving error - 501 when trying to covert mrk to mrc format. [snip] Is there any other way how I can covert this data to mrc format.
Yes, if you do programming or have access to a programmer adept at handling MARC21 data, you can.
If you have details, please make them available. We looked into this (again after MarcEdit attempts), but ran into problems (from memory, more_subfields_xml) ... and we've got about .25 million items to add ;=}
Thanks -- Paul
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
--- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca> _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
participants (2)
-
Karl Holten -
Paul A