[Koha] Best practices for koha - journal migration using marcedit

[Karl Holten]

I found the bug that describes the Zebra size limit. To be searchable in Koha, a MARCXML record has to be below 1 MB in size: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15399

Regards,
Karl Holten
Systems Integration Specialist
SWITCH Inc
414-382-6711

-----Original Message-----
From: Koha [mailto:koha-bounces at lists.katipo.co.nz] On Behalf Of Karl Holten
Sent: Friday, April 29, 2016 1:07 PM
To: Paul A <paul.a at navalmarinearchive.com>; koha <koha at lists.katipo.co.nz>
Subject: Re: [Koha] Best practices for koha - journal migration using marcedit

As far as I know, Koha converts the MRC file into MARCXML format and populates the MySQL database with the information in it. So perhaps he wouldn't need to compile to MRC at all if he went that route. 

If I'm reading schemaspy right, MySQL has a hard limit of 2147483647 bytes, which is good enough for anyone's needs, I would hope. Zebra has a lower ceiling. I don't know what that is, but we ran into it after migration when we discovered that large serials were not showing up in search results. We deleted some legacy data, which pushed us below that size limit. We're moving to ElasticSearch soon enough and I'm hoping it's not going to be an issue there. (Knock on wood)

Regards,
Karl Holten
Systems Integration Specialist
SWITCH Inc
414-382-6711

-----Original Message-----
From: Koha [mailto:koha-bounces at lists.katipo.co.nz] On Behalf Of Paul A
Sent: Thursday, April 28, 2016 5:44 PM
To: koha <koha at lists.katipo.co.nz>
Subject: Re: [Koha] Best practices for koha - journal migration using marcedit

At 09:39 PM 4/28/2016 +0000, Karl Holten wrote:
>The MARC format has an individual record size limit of 99999 bytes, but 
>big serials in Koha can eclipse that size. The result is that really 
>big serials don't compile as valid MARC if you have a ton of items 
>attached to them.

Thanks for the comment. I think that Indranil Das Gupta was proposing a direct intervention at MySQL level -- that's certainly what I am looking for.

I'm uncertain as to _where_ the limit of 99999 bytes applies; it's the formal definition and MarcEdit probably respects that (I only have a very old version, and know very little about it, so cannot properly verify) but I don't think MySQL has any hard limits, nor has Koha unless I'm mistaken. 
Not at all sure about Zebra...

I've just looked at one rather large record <http://opac.navalmarinearchive.com/cgi-bin/koha/opac-detail.pl?biblionumber=23866>
but it has only 94 items (and MarcEdit swallows it, but it's only 31242 bytes, so nowhere near "the limit."

We're looking at quite a few serial runs of over 100 years of monthly issues! already in ODS format so a direct import into MySQL might be the fastest way. I keep pushing this project to the back of the queue, but will have to face it sooner or later.

Paul


>You could try compiling to MARCXML, but I don't think the file importer 
>for Koha can handle MARCXML.
>
>The closest I got to creating a massive MRC serials record was to use 
>YAZ-MARCDUMP (http://www.indexdata.com/yaz/doc/yaz-marcdump.html) to 
>convert the big MARCXML record into MARC21. It seemed to compile but it 
>was hard to tell. It wasn't valid MARC after all, and I couldn't 
>exactly look at it in MARCEDIT.
>
>Regards,
>Karl Holten
>Systems Integration Specialist
>SWITCH Inc
>414-382-6711
>
>-----Original Message-----
>From: Koha [mailto:koha-bounces at lists.katipo.co.nz] On Behalf Of Paul A
>Sent: Thursday, April 28, 2016 2:39 PM
>To: koha <koha at lists.katipo.co.nz>
>Subject: Re: [Koha] Best practices for koha - journal migration using 
>marcedit
>
>At 08:24 PM 4/28/2016 +0530, Indranil Das Gupta wrote:
>[snip
> > > I would like to know how to covert excel data of journals 
> > > especially with large number of issues(having more than 600
> > > issues) for each journal. Tried using marcedit and it is giving 
> > > error - 501 when trying to covert mrk to mrc format.
>[snip]
> > > Is there any other way how I can covert this data to mrc format.
> >
> >Yes, if you do programming or have access to a programmer adept at 
> >handling MARC21 data, you can.
>
>If you have details, please make them available. We looked into this 
>(again after MarcEdit attempts), but ran into problems (from memory,
>more_subfields_xml) ... and we've got about .25 million items to add 
>;=}
>
>Thanks -- Paul
>
>_______________________________________________
>Koha mailing list  http://koha-community.org Koha at lists.katipo.co.nz 
>https://lists.katipo.co.nz/mailman/listinfo/koha

---
Maritime heritage and history, preservation and conservation, research and education through the written word and the arts.
<http://NavalMarineArchive.com> and <http://UltraMarine.ca>

_______________________________________________
Koha mailing list  http://koha-community.org Koha at lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________
Koha mailing list  http://koha-community.org Koha at lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha