Batch import-updating of existing biblios
This isn't just a question on Koha's capabilities - I'm also seeking general advice from a workflow/process POV. Say I have a set of biblio records that are just the way I want them, in the sense of the accuracy of the existing data in the currently populated fields. And I would like to "add value" to them later on - say I find a MARC source that assigns subject headings in a way that I like, or adds links to cover images, etc. looking to selectively overlay/insert a subset of the new records into my existing ones. How does Koha handle this? Or would I be better off exporting my current records to MARC, using an external MARC editing tool to handle the merge and then re-import? Can anyone suggest tools for me to look at? I'm aware of biblios.net and MarcEdit so far. Obviously being able to do this in batch mode with reasonable matching rules would be ideal rather than one at a time, as long as I can review the results before "committing" the update. Or if it's really better to do this one record at a time, does that change the recommendation as to using Koha for the process? And finally, what if there are certain field values that I do want overwritten in addition to the new ones being brought in, but some fields shouldn't be touched at all? Thanks in advance. . .
You can define matching rules on various indexed fields -- the most common are ISBN and ISSN. However, what happens when you load a matching record is a complete replacement of the existing record. It's not an "overlay" in the sense of keeping whatever fields are now present and adding new ones. Koha does not presently have that capability. Hope that helps. Jane Wagner Senior Project Manager LibLime, a division of PTFS Content Management and Library Solutions 6400 Goldsboro Road, Suite 200 Bethesda, MD 20817 (301) 654-8088 x 151 jwagner@liblime.com -----Original Message----- From: koha-bounces@lists.katipo.co.nz [mailto:koha-bounces@lists.katipo.co.nz] On Behalf Of hansbkk@gmail.com Sent: Tuesday, February 22, 2011 12:07 PM To: koha@lists.katipo.co.nz Subject: [Koha] Batch import-updating of existing biblios This isn't just a question on Koha's capabilities - I'm also seeking general advice from a workflow/process POV. Say I have a set of biblio records that are just the way I want them, in the sense of the accuracy of the existing data in the currently populated fields. And I would like to "add value" to them later on - say I find a MARC source that assigns subject headings in a way that I like, or adds links to cover images, etc. looking to selectively overlay/insert a subset of the new records into my existing ones. How does Koha handle this? Or would I be better off exporting my current records to MARC, using an external MARC editing tool to handle the merge and then re-import? Can anyone suggest tools for me to look at? I'm aware of biblios.net and MarcEdit so far. Obviously being able to do this in batch mode with reasonable matching rules would be ideal rather than one at a time, as long as I can review the results before "committing" the update. Or if it's really better to do this one record at a time, does that change the recommendation as to using Koha for the process? And finally, what if there are certain field values that I do want overwritten in addition to the new ones being brought in, but some fields shouldn't be touched at all? Thanks in advance. . . _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
On Wed, Feb 23, 2011 at 12:36 AM, Wagner, Jane <jwagner@liblime.com> wrote:
You can define matching rules on various indexed fields -- the most common are ISBN and ISSN. However, what happens when you load a matching record is a complete replacement of the existing record. It's not an "overlay" in the sense of keeping whatever fields are now present and adding new ones. Koha does not presently have that capability.
Thanks for the clarification - I assume you mean not only in batch mode, but even one at a time 8-( So it seems I need to export my biblios merge/overlay the new data with an external MARC editor and then re-import. The silver lining is the certainty of having reliable matching data (most of my biblios have neither ISBN nor ISSN). So the question remains as to editor suggestions, especially if one is available that does handle batch updates; although if one is particularly good at one-at-a-time merging that's valuable information as well.
I'm aware of biblios.net and MarcEdit so far.
I'm assuming bringing in whole new fields and adding new multiple-entry ones shouldn't be a problem, but I'd like to be able to specify the fields-in-common, as to which are OK to over-write vs which shouldn't be touched.
-----Original Message----- Say I have a set of biblio records that are just the way I want them, in the sense of the accuracy of the existing data in the currently populated fields.
And I would like to "add value" to them later on - say I find a MARC source that assigns subject headings in a way that I like, or adds links to cover images, etc. looking to selectively overlay/insert a subset of the new records into my existing ones.
How does Koha handle this?
I'm told that you can do this more easily by uploading the new record into Koha and using the 3.2 biblio merge tool to merge in the additional fields. This should allow you to combine choosing the fields you want. Liz Rea NEKLS On Feb 22, 2011, at 11:54 AM, hansbkk@gmail.com wrote:
On Wed, Feb 23, 2011 at 12:36 AM, Wagner, Jane <jwagner@liblime.com> wrote:
You can define matching rules on various indexed fields -- the most common are ISBN and ISSN. However, what happens when you load a matching record is a complete replacement of the existing record. It's not an "overlay" in the sense of keeping whatever fields are now present and adding new ones. Koha does not presently have that capability.
Thanks for the clarification - I assume you mean not only in batch mode, but even one at a time
8-(
So it seems I need to export my biblios merge/overlay the new data with an external MARC editor and then re-import. The silver lining is the certainty of having reliable matching data (most of my biblios have neither ISBN nor ISSN).
So the question remains as to editor suggestions, especially if one is available that does handle batch updates; although if one is particularly good at one-at-a-time merging that's valuable information as well.
I'm aware of biblios.net and MarcEdit so far.
I'm assuming bringing in whole new fields and adding new multiple-entry ones shouldn't be a problem, but I'd like to be able to specify the fields-in-common, as to which are OK to over-write vs which shouldn't be touched.
-----Original Message----- Say I have a set of biblio records that are just the way I want them, in the sense of the accuracy of the existing data in the currently populated fields.
And I would like to "add value" to them later on - say I find a MARC source that assigns subject headings in a way that I like, or adds links to cover images, etc. looking to selectively overlay/insert a subset of the new records into my existing ones.
How does Koha handle this?
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
Yes, that's a good point -- I didn't think about the bib merge tool. However, if I remember correctly that would mean processing title records one at a time -- no batch functionality. And to confirm your suspicion, the "replace not overlay" happens even if you're doing one record at a time. Jane Wagner Senior Project Manager LibLime, a division of PTFS Content Management and Library Solutions 6400 Goldsboro Road, Suite 200 Bethesda, MD 20817 (301) 654-8088 x 151 jwagner@liblime.com -----Original Message----- From: koha-bounces@lists.katipo.co.nz [mailto:koha-bounces@lists.katipo.co.nz] On Behalf Of Liz Rea Sent: Tuesday, February 22, 2011 1:03 PM To: hansbkk@gmail.com Cc: koha@lists.katipo.co.nz koha Subject: Re: [Koha] Batch import-updating of existing biblios I'm told that you can do this more easily by uploading the new record into Koha and using the 3.2 biblio merge tool to merge in the additional fields. This should allow you to combine choosing the fields you want. Liz Rea NEKLS On Feb 22, 2011, at 11:54 AM, hansbkk@gmail.com wrote:
On Wed, Feb 23, 2011 at 12:36 AM, Wagner, Jane <jwagner@liblime.com> wrote:
You can define matching rules on various indexed fields -- the most common are ISBN and ISSN. However, what happens when you load a matching record is a complete replacement of the existing record. It's not an "overlay" in the sense of keeping whatever fields are now present and adding new ones. Koha does not presently have that capability.
Thanks for the clarification - I assume you mean not only in batch mode, but even one at a time
8-(
So it seems I need to export my biblios merge/overlay the new data with an external MARC editor and then re-import. The silver lining is the certainty of having reliable matching data (most of my biblios have neither ISBN nor ISSN).
So the question remains as to editor suggestions, especially if one is available that does handle batch updates; although if one is particularly good at one-at-a-time merging that's valuable information as well.
I'm aware of biblios.net and MarcEdit so far.
I'm assuming bringing in whole new fields and adding new multiple-entry ones shouldn't be a problem, but I'd like to be able to specify the fields-in-common, as to which are OK to over-write vs which shouldn't be touched.
-----Original Message----- Say I have a set of biblio records that are just the way I want them, in the sense of the accuracy of the existing data in the currently populated fields.
And I would like to "add value" to them later on - say I find a MARC source that assigns subject headings in a way that I like, or adds links to cover images, etc. looking to selectively overlay/insert a subset of the new records into my existing ones.
How does Koha handle this?
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
On Wed, Feb 23, 2011 at 1:02 AM, Liz Rea <lrea@nekls.org> wrote:
I'm told that you can do this more easily by uploading the new record into Koha and using the 3.2 biblio merge tool to merge in the additional fields. This should allow you to combine choosing the fields you want.
Thanks Liz. I found this, assume it applies - to the developers, should this level of detail be reflected in the documentation? I'd be happy to help if it's accessible - wiki? http://lists.koha-community.org/pipermail/koha-patches/2010-October/012789.h... BTW Liz I enjoyed your blog post about the Liblime vs community issues. Finally, anyone have suggestions as to standalone MARC editor capable of batch merging rather than one-on-one?
I'm aware of biblios.net and MarcEdit so far.
I'm assuming bringing in whole new fields and adding new multiple-entry ones shouldn't be a problem, but I'd like to be able to specify the fields-in-common, as to which are OK to over-write vs which shouldn't be touched.
On Wed, Feb 23, 2011 at 1:50 AM, <hansbkk@gmail.com> wrote:
should this level of detail be reflected in the documentation? I'd be happy to help if it's accessible - wiki?
http://lists.koha-community.org/pipermail/koha-patches/2010-October/012789.h...
Turns out the above is something different, having to do with duplicate items in a list (virtual shelf) ?? I found this in the docs http://koha-community.org/documentation/3-2-manual/?ch=x4484#AEN4901 Looks like it's also somewhat relevant, could use some filling-in, so here are my ideas - what's the proper channel for docs input? ---------------------------------------------- Choose a unique name and enter it in the 'Matching rule code' field 'Description' can be anything you want to make it clear to you what rule you're picking --> 'Match threshold' - Choose a number so that a sufficient number of "match points" from below will trigger a higher total - if you only have one match point, make this number that point's "Score" minus 1 Match points are set up to determine what fields to match on 'Search index' can be found by looking at the ccl.properties file on your system which tells the zebra indexing what data to search for in the MARC data" --> above needs the name/location of the file --> 'Score' - Choose a number to determine this point's contribution to the total toward triggering a match. If you only have one match point, make it the above "match threshold" +1 Enter the MARC tag you want to match on in the 'Tag' field --> Enter the MARC tag subfield you want to match on in the 'Subfields' field, *or* --> 'Offset' - for fixed MARC subfields that use character-count location rather than subfield codes, combined with --> 'Length' - the number of characters to count from the offset Koha currently has only one 'Normalization rule' that removes extra characters such as commas and semicolons. The value you enter in this field is irrelevant to the normalization process. 'Required match checks' - ?? Then I try the online help, which it turns out explains things pretty well - excellent! Which brings me to a "big picture" question - wouldn't it be a good idea to have the online help (within the Koha interface) and the website's documentation coordinated. Even if it's basically duplicated, better than having them maintained separately, bits missing from each or maybe getting out of sync? ---------------------------------------------- Record Matching Rules IMPORTANT: This is an advanced feature and should not be altered without knowing how it will effect data migration. Use this tool to create rules to apply during the data migration process. It will prevent duplicates from coming into the system when importing MARC records. An import rule or matching rule consists of one or more 'match points' and zero or more 'match checks'. Each match point specifies a 'search index' and a MARC 'tag', 'subfield', or 'length' (fixed field position) when a record is imported. For each match point, a string is constructed from the tag specified in the match point and the related index is searched. The set of matching records are assigned a score (the value of which is determined by the match point rule). Then, the rest of the match points are considered and the scores of each set of matches is added up. The set of matching records whose total score is over a threshold value defined in the matching rule are candidate matches. Match checks are applied for all candidate matches. Each match check specifies a tag in the incoming record and a tag in the possible matching record. The values must be the same for a match to be considered good (e.g., doing a match check on title, or publication date). ---------------------------------------------- And finally, does this functionality kick in when bringing records into the reservoir, or from there into the database itself?
participants (3)
-
hansbkk＠gmail.com -
Liz Rea -
Wagner, Jane