Koha - creating OAI datasets
I have defined an OAI Data set. In my set mapping I have "595s matches SDAPI" (but in the nomenclature of Koha) After running the program to build the set, the resulting OAI file has _all_ records the database. Not just the ones that matches the map. What am I not doing correctly? Note: We are on v3.16.02.000 Thanks Steve sowder@andrews.edu BTW: A "should not be there record" is at http://koha.andrews.edu/cgi-bin/koha/oai.pl?verb=GetRecord&identifier=SDAPI:1&metadataPrefix=oai_dc A "is and should be there" is at http://koha.andrews.edu/cgi-bin/koha/oai.pl?verb=GetRecord&identifier=SDAPI:105000&metadataPrefix=oai_dc
Hi, On Thu, Jul 24, 2014 at 11:26 AM, Steve Sowder <sowder@andrews.edu> wrote:
I have defined an OAI Data set. In my set mapping I have "595s matches SDAPI" (but in the nomenclature of Koha)
Could you reply with the exact mappings you've defined for that set?
After running the program to build the set, the resulting OAI file has _all_ records the database. Not just the ones that matches the map.
Can you confirm whether the oai_sets_biblios tables has any rows? Based on this http://koha.andrews.edu/cgi-bin/koha/oai.pl?verb=ListIdentifiers&metadataPrefix=oai_dc&set=SDAPI the SDAPI set actually appears to be empty at the moment.
BTW: A "should not be there record" is at http://koha.andrews.edu/cgi-bin/koha/oai.pl?verb=GetRecord&identifier=SDAPI:1&metadataPrefix=oai_dc
A "is and should be there" is at http://koha.andrews.edu/cgi-bin/koha/oai.pl?verb=GetRecord&identifier=SDAPI:105000&metadataPrefix=oai_dc
In both cases, there aren't any setSpec elements in the record header, which is consistent with the set being empty. Regards, Galen -- Galen Charlton Manager of Implementation Equinox Software, Inc. / The Open Source Experts email: gmc@esilibrary.com direct: +1 770-709-5581 cell: +1 404-984-4366 skype: gmcharlt web: http://www.esilibrary.com/ Supporting Koha and Evergreen: http://koha-community.org & http://evergreen-ils.org
On 7/25/2014 12:08 PM, Galen Charlton wrote:
Hi,
On Thu, Jul 24, 2014 at 11:26 AM, Steve Sowder <sowder@andrews.edu> wrote:
I have defined an OAI Data set. In my set mapping I have "595s matches SDAPI" (but in the nomenclature of Koha)
Could you reply with the exact mappings you've defined for that set?
Mapping on tag "595" subfield "s" match on "SDAPI". There are at least 70,000 such records in the database. (I have attached a gif file which is a screen shot. I don't know if this listserv allows attachments. If not, then go to http://www.andrews.edu/~sowder/SDAPI-Map.gif) There is a record that looks like: =LDR 00988nam 2200277 a 4500 =001 00022977 =008 \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\eng\\ =024 \\$a02-1358036x =035 \\$a.b1358036x$bper $ca =035 \\$a.b1358036x$bper $cp =100 2\$aKnight, George R. =245 10$aBiblio file] [A tale of two books. =260 \\$c1990 v63 Sep p29. =260 \\$c1990. =400 \\$aMinistry. =400 \\$g1990 v63 Sep p29. =440 \\$aBiblio file. =513 \\$aBook review. =595 \\$sADVLIB =595 \\$sJWL =595 \\$sSDALIB =595 \\$sSDAPI =705 \\$aTucker, Ruth A. =715 \\$aAnother gospel: alternative religions and the New Age movement. =830 \\$aSDAPI - Seventh-day Adventist Periodical Index. =856 41\$zLink to original record at SDAPI$uhttp://jewel.andrews.edu:82/record=b1358036 =856 \\$zFull-text in DjVU format$uhttp://www.adventistarchives.org/docs/MIN/MIN19900901-V63-09__B/index.djvu?djvuopts&page=29 =856 \\$zFull-text in PDF format$uhttp://www.adventistarchives.org/docs/MIN/MIN19900901-V63-09__B.pdf#page=29&view=fit =952 \\$aSDAPI$bCITE$oSDAPI.b1358036x This can be seen at http://koha.andrews.edu/cgi-bin/koha/oai.pl?verb=GetRecord&identifier=SDAPI:125000&metadataPrefix=oai_dc Although the oai version seems to be missing a lot of stuff.
After running the program to build the set, the resulting OAI file has _all_ records the database. Not just the ones that matches the map.
Can you confirm whether the oai_sets_biblios tables has any rows? Based on this
http://koha.andrews.edu/cgi-bin/koha/oai.pl?verb=ListIdentifiers&metadataPrefix=oai_dc&set=SDAPI
the SDAPI set actually appears to be empty at the moment.
But if you look at it with the following there are records. But, it does not specify a set. http://koha.andrews.edu/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc Thus: It looks like the build_oai_sets creates an OAI file with _all_ of the records in the database (no SetSpec) and includes those that map as subsets (with a SetSpec). Is that correct? I don't know why my oai-set map is not finding the 70,000+ records. Do you have any suggestions? Thanks Steve sowder@andrews.edu
BTW: A "should not be there record" is at http://koha.andrews.edu/cgi-bin/koha/oai.pl?verb=GetRecord&identifier=SDAPI:1&metadataPrefix=oai_dc
A "is and should be there" is at http://koha.andrews.edu/cgi-bin/koha/oai.pl?verb=GetRecord&identifier=SDAPI:105000&metadataPrefix=oai_dc
In both cases, there aren't any setSpec elements in the record header, which is consistent with the set being empty.
Regards,
Galen
This is what I know. I have read again the info at http://www.openarchives.org/OAI/openarchivesprotocol.html#Set It now makes more sense to me. OAI databases are created with the full records of the database. In our case that is about 190,000 records. SetSpecs are subsets of the database (for "Selective Harvesting"). Any record can be in zero or more SetSpecs. In Koha, when an OAI set is defined that new SetSpec will appear in the ListSets. (i.e. http://koha.andrews.edu/cgi-bin/koha/oai.pl?verb=ListSets) without rerunning the program build_oai-set.pl. At least, it appears after overnight. However, there are still no records (should be 70,000) in either of the SetSpecs. Which is what I don't understand. The mapping follows the instructions given in the documentation. So, questions: Is mapping only available to certain MARC tags? Indexed tags only? For those of you that have SetSpecs with records please send me examples of your set mapping. Thanks Steve sowder@andrews.edu
I have discovered: 1. Tags used in mapping have to be indexed. Otherwise the SetSpec won't have anything in it. This should be in the manual. Perhaps everyone just thinks "well everybody knows that". But it should be in the manual because not everyone knows. 2. Requesting a list with an invalid SetSpec (i.e. "joke") gives you all of the records. Same as no SetSpec at all. It probably should give an error. No such "set". Steve sowder@andres.edu On 7/30/2014 11:36 AM, Steve Sowder wrote:
This is what I know.
I have read again the info at http://www.openarchives.org/OAI/openarchivesprotocol.html#Set
It now makes more sense to me.
OAI databases are created with the full records of the database. In our case that is about 190,000 records.
SetSpecs are subsets of the database (for "Selective Harvesting").
Any record can be in zero or more SetSpecs.
In Koha, when an OAI set is defined that new SetSpec will appear in the ListSets. (i.e. http://koha.andrews.edu/cgi-bin/koha/oai.pl?verb=ListSets) without rerunning the program build_oai-set.pl. At least, it appears after overnight.
However, there are still no records (should be 70,000) in either of the SetSpecs. Which is what I don't understand.
The mapping follows the instructions given in the documentation.
So, questions: Is mapping only available to certain MARC tags? Indexed tags only?
For those of you that have SetSpecs with records please send me examples of your set mapping.
Thanks
Steve sowder@andrews.edu
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
Sorry for the late reply! You did not mention running misc/migration_tools/build_oai_sets.pl. I guess there might be two reasons for that: - you don't know about it - it's so obvious you did not mention it My apologies if it is the latter. If it is the former, you might want to have a look at the documentation for that script: http://perldoc.koha-community.org/misc/migration_tools/build_oai_sets.html Best regards, Magnus On 30 July 2014 17:36, Steve Sowder <sowder@andrews.edu> wrote:
This is what I know.
I have read again the info at http://www.openarchives.org/OAI/openarchivesprotocol.html#Set
It now makes more sense to me.
OAI databases are created with the full records of the database. In our case that is about 190,000 records.
SetSpecs are subsets of the database (for "Selective Harvesting").
Any record can be in zero or more SetSpecs.
In Koha, when an OAI set is defined that new SetSpec will appear in the ListSets. (i.e. http://koha.andrews.edu/cgi-bin/koha/oai.pl?verb=ListSets) without rerunning the program build_oai-set.pl. At least, it appears after overnight.
However, there are still no records (should be 70,000) in either of the SetSpecs. Which is what I don't understand.
The mapping follows the instructions given in the documentation.
So, questions: Is mapping only available to certain MARC tags? Indexed tags only?
For those of you that have SetSpecs with records please send me examples of your set mapping.
Thanks
Steve sowder@andrews.edu
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
It was obvious. I did run build_oai_sets.pl Bottom line: a record cannot be in two oai sets that use the same indexed MARC tag. i.e. ... 590 a SET1 590 a SET2 ... This record will be in the oai dataset SET1 but will not be in SET2. Only the first occurrence of the 590a is used to create oai datasets. Steve On 8/25/2014 3:17 AM, Magnus Enger wrote:
Sorry for the late reply!
You did not mention running misc/migration_tools/build_oai_sets.pl. I guess there might be two reasons for that: - you don't know about it - it's so obvious you did not mention it
My apologies if it is the latter. If it is the former, you might want to have a look at the documentation for that script: http://perldoc.koha-community.org/misc/migration_tools/build_oai_sets.html
Best regards, Magnus
On 30 July 2014 17:36, Steve Sowder <sowder@andrews.edu> wrote:
This is what I know.
I have read again the info at http://www.openarchives.org/OAI/openarchivesprotocol.html#Set
It now makes more sense to me.
OAI databases are created with the full records of the database. In our case that is about 190,000 records.
SetSpecs are subsets of the database (for "Selective Harvesting").
Any record can be in zero or more SetSpecs.
In Koha, when an OAI set is defined that new SetSpec will appear in the ListSets. (i.e. http://koha.andrews.edu/cgi-bin/koha/oai.pl?verb=ListSets) without rerunning the program build_oai-set.pl. At least, it appears after overnight.
However, there are still no records (should be 70,000) in either of the SetSpecs. Which is what I don't understand.
The mapping follows the instructions given in the documentation.
So, questions: Is mapping only available to certain MARC tags? Indexed tags only?
For those of you that have SetSpecs with records please send me examples of your set mapping.
Thanks
Steve sowder@andrews.edu
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
participants (3)
-
Galen Charlton -
Magnus Enger -
Steve Sowder