- Koha - lists.katipo.co.nz

Ramblings on a Catalogue API
by Tonnesen Steve 20 Jun '01

20 Jun '01

More technical mumbo jumbo. I'm thinking the new Catalogue API should have two interfaces: one that is modelled on the biblio,biblioitems,items table and one that is modeled on the MARC data. Here are my ramblings on the first interface, which would need to be implemented first anyway. Adding a new item: Create a $item object: $item->{'biblioitemnumber'}=xxxx; $item->{'barcode'}=xxxxx; . . . dateaccessioned=xxxxx; booksellerid=xxxxxx; homebranch=xxxxxxx; price= replacementprice= replacementpricedate= notforloan= restricted= holdingbranch= itemnotes= Call Catalogue->AddItem($env, $item) The AddItem subroutine can insert the data into a new items record as before, as well as: Grab the MARC Record_ID number from the biblioitems table for biblioitemnumber given in the $item object and add an 852 tag for the new item in the MARC record. We still have to work out where in the 852 tag the different bits of information will be stored, or if some information maybe should be stored elsewhere, like in 859 (any tag, subfield, or indicator with a 9 in it is a local implementation). Catalogue->UpdateItem($env, $item) could be used to modify an existing item. The $item object in this case would also have to have an itemnumber. The MARC side could grab the barcode from the items table to find the corresponding 852 entry in the MARC record. Catalogue->AddBiblioitem($env, $biblioitem) $biblioitem->{'biblionumber'}=xxxx; volume= number= classification= itemtype= isbn= dewey= subclass= etc. Again, this would add an entry to the biblioitems table, and create a new MARC record. The Record_ID value from the newly created MARC record would also be inserted into the new biblioitems record in a new Record_ID field. Catalogue->UpdateBiblioitem($env, $biblioitem) Same idea as UpdateItem... Catalogue->AddBiblio($env, $biblio) Catalogue->UpdateBiblio($env, $biblio) These two subroutines have no counterpart in the MARC side. The data contained in the biblio table is stored in the MARC record for each of the corresponding biblioitems. All of the MARC records associated with a given biblio would have a unique id (same as the biblionumber) stored somewhere in the MARC record, location still to be determined. I'm not sure at the moment how many places in the existing Koha code will have to be modified to use this new API. All over the acquisitions code, for sure, as well as the catalogue editing code. Any other places that need looking at? Any spots in the code where updates may not fit with my proposed API here? Reading subroutines would be similar to the writing subroutines: Catalogue->GetBiblio($env, $biblionumber); Catalogue->GetBiblioitem($env, $biblioitemnumber); Catalogue->GetItem($env, $itemnumber); something like that. These three subroutines would _always_ read from the MARC record data. This will be more difficult than the current setup which just grabs the entire row from an SQL select statement and returns the result as the object, but it ensures that there is a single authoritative data source and allows for the eventual removal of the biblio, biblioitems and items tables once their functionality is completely met by the MARC records. These subroutines, specifically the GetBiblioitem and GetItem would also return some structured MARC data: 1: a full MARC record, 2: a text structured MARC record, 3: an HTML formatted MARC record. Steve.

1 0

MARC Support
by Tonnesen Steve 20 Jun '01

20 Jun '01

I never realized what a hassle it is to sort lists where numbers come after letters (as is the norm with MARC subfield marks). MySQL will only sort with numbers before the letters. I've managed to get the MARC records sorting in the right order now, and it can optionally display both tag names and subfield names for those of us for whom MARC is somewhat greek. When adding new subfields, there is a drop down box with subfield marks and names, or a simple text entry box where you can just type in the subfield mark. Now I really need to start thinking about how to tie this in to Koha. I don't think it should be too difficult to generate MARC records from the existing biblio, biblioitems, items table data. I suppose the easiest thing to do would be to insert some kind of API everywhere that modifies those tables so that they also modify the MARC records. This would be a first step towards adding MARC support. Second step would be inserting an API for every routine that _reads_ from those tables. At that point, all reads could gather data from the MARC records, and all writes could update both the old tables and the new MARC tables. MARC records would be linked at the biblioitems level. The items table data would, I believe, mostly be recorded in an 852 - Location tag in the Marc data. Links to the biblio table could be stored in a local 9XX tag. For example, if there is a biblio entry for "King Lear" with biblionumber 5214, and there are 4 biblioitems for that biblio (hardcover, paberback, cassette, large print), the MARC records for each of those four would have a tag like: 944 ## a 5214 When doing searches on MARC records, all four results with the same 944 a value could be lumped together under "King Lear" in the results output. I believe this would result in the same behaviour obtained with the three split tables? ------------------------ Technical Stuff Below!!! ------------------------ Rough outline of the Schema I used: I used the schema developed for the Pytheas project. It is fully capable of storing all of the information in any MARC record, and is not overly difficult to search. One feature I'd like to add is a keyword index so that soundex searches would work (ie soundex(elefant) = soundex(elephant)). The Resource_Table contains an entry for every MARC record. Resource_Table Resource_ID int auto_increment Record_ID int The Bib_Table contains an entry for every tag in a record. The Record_ID is used to link all tags in a MARC record together. Each entry in Bib_Table will have one of the Tag_#XX_ID fields filled out with a number that links to the appropriate #XX_Tag_Table. The Control_ID field links to a Control_Table that is used for the fixed length data elements like 008. Bib_Table Bib_ID int auto_increment Record_ID int Tag_0XX_ID int Tag_1XX_ID int Tag_2XX_ID int Tag_3XX_ID int Tag_4XX_ID int Tag_5XX_ID int Tag_6XX_ID int Tag_7XX_ID int Tag_8XX_ID int Tag_9XX_ID int Control_ID int The 2XX_Tag_Table is representative of 10 tables (0XX_Tag_Table through 9XX_Tag_Table). It contains one entry for each tag of a record. The Tag_ID value is linked back to the Bib_Table Tag_2XX_ID field. The Subfield_ID is used to link to the 2XX_Subfield_Table which contains all of the subfields for this tag. 2XX_Tag_Table Tag_Key int auto_increment Tag_ID int Indicator1 int Indicator2 int Tag char(3) Subfield_ID int The 2XX_Subfield_Table is again representative of 10 such tables. The Subfield_ID is used to link all of the subfields of a given tag together and back to the 2XX_Tag_Table entry. Small values can be stored directly in the Subfield_Value field. Larger values are stored in separate blob tables using the Storage_ID field as the link. 2XX_Subfield_Table Subfield_Key int auto_increment Subfield_ID int Subfield_Mark char(1) Subfield_Value char(255) Storage_ID int The Storage_Table and *Blob_Table is where the data elements themselves are stored. Depending on the size of the data, they can be stored in the Blob_Table, Med_Blob_Table, or the Long_Blob_Table, based on the value of Storage_Type. Storage_Table Storage_Key int auto_increment Storage_ID int Blob_ID int Med_Blob_ID int Long_Blob_ID int Storage_Type enum('B', 'MB', 'LB'); The Med_Blob_Table and Long_Blob_Table are identical to Blob_Table except that the Blob_Data field is mediumblob or longblob respectively. The Seq_No field can be used to break a huge record into multiple longblob's if necessary. Blob_Table Blob_Key int auto_increment Blob_ID int Blob_Data blob Seq_No int If I wanted to find the title and author of all records that had "bear" in any subject (6XX) tag, I would use: select B.Record_ID from Bib_Table B, 6XX_Tag_Table T, 6XX_Subfield_Table S where B.Tag_6XX_ID=T.Tag_ID and T.Subfield_ID=S.Subfield_ID and S.Subfield_Mark='a' and S.Subfield_Value like '%bear%'; for each Record_ID returned, run the following to get the author: select S.Subfield_Value from Bib_Table B, 1XX_Tag_Table T, 1XX_Subfield_Table S where B.Record_ID=$Record_ID and B.Tag_1XX_ID=T.Tag_ID and T.Subfield_ID=S.Subfield_ID and T.Tag='100'; and the following to get the title: select S.Subfield_Value from Bib_Table B, 2XX_Tag_Table T, 2XX_Subfield_Table S where B.Record_ID=$Record_ID and B.Tag_2XX_ID=T.Tag_ID and T.Subfield_ID=S.Subfield_ID and T.Tag='245';

1 0

Re: [Koha] More MARC development
by COURYHOUSE＠aol.com 19 Jun '01

19 Jun '01

Hi Steve! Looks neat keep up the good work! Sounds like the open source meeting AT ALA in SF calif. USA was real interesting here is what was there...... I was not able to make it but Bette Sharpe did attend and will be bringing me back a stash of whatever she could grab.... Ed Sharpe LITA Open Source Systems Interest Group Program: "Web Tools and Digital Resources: Open Source Then and Now" Sunday, June 17, 2001, 8:30am-10:30am Marriot Golden Gate Section B Access and services that libraries provide is often limited by the software and services that they are able to purchase. But an alternative--freely-available open source software--is available to help libraries improve and shape services and better meet users' needs. But how can librarians get involved in development of these products and contribute to the process? This program features open source developers discussing their projects: Larry Wall and PERL, Eric Lease Morgan and <A HREF="mailto:MyLibrary@NCState">MyLibrary@NCState</A>, and Willem Schulton with projects in rural public libraries. Speakers: 1. Willem Scholten, Executive Director, Waitt Family Foundation Technology Resource Center, Seattle, WA 2. Eric Lease Morgan, Networked Resources Development Librarian, North Carolina State University Libraries, Raleigh, NC 27695-7111 3. Larry Wall, PERL Inventor and author, c/o O'Reilly & Associates, Sebastopol, CA 95472 > Subj: [Koha] More MARC development > Date: 6/18/01 12:38:42 PM US Mountain Standard Time > From: tonnesen(a)cmsd.bc.ca (Tonnesen Steve) > Sender: koha-admin(a)lists.katipo.co.nz > Reply-to: <A HREF="mailto:tonnesen@cmsd.bc.ca">tonnesen(a)cmsd.bc.ca</A> (Tonnesen Steve) > To: koha(a)lists.katipo.co.nz (Koha Open Source Library System) > > > > > > My MARC editor has come a little farther along. It is now possible to add > new fields and subfields, edit existing subfield values, and delete > subfield entries. > > > Still to be done: > > - the interface needs some major improvements > - the code needs to be broken down into modules of some sort. > - no way to add a subfield if all of the existing subfields are > deleted... workaround is probably to delete the tag when the last > subfield is deleted, so that the tag has to be recreated with an > initial subfield. > - Lots of work actually tying this code into Koha somehow. This includes > running alongside the existing database layout, as well as tying into > things like catalogue searching, etc. > - Setting up standard locations for Koha data that may not be covered by > MARC (ie withdrawn, lost, itemtype). > > I've parsed all of the MARC21 field and subfield descriptions so they all > show up in the interface as well. I gather that this is the part that > will likely change in different versions of MARC though? It would be > fairly simple to modify the field and subfield definitions based on a > preference setting for the version of MARC (MARC21, UKMARC, NZMARC, > MARSMARC, etc.). > > Note that the data on my test site is free to edit and play with. It's > actually running on my laptop, and is a copy of a library's data. Wreck > it at your leisure, I can restore it quickly if it gets really bad. :) > > Demo is at: > > http://142.28.224.11/cgi-bin/koha/acqui.simple/viewmarc.pl > > >

2 1

Re: [Koha] Text based front end
by COURYHOUSE＠aol.com 18 Jun '01

18 Jun '01

as I remember it can run on the linux server as well as the opac and in theory you could hook vt 100 type terminals to it. most companies throw these old terminals away! thus using less pc's and perhaps having a little better security. Ed Sharpe archivist for SMECC > Subj: [Koha] Text based front end > Date: 6/18/01 12:41:38 PM US Mountain Standard Time > From: nsr4n(a)tetra.mail.virginia.edu (Nicholas Stephen Rosasco) > Sender: koha-admin(a)lists.katipo.co.nz > To: koha(a)lists.katipo.co.nz > > > > > A few months/weeks ago I randomly asked who would be interested in a text > front end for the user side/opac, and someone saved me from creating more > code work by suggesting the use of a text based WWW browser. > > I don't know how many of you have used "lynx", which is nice and available > for free (lynx.browser.org) > or links (http://artax.karlin.mff.cuni.cz/~mikulas/links/) which is a bit > simpler. > > both would server well-- particularly if you have color screen on your > terminals. > >

1 0

More MARC development
by Tonnesen Steve 18 Jun '01

18 Jun '01

My MARC editor has come a little farther along. It is now possible to add new fields and subfields, edit existing subfield values, and delete subfield entries. Still to be done: - the interface needs some major improvements - the code needs to be broken down into modules of some sort. - no way to add a subfield if all of the existing subfields are deleted... workaround is probably to delete the tag when the last subfield is deleted, so that the tag has to be recreated with an initial subfield. - Lots of work actually tying this code into Koha somehow. This includes running alongside the existing database layout, as well as tying into things like catalogue searching, etc. - Setting up standard locations for Koha data that may not be covered by MARC (ie withdrawn, lost, itemtype). I've parsed all of the MARC21 field and subfield descriptions so they all show up in the interface as well. I gather that this is the part that will likely change in different versions of MARC though? It would be fairly simple to modify the field and subfield definitions based on a preference setting for the version of MARC (MARC21, UKMARC, NZMARC, MARSMARC, etc.). Note that the data on my test site is free to edit and play with. It's actually running on my laptop, and is a copy of a library's data. Wreck it at your leisure, I can restore it quickly if it gets really bad. :) Demo is at: http://142.28.224.11/cgi-bin/koha/acqui.simple/viewmarc.pl Steve.

2 1

Text based front end
by Nicholas Stephen Rosasco 18 Jun '01

18 Jun '01

A few months/weeks ago I randomly asked who would be interested in a text front end for the user side/opac, and someone saved me from creating more code work by suggesting the use of a text based WWW browser. I don't know how many of you have used "lynx", which is nice and available for free (lynx.browser.org) or links (http://artax.karlin.mff.cuni.cz/~mikulas/links/) which is a bit simpler. both would server well-- particularly if you have color screen on your terminals. Nick

2 1

Beginning of Marc support
by Tonnesen Steve 14 Jun '01

14 Jun '01

http://142.28.224.11/cgi-bin/koha/acqui.simple/viewmarc.pl Pretty rudimentary so far. Can add a single new field, checks if non-repeatable tags already exist. No edit feature, no ability to add extra subfields. Steve.

1 0

Re: marc and stuff!
by Tonnesen Steve 07 Jun '01

07 Jun '01

On Wed, 23 May 2001, Chris Cormack wrote: > > we may be at that point someday ... I guess the biggest question > > is how to load the marc records into the database can you point me > > somewhere for this? > > thanks for all the advise! And I love the way the software works.... > > > Ok Steve Tonnessen who is on this list has been working on this and I think > he has a working marc importer running. > Ill cc him in this reply. I have a couple of schools who have been downloading marc records for their books and importing then one at a time. They check my automatic conversion of the marc records and make whatever modifications they need. Since the site they have been getting marc records from is Canadian, they have needed to modify the ISBN to the canadian publication's ISBN in every case anyway. I've also started working on Z39.50 support. Just signed the contract for access to the National Library of Canada's Z39.50 services today. I've got a perl module called Zeta working for Z39.50 searches, but it only seems to return SUTRS format (unstructured text) and not MARC format results. It seems that Net::Z3950 is a better module, but I haven't got it working yet. Also the YAZ toolkit also has a perl-based Z39.50 server that would really be nice to tie into Koha. LCCN should not be just overwritten on the dewey field, it should have a new field created for it. I guess the hitch with this is reworking the scripts for entering and editing books... Here comes some long rambling on my part... I think Koha really needs to be moved to a system that is at least partly based on Marc. I think that the records for each bibliography should be in MARC format, and that changes can be made either to the Koha Mysql tables, or the underlying MARC records and changes will be made in both places. The way that Koha stores records currently (broken into biblio and biblioitems) might make this a little difficult, but we could maybe associate particular MARC records with biblioitems, I guess, and then just have groups of MARC records associated with one biblio, as is done now. I'm thinking maybe about having one module that is used for modifying records. Something like: modifybibitem($env, $bibitemnumber, "author=Doe, Jane"); would change the author for biblioitem $bibitemnumber to Jane Doe. The modifybibitem routine would be responsible for modifying both the MARC record, and the MySQL tables. This all gets a little tricky when you consider that the MARC record stores a fair amount more information than the MySQL tables, for example, if a particular MARC record has a subject added entry like: 650 a Extinct animals x Encyclopedias, Juvenile. The corresponding entry in the MySQL bibliosubjects table would be: Extinct animals -- Encyclopedias, Juvenile Now if I edit the bibliosubjects entry to read: Extinct animals -- Encyclopedias It would be difficult to determine how to modify the MARC record automatically in a generic way. If the entry was changed to: Extinct animals -- dinosaurs It becomes even more difficult to incorporate the change back into the MARC record. I suppose the web interface could pop up a form saying something like, --- Need more information for MARC record modification: 650 a Extinct animals __ Encyclopedias --- with an 'x' as a suggestion for the text entry box. Steve Tonnesen Coast Mountains School District

5 6

Net::Z3950 module
by Tonnesen Steve 01 Jun '01

01 Jun '01

Anybody here have any experience with the perl Net::Z3950 module? I've got it doing successful searches of the National Library of Canada, but I can't figure out how to get raw MARC results out of it. I can get GRS1 results, or structured text results no problem, but I can't get MARC for some reason. When I use: print Net::Z3950::Record::USMARC::render($rec); I get the following error: Can't call method "output" on an undefined value at /usr/local/lib/site_perl/i386-linux/Net/Z3950/Record.pm line 236. Steve.

1 0

Linux...but WHICH Linix!
by COURYHOUSE＠aol.com 29 May '01

29 May '01

Good morning Folks! OK found an old Corel LINUX CD, loaded up one of the systems, have it on the network and talking, but I would like to hear from some of the other users out there which distribution of LINUX they have had the most luck with. Any input is great! thanks Ed Sharpe archivist for SMECC

1 0