Re: [Koha] Identifying Records with Non-Roman Characters
Hi Charles, I'm not 100% sure what you're asking here. Are you asking to find all records where there is a 245 title that isn't Romanized? You could try something like this: SELECT * FROM biblio WHERE title <> CONVERT(title USING latin1); I've tried that out on one of my multilingual libraries and it had some decent results. However, it's worth noting that it isn't a perfect solution. There are certain characters (I've noticed a particular hyphen) that won't have a latin1 equivalent, but they'd still be "Roman" (like hyphen which is made of the hexadecimal bytes E2 80 90, but is not to be confused with hyphen-minus which is ASCII and represented by the byte 2D). Other examples are emoji and other symbols like ↔️. You could then tweak the query or do a visual scan through to filter out any results that are irrelevant. Anyway, I hope that helps you advance your work. David Cook Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 -----Original Message----- Date: Mon, 7 Jun 2021 09:53:24 +0900 From: Charles Kelley <cmkelleymls@gmail.com> To: Discussion Group Koha <koha@lists.katipo.co.nz> Subject: [Koha] Identifying Records with Non-Roman Characters Message-ID: <CAM8F7wqwMav=yuhDjPM7r8ZDZKjzHWONtY397AxEc-TWw-Mb5w@mail.gmail.com> Content-Type: text/plain; charset="UTF-8" Hello, all! Our catalog has 8766 records that lack the 880 field for non-Roman titles. Examples: =245 10$aВеликая отецественная война 1941-1945 /$cВ. И. Чуйков =246 11$a Velikaia otechestvennaia voina 1941-1945 =245 10$a英汉常用物理学小词典 /$cYing Li, Lian Wei =246 11$aYing-Han changyong wulixue xiaocidian Of those, a significant portion is in Cyrillic, CJK, Greek, etc. Is there a way in either Koha or MarcEdit to isolate said records? The problem is to turn the 245 into 880, turn the 246 into 245 fields and then pair the resulting 880 and the new 245 into something like =880 10$6245-45/(N$aВеликая отецественная война 1941-1945 /$cВ. И. Чуйков. =245 10$6880-45$aVelikaia otechestvennaia voina, 1941-1945. =880 10$6245-45/[dollar]1$a英汉常用物理学小词典 /$cYing Li, Lian Wei. =245 10$6880-45$aYing-Han changyong wulixue xiaocidian. The indicators are easy to fix after the 245 and 880 fields are reconstructed. The 245 $c subfield can be copied from the 880 $c subfield. I'm just trying to automate the process as much as possible and thereby avoid the painstaking manual corrections if they can be avoided. This is an interim step. Eventually the catalog will have to be recataloged from scratch, I think. Many thanks, all! -- 気を付けて。 /ki wo tukete/ = Take care. -- Charles. Charles Kelley, MLS PSC 704 Box 1029 APO AP 96338 Charles Kelley Tsukimino 1-Chome 5-2 Tsukimino Gaadenia #210 Yamato-shi, Kanagawa-ken 〒242-0002 JAPAN +1-301-741-7122 [US cell] +81-80-4356-2178 [JPN cell] mnogojazyk@aol.com [h] cmkelleymls@gmail.com [p] linkedin.com/in/cmkelleymls <http://www.linkedin.com/in/cmkelleymls> Meeting Your Information Needs. Virtually.
Hello, David! In our latest exchange on 8 June 2021 at 13:33 [JST], I received from you the following: I'm not 100% sure what you're asking here. Are you asking to find all
records where there is a 245 title that isn't Romanized?
You could try something like this: SELECT * FROM biblio WHERE title <> CONVERT(title USING latin1);
I've tried that out on one of my multilingual libraries and it had some decent results.
However, it's worth noting that it isn't a perfect solution. There are certain characters (I've noticed a particular hyphen) that won't have a latin1 equivalent, but they'd still be "Roman" (like hyphen which is made of the hexadecimal bytes E2 80 90, but is not to be confused with hyphen-minus which is ASCII and represented by the byte 2D). Other examples are emoji and other symbols like ↔️.
You could then tweak the query or do a visual scan through to filter out any results that are irrelevant.
Anyway, I hope that helps you advance your work.
Yes, I'm trying to get the bibliographic records whose 245 field is not in the Latin alphabet. The 245 field in the non-Latin alphabet is immediately followed by a 246 in Latin transliteration. This solution may just work, even if it generates false positives and even if it reduces the count to 6500 records instead of the current 8766. Then I can carefully automate the process of changing 245 tag to 880 with $6 and the first 246 tag to 245 with $6. Thank you very much. -- 気を付けて。 /ki wo tukete/ = Take care. -- Charles. Charles Kelley, MLS PSC 704 Box 1029 APO AP 96338 Charles Kelley Tsukimino 1-Chome 5-2 Tsukimino Gaadenia #210 Yamato-shi, Kanagawa-ken 〒242-0002 JAPAN +1-301-741-7122 [US cell] +81-80-4356-2178 [JPN cell] mnogojazyk@aol.com [h] cmkelleymls@gmail.com [p] linkedin.com/in/cmkelleymls <http://www.linkedin.com/in/cmkelleymls> Meeting Your Information Needs. Virtually.
participants (2)
-
Charles Kelley -
dcook＠prosentient.com.au