[Koha] BOM in MARC 21 data files

[Chris Cormack]

Yes it isn't part of the standard. However vendors very rarely follow standards. Worse they often exhibit attitudes like this, that others should have to deal with cleaning up for them.
I'd estimate that 30% of the code in Koha is dealing with the fact that others don't follow standards :)

However we should be able to deal with the BOM without too much effort so we should do that.

Ill try to have a look over the Xmas break

Happy holidays all
Chris 

On 25 December 2017 8:37:22 AM NZDT, Michael Kuhn <mik at adminkuhn.ch> wrote:
>Hi
>
>Recently we have the problem some vendors are delivering their 
>UTF-8-encoded MARC 21 data files (ISO 2709) starting with a byte order 
>mark / BOM (see https://en.wikipedia.org/wiki/Byte_order_mark). Most 
>vendors I know (e. g. Overdrive) don't include such a BOM, but in 
>Germany some do. When asking them to change this they say "every ILS 
>should be able to handle this".
>
>However, when trying to import such a file into Koha the first record
>is 
>simply ignored. This is especially fatal when importing a data file 
>containing only one record, of course.
>
>1. As far as I know the BOM is not part of the MARC 21 (ISO 2709) 
>format, so there mustn't be any leading BOM. Am I correct?
>
>2. Is there a way within Koha to remove that unwanted BOM or to teach 
>Koha to ignore it and import the data anyway?
>
>Of course such a BOM could be manually removed by using sed, but I feel
>
>that a) either the vendor shouldn't include the BOM at all or b) Koha 
>"should be able to handle this".
>
>What do you think?
>
>Best wishes and happy holidays! Michael
>-- 
>Geschäftsführer · Diplombibliothekar BBS, Informatiker eidg.
>Fachausweis
>Admin Kuhn GmbH · Pappelstrasse 20 · 4123 Allschwil · Schweiz
>T 0041 (0)61 261 55 61 · E mik at adminkuhn.ch · W www.adminkuhn.ch
>_______________________________________________
>Koha mailing list  http://koha-community.org
>Koha at lists.katipo.co.nz
>https://lists.katipo.co.nz/mailman/listinfo/koha

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.