[Koha] Taking patron images out of db into a directory

[dcook at prosentient.com.au]

Recently, I've been re-thinking the scalability of Koha, so it's interesting to hear about this from you, Mengu, as the person who I think runs the biggest Koha installation in the world. 

Originally, I thought putting images in the database was a bad idea, but it is an easy way to provide that image data over a network to multiple nodes and is part of an established community-supported application backup strategy. 

But fetching arbitrary blobs from a database could be slow and it could inflate Koha's metadata/transaction database. Storing image data in the database seems convenient - especially for small implementations - but is more problematic for large implementations.

The probable solution to me for a large implementation is a network file share or object storage (e.g. AWS S3, Ceph, Minio, OpenStack Swift, etc). But that puts more of a burden on system administrators to make sure things are backed up. Not a problem for large implementations but it would be for small implementations.

The ultimate solution to me would be a standard API (Koha::UploadedFiles I presume) in Koha with different storage backends. This is a common approach used by other FOSS systems. You can have drivers for database storage, file system storage, object storage - and you can choose the driver best suited to your situation with the default option being the most convenient for small implementations (ie probably database storage). 

Recently, we've locally provided an object storage integration to provide people with a robust low-cost method for storing documents in Koha, which seems to be working well.

Using a caching layer would be nice too, so loading the same page 2 times doesn't equal 2 high cost trips to the data storage. I've been using the IIIF Image Server Loris backed by Fedora Commons backed by AWS S3 and caching makes a world of difference for end users. 

Anyway, that's my 2 cents. Ideally, the work done on this would be generalized, so that it could be used for things like biblio images, documents, etc. (I suppose storing URIs might be a useful way to do it. That way you could use file:///local/disk, file:///network/disk, s3://mybucket... maybe mysql://..)

(I'd be interested too to hear about optimizations people have for serving image data. Starman won't perform this task as well as Nginx or Apache, but I think we need to have this data behind an authorization layer, which currently means Perl. But that's really a completely different discussion.)

We might also want to move this from koha to koha-devel listserv?

David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St
Ultimo, NSW 2007
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-----Original Message-----
Date: Thu, 9 Apr 2020 17:40:17 +0300
From: Mengu Yazicioglu <mengu at devinim.com.tr>
To: Koha <koha at lists.katipo.co.nz>
Subject: [Koha] Taking patron images out of db into a directory
Message-ID: <e332b7e3-6a35-c7ab-3ff4-0b603b44af9c at devinim.com.tr>
Content-Type: text/plain; charset=utf-8; format=flowed

Hi all,

We'd like to write a patch to move patron images out of db into a directory in the server.

By this change, we'd like to satisfy following.

1. The size of database backup will be decreased, by this way one may store db backups more.

2. Database performance will increase

3. It will be possible to put patron images in different disk in the server

4. Easily put images gathered from other systems like (university system, personnel system, etc)

What is your idea?

--
Mengü Yazıcıoğlu
Devinim Yazılım Eğitim Danışmanlık
Reşit Galip Cad. No: 29/6
Çankaya/ANKARA

Tel: (312) 446 86 68
Cep: (532) 701 90 27


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 484 bytes
Desc: not available
URL: <https://lists.katipo.co.nz/pipermail/koha/attachments/20200415/bbd6c20f/attachment.sig>