Hi to all. I've tried to import a file with 150mb and the the job progress don't leave 0%. There is an maximun file size to import or the operation is quite slow? Regards. Vitor Fernandes
I imported lots of big files when I did our migration. Sometimes they were very quick, sometimes very slow and I had to leave it for a few minutes. I think it depends on the Zebra indexing catching up. Andrew Buxton IDS, Brighton, UK -----Original Message----- From: koha-bounces@lists.katipo.co.nz [mailto:koha-bounces@lists.katipo.co.nz] On Behalf Of Vitor Fernandes Sent: 11 November 2010 12:14 To: koha@lists.katipo.co.nz Subject: [Koha] Stage MARC Records For Import Hi to all. I've tried to import a file with 150mb and the the job progress don't leave 0%. There is an maximun file size to import or the operation is quite slow? Regards. Vitor Fernandes _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha This message is for the addressee only and may contain privileged or confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any views or opinions expressed are solely those of the author and do not necessarily represent those of IDS. Institute of Development Studies at the University of Sussex, Brighton BN1 9RE Tel: +44 (0)1273 606261; Fax: +44 (0)1273 621202 IDS, a charitable company limited by guarantee: Registered Charity No. 306371; Registered in England 877338; VAT No. GB 350 899914
When i've sent the previous email, i was trying to import a 150mb file. Now the job progress is still 0%, which is very very slow... I'll try to run the script misc/stage_biblios_file.pl and see if it works... Thanks for your answers. Regards Vitor Fernandes 2010/11/11 Andrew Buxton <A.Buxton@ids.ac.uk>:
I imported lots of big files when I did our migration. Sometimes they were very quick, sometimes very slow and I had to leave it for a few minutes. I think it depends on the Zebra indexing catching up.
Andrew Buxton IDS, Brighton, UK
-----Original Message----- From: koha-bounces@lists.katipo.co.nz [mailto:koha-bounces@lists.katipo.co.nz] On Behalf Of Vitor Fernandes Sent: 11 November 2010 12:14 To: koha@lists.katipo.co.nz Subject: [Koha] Stage MARC Records For Import
Hi to all.
I've tried to import a file with 150mb and the the job progress don't leave 0%. There is an maximun file size to import or the operation is quite slow?
Regards. Vitor Fernandes _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
This message is for the addressee only and may contain privileged or confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any views or opinions expressed are solely those of the author and do not necessarily represent those of IDS.
Institute of Development Studies at the University of Sussex, Brighton BN1 9RE Tel: +44 (0)1273 606261; Fax: +44 (0)1273 621202 IDS, a charitable company limited by guarantee: Registered Charity No. 306371; Registered in England 877338; VAT No. GB 350 899914
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
The biggest that I imported were about 500 kb and they sometimes took a couple of minutes. I didn't try the Perl script. But if you are checking for duplicates then presumably you are constrained by the speed of indexing. Andrew -----Original Message----- From: Vitor Fernandes [mailto:vitorfernandes87@gmail.com] Sent: 11 November 2010 12:50 To: Andrew Buxton Cc: koha@lists.katipo.co.nz Subject: Re: [Koha] Stage MARC Records For Import When i've sent the previous email, i was trying to import a 150mb file. Now the job progress is still 0%, which is very very slow... I'll try to run the script misc/stage_biblios_file.pl and see if it works... Thanks for your answers. Regards Vitor Fernandes 2010/11/11 Andrew Buxton <A.Buxton@ids.ac.uk>:
I imported lots of big files when I did our migration. Sometimes they were very quick, sometimes very slow and I had to leave it for a few minutes. I think it depends on the Zebra indexing catching up.
Andrew Buxton IDS, Brighton, UK
-----Original Message----- From: koha-bounces@lists.katipo.co.nz [mailto:koha-bounces@lists.katipo.co.nz] On Behalf Of Vitor Fernandes Sent: 11 November 2010 12:14 To: koha@lists.katipo.co.nz Subject: [Koha] Stage MARC Records For Import
Hi to all.
I've tried to import a file with 150mb and the the job progress don't leave 0%. There is an maximun file size to import or the operation is quite slow?
Regards. Vitor Fernandes _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
This message is for the addressee only and may contain privileged or confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any views or opinions expressed are solely those of the author and do not necessarily represent those of IDS.
Institute of Development Studies at the University of Sussex, Brighton BN1 9RE Tel: +44 (0)1273 606261; Fax: +44 (0)1273 621202 IDS, a charitable company limited by guarantee: Registered Charity No. 306371; Registered in England 877338; VAT No. GB 350 899914
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
This message is for the addressee only and may contain privileged or confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any views or opinions expressed are solely those of the author and do not necessarily represent those of IDS. Institute of Development Studies at the University of Sussex, Brighton BN1 9RE Tel: +44 (0)1273 606261; Fax: +44 (0)1273 621202 IDS, a charitable company limited by guarantee: Registered Charity No. 306371; Registered in England 877338; VAT No. GB 350 899914
Vitor, The operation is very slow, and, in my experience, breaks when you try to import more than about 20MB worth of records through the web interface. I understand this has something to do with network conditions, so it's not a hard limit. The best way to import large amounts of data is to use the misc/ stage_biblios_file.pl script. That can handle as much data as you can throw at it. Also, there is an important issue when importing large files: there are certain circumstances when trying to stage a large file results in the database growing to about six times the size it needs to be to store that number of records. Unfortunately, I don't know exactly what the conditions are. The way I dealt with this ultimately was to clean all the failed imports, do a mysqldump of the entire database, drop the database, and recreate it. Good luck. Regards, Jared Camins-Esakov On Thu, Nov 11, 2010 at 7:14 AM, Vitor Fernandes <vitorfernandes87@gmail.com
wrote:
Hi to all.
I've tried to import a file with 150mb and the the job progress don't leave 0%. There is an maximun file size to import or the operation is quite slow?
Regards. Vitor Fernandes _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz http://lists.katipo.co.nz/mailman/listinfo/koha
-- Jared Camins-Esakov Freelance bibliographer, C & P Bibliography Services, LLC (phone) +1 (917) 727-3445 (e-mail) jcamins@cpbibliography.com (web) http://www.cpbibliography.com/
participants (3)
-
Andrew Buxton -
Jared Camins-Esakov -
Vitor Fernandes