We're seeing a large quantity of records filling the action_logs table... nearly 18,000 in the last 24 hours. This in turn uses a LOT of disk space... it was briefly over 90GB until I intervened, and because MySQL is dumb about shrinking tables we were fortunate I caught it in time. It looks like the update_totalissues.pl script is running every night and putting the entire biblio record into the info column in the table for every record we have. What could be causing this, and how can I stop it (without disabling the job completely, which I assume is part of Koha and important)? *Joel Coehoorn* Director of Information Technology *York University* Office: 402-363-5603 | jcoehoorn@york.edu | york.edu *Please contact helpdesk@york.edu <helpdesk@york.edu> for technical assistance.* The mission of York University is to transform lives through Christ-centered education and to equip students for lifelong service to God, family, and society
Hi Joel, We could skip the logging when the biblio is modified by this cronjob, please report this problem in a new bug report. Regards, Jonathan Le mar. 29 août 2023 à 21:22, Coehoorn, Joel <jcoehoorn@york.edu> a écrit :
We're seeing a large quantity of records filling the action_logs table... nearly 18,000 in the last 24 hours. This in turn uses a LOT of disk space... it was briefly over 90GB until I intervened, and because MySQL is dumb about shrinking tables we were fortunate I caught it in time.
It looks like the update_totalissues.pl script is running every night and putting the entire biblio record into the info column in the table for every record we have.
What could be causing this, and how can I stop it (without disabling the job completely, which I assume is part of Koha and important)?
*Joel Coehoorn* Director of Information Technology *York University* Office: 402-363-5603 | jcoehoorn@york.edu | york.edu
*Please contact helpdesk@york.edu <helpdesk@york.edu> for technical assistance.*
The mission of York University is to transform lives through Christ-centered education and to equip students for lifelong service to God, family, and society _______________________________________________
Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Do you use update_totalissues.pl with --incremental ? Le 30/08/2023 à 21:37, Jonathan Druart a écrit :
Hi Joel,
We could skip the logging when the biblio is modified by this cronjob, please report this problem in a new bug report.
Regards, Jonathan
Le mar. 29 août 2023 à 21:22, Coehoorn, Joel <jcoehoorn@york.edu> a écrit :
We're seeing a large quantity of records filling the action_logs table... nearly 18,000 in the last 24 hours. This in turn uses a LOT of disk space... it was briefly over 90GB until I intervened, and because MySQL is dumb about shrinking tables we were fortunate I caught it in time.
It looks like the update_totalissues.pl script is running every night and putting the entire biblio record into the info column in the table for every record we have.
What could be causing this, and how can I stop it (without disabling the job completely, which I assume is part of Koha and important)?
*Joel Coehoorn* Director of Information Technology *York University* Office: 402-363-5603 | jcoehoorn@york.edu | york.edu
*Please contact helpdesk@york.edu <helpdesk@york.edu> for technical assistance.*
The mission of York University is to transform lives through Christ-centered education and to equip students for lifelong service to God, family, and society _______________________________________________
Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________
Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
-- Fridolin SOMERS <fridolin.somers@biblibre.com> Software and system maintainer 🦄 BibLibre, France
I found the --incremental was missing. This is an installation that has evolved since before 2016, so I suspect it predates that addition. I've added it and am monitoring to see if growth is more reasonable. What I may submit as a bug report or feature request is an option to rotate these logs. So we'd add a background task to delete records older than some reasonable (possibly configurable) number. This would keep the table from growing indefinitely. *Joel Coehoorn* Director of Information Technology *York University* Office: 402-363-5603 | jcoehoorn@york.edu | york.edu On Thu, Aug 31, 2023 at 3:22 AM Fridolin SOMERS < fridolin.somers@biblibre.com> wrote:
Do you use update_totalissues.pl with --incremental ?
Hi Joel,
We could skip the logging when the biblio is modified by this cronjob, please report this problem in a new bug report.
Regards, Jonathan
Le mar. 29 août 2023 à 21:22, Coehoorn, Joel <jcoehoorn@york.edu> a écrit :
We're seeing a large quantity of records filling the action_logs
Le 30/08/2023 à 21:37, Jonathan Druart a écrit : table...
nearly 18,000 in the last 24 hours. This in turn uses a LOT of disk space... it was briefly over 90GB until I intervened, and because MySQL is dumb about shrinking tables we were fortunate I caught it in time.
It looks like the update_totalissues.pl script is running every night and putting the entire biblio record into the info column in the table for every record we have.
What could be causing this, and how can I stop it (without disabling the job completely, which I assume is part of Koha and important)?
*Joel Coehoorn* Director of Information Technology *York University* Office: 402-363-5603 | jcoehoorn@york.edu | york.edu
*Please contact helpdesk@york.edu <helpdesk@york.edu> for technical assistance.*
The mission of York University is to transform lives through Christ-centered education and to equip students for lifelong service to God, family, and society _______________________________________________
Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
-- Fridolin SOMERS <fridolin.somers@biblibre.com> Software and system maintainer 🦄 BibLibre, France _______________________________________________
Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Hi, You can use daily cron : cleanup_database.pl --logs DAYS https://git.koha-community.org/Koha-community/Koha/src/branch/master/misc/cr... Le 31/08/2023 à 06:17, Coehoorn, Joel a écrit :
I found the --incremental was missing. This is an installation that has evolved since before 2016, so I suspect it predates that addition. I've added it and am monitoring to see if growth is more reasonable.
What I may submit as a bug report or feature request is an option to rotate these logs. So we'd add a background task to delete records older than some reasonable (possibly configurable) number. This would keep the table from growing indefinitely.
*Joel Coehoorn* Director of Information Technology *York University* Office: 402-363-5603 | jcoehoorn@york.edu <mailto:jcoehoorn@york.edu> | york.edu <https://york.edu>
On Thu, Aug 31, 2023 at 3:22 AM Fridolin SOMERS <fridolin.somers@biblibre.com <mailto:fridolin.somers@biblibre.com>> wrote:
Do you use update_totalissues.pl <http://update_totalissues.pl> with --incremental ?
Le 30/08/2023 à 21:37, Jonathan Druart a écrit : > Hi Joel, > > We could skip the logging when the biblio is modified by this cronjob, > please report this problem in a new bug report. > > Regards, > Jonathan > > Le mar. 29 août 2023 à 21:22, Coehoorn, Joel <jcoehoorn@york.edu <mailto:jcoehoorn@york.edu>> a écrit : >> >> We're seeing a large quantity of records filling the action_logs table... >> nearly 18,000 in the last 24 hours. This in turn uses a LOT of disk >> space... it was briefly over 90GB until I intervened, and because MySQL is >> dumb about shrinking tables we were fortunate I caught it in time. >> >> It looks like the update_totalissues.pl <http://update_totalissues.pl> script is running every night and >> putting the entire biblio record into the info column in the table for >> every record we have. >> >> What could be causing this, and how can I stop it (without disabling the >> job completely, which I assume is part of Koha and important)? >> >> *Joel Coehoorn* >> Director of Information Technology >> *York University* >> Office: 402-363-5603 | jcoehoorn@york.edu <mailto:jcoehoorn@york.edu> | york.edu <http://york.edu> >> >> *Please contact helpdesk@york.edu <mailto:helpdesk@york.edu> <helpdesk@york.edu <mailto:helpdesk@york.edu>> for technical >> assistance.* >> >> >> The mission of York University is to transform lives through >> Christ-centered education and to equip students for lifelong service to >> God, family, and society >> _______________________________________________ >> >> Koha mailing list http://koha-community.org <http://koha-community.org> >> Koha@lists.katipo.co.nz <mailto:Koha@lists.katipo.co.nz> >> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha <https://lists.katipo.co.nz/mailman/listinfo/koha> > _______________________________________________ > > Koha mailing list http://koha-community.org <http://koha-community.org> > Koha@lists.katipo.co.nz <mailto:Koha@lists.katipo.co.nz> > Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha <https://lists.katipo.co.nz/mailman/listinfo/koha>
-- Fridolin SOMERS <fridolin.somers@biblibre.com <mailto:fridolin.somers@biblibre.com>> Software and system maintainer 🦄 BibLibre, France _______________________________________________
Koha mailing list http://koha-community.org <http://koha-community.org> Koha@lists.katipo.co.nz <mailto:Koha@lists.katipo.co.nz> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha <https://lists.katipo.co.nz/mailman/listinfo/koha>
-- Fridolin SOMERS <fridolin.somers@biblibre.com> Software and system maintainer 🦄 BibLibre, France
Great. Looks like adding --incremental has done the trick. Last night we only added 513 new records, and only a few of those were from the update_totalissues.pl script. Since it looks like there's already a log rotation option I can tweak, I think I'm in a good place now. If I were to still suggest any changes, it would be an option to _**not put the entire biblio record into the log**_ for that task, and instead maybe just log the prior issues counts. *Joel Coehoorn* Director of Information Technology *York University* Office: 402-363-5603 | jcoehoorn@york.edu | york.edu On Thu, Aug 31, 2023 at 5:38 PM Fridolin SOMERS < fridolin.somers@biblibre.com> wrote:
Hi,
You can use daily cron : cleanup_database.pl --logs DAYS
https://git.koha-community.org/Koha-community/Koha/src/branch/master/misc/cr...
Le 31/08/2023 à 06:17, Coehoorn, Joel a écrit :
I found the --incremental was missing. This is an installation that has evolved since before 2016, so I suspect it predates that addition. I've added it and am monitoring to see if growth is more reasonable.
What I may submit as a bug report or feature request is an option to rotate these logs. So we'd add a background task to delete records older than some reasonable (possibly configurable) number. This would keep the table from growing indefinitely.
*Joel Coehoorn* Director of Information Technology *York University* Office: 402-363-5603 | jcoehoorn@york.edu <mailto:jcoehoorn@york.edu> | york.edu <https://york.edu>
On Thu, Aug 31, 2023 at 3:22 AM Fridolin SOMERS <fridolin.somers@biblibre.com <mailto:fridolin.somers@biblibre.com>> wrote:
Do you use update_totalissues.pl <http://update_totalissues.pl> with --incremental ?
Le 30/08/2023 à 21:37, Jonathan Druart a écrit : > Hi Joel, > > We could skip the logging when the biblio is modified by this cronjob, > please report this problem in a new bug report. > > Regards, > Jonathan > > Le mar. 29 août 2023 à 21:22, Coehoorn, Joel <jcoehoorn@york.edu <mailto:jcoehoorn@york.edu>> a écrit : >> >> We're seeing a large quantity of records filling the action_logs table... >> nearly 18,000 in the last 24 hours. This in turn uses a LOT of disk >> space... it was briefly over 90GB until I intervened, and because MySQL is >> dumb about shrinking tables we were fortunate I caught it in time. >> >> It looks like the update_totalissues.pl <http://update_totalissues.pl> script is running every night and >> putting the entire biblio record into the info column in the table for >> every record we have. >> >> What could be causing this, and how can I stop it (without disabling the >> job completely, which I assume is part of Koha and important)? >> >> *Joel Coehoorn* >> Director of Information Technology >> *York University* >> Office: 402-363-5603 | jcoehoorn@york.edu <mailto:jcoehoorn@york.edu> | york.edu <http://york.edu> >> >> *Please contact helpdesk@york.edu <mailto:helpdesk@york.edu> <helpdesk@york.edu <mailto:helpdesk@york.edu>> for technical >> assistance.* >> >> >> The mission of York University is to transform lives through >> Christ-centered education and to equip students for lifelong service to >> God, family, and society >> _______________________________________________ >> >> Koha mailing list http://koha-community.org <http://koha-community.org> >> Koha@lists.katipo.co.nz <mailto:Koha@lists.katipo.co.nz> >> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha <https://lists.katipo.co.nz/mailman/listinfo/koha> > _______________________________________________ > > Koha mailing list http://koha-community.org <http://koha-community.org> > Koha@lists.katipo.co.nz <mailto:Koha@lists.katipo.co.nz> > Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha <https://lists.katipo.co.nz/mailman/listinfo/koha>
-- Fridolin SOMERS <fridolin.somers@biblibre.com <mailto:fridolin.somers@biblibre.com>> Software and system maintainer 🦄 BibLibre, France _______________________________________________
Koha mailing list http://koha-community.org < http://koha-community.org> Koha@lists.katipo.co.nz <mailto:Koha@lists.katipo.co.nz> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha <https://lists.katipo.co.nz/mailman/listinfo/koha>
-- Fridolin SOMERS <fridolin.somers@biblibre.com> Software and system maintainer 🦄 BibLibre, France _______________________________________________
Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
participants (3)
-
Coehoorn, Joel -
Fridolin SOMERS -
Jonathan Druart