[Koha] Out of memory when Koha starts due to opac-search.pl and 500.pl

[Mike Lake]

Hi all

I'm having serious problems with my Koha instance. It serves the OPAC 
for the Australian Speleological Federation.  We are currently on Koha 
23.11 on a Debian 10.13. The system has been running fine for ages.

I was getting errors from the OOM killer:

oom_reaper: reaped process 1554 (opac-search.pl), now anon-rss:0kB, 
file-rss:0kB, shmem-rss:0kB
opac-search.pl invoked oom-killer: 
gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, 
oom_score_adj=0
opac-search.pl cpuset=/ mems_allowed=0

So I shutdown Koha (took a while as I was out of memory)
systemctl stop koha-common.service

Rebooted the machine and when i bought Koha up:
systemctl start koha-common.service
Now I'm still getting 96 processes & errors taking all CPU and memory:

3620  R  /usr/bin/perl 
/usr/share/koha/opac/cgi-bin/opac/opac-imageviewer.pl
3622  R  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
3624  R  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
3625  D  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-search.pl
3627  R  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
3629  D  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-search.pl
3630  R  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
3633  D  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl

Actually its 96 x opac-search.pl + 57 x 500.pl

A reboot does not help. Every time I start Koha those processes appear 
and take all cores and memory.

I do see that the MariaDB is down. "Failed to start MariaDB 10.3.39 
database server."
Attempts to start it: systemctl start mariadb.service
give that error probably because I'm out of memory does to the 100 perl 
processes running.

A "systemctl stop koha-common.service" does not stop or end those 
opac-search.pl or 500.pl processes.

The /var/log/koha/opac/opac-error.log says:

[cgi:error] [pid 6911] [client 124.243.148.74:47310] End of script 
output before headers: 500.pl
[cgi:error] [pid 6959] [client 190.92.216.104:41580] End of script 
output before headers: opac-search.pl

Something is borked :-(   Help most welcome.

-- 
Mike
Sys Admin
Australian Speleological Federation