[Koha] Feb. 2024 possibility of Koha list delivery problems for Gmail, etc.

[Thomas Dukleth]

There is the possibility that users of large email services, such as
Gmail, Yahoo, etc. may have problems receiving some email sometime in
February for lack of correct implementation of anti-spam authentication
protocols.

1.  Adequate Corrections Mostly Prepared But Extra Caution for the Koha
General Mailing List.

The Koha mailing lists should have all been corrected in a way which may
be adequate while people work on setting up a new system for the mailing
lists.  However, the Koha general mailing is lagging behind good response
perhaps in the hope that another party to take over the list would have
taken over already, or perhaps it was partly fixed but someone forgot to
update the serial number and I had omitted the serial number from my
previous instructions.

The new email delivery policy at large email providers, such as Gmail etc.
could include some subscribers to the Koha general mailing list affecting
their ability to receive email from sources not properly authenticated. 
The new policy has distinct rules for both large volume senders which
could possibly be the mailing list with enough Gmail subscribers but the
Koha general mailing list may need to meet the large volume sender
requirement with DMARC support so that what is missing from the small
volume sender requirement continues to be given a pass.


2.  Related Issue for Koha Installations.

If mail servers for systems sending notices etc. to patrons do not have
all the expected configuration, patron recipients using Gmail, etc. may
also be affected but probably would have had problems previously.  It
would be prudent to confirm that the mail server sending messages is
reaching Gmail, etc. users without problem by having SPF and DMARC
settings in the DNS and applies DKIM signatures to messages.


3.  Effect of February Changes for Gmail, etc. Mailing List Subscribers.

We do not know what will happen if anything but people should be prepared
for the possibility of a disruption in receiving some email at large
providers, such as Gmail, etc. which might not appear in the spam box or
anywhere.  If mailing list messages appear in the spam box, a filter can
be added by people in their Gmail, etc. settings.  If messages stop
arriving even and do not even appear in the spam box the server sending
email for the mailing list will need more configuration.

Sending messages to the list will continue to work as will the mailing
list archives which can be read while we set up a new system.  If you are
unlucky enough to confirm that no mailing list messages are arriving for
you not even in your spam box and yet they appear in the mailing list
archive you may report that the issue affects this mailing list system
which they use, Gmail etc. in
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 .  The
mailing list would be shown to need more server configuration while people
work on setting up a new system.  Meanwhile, you might also try at least
temporarily resubscribing to the mailing list from a different email
provider with less strict authentication policies for delivering messages.

Mailing lists are not the primary target of the new stricter policies for
Gmail, etc. but they are included.

Mailing list messages would have failed long ago for Gmail, etc. except
that we have the advantage of having authentication attached to messages
from the original author forwarded via the mailing list.  They mostly
carry the DKIM signature from the original author.  Gmail has not recently
been validating DKIM signatures which would fail when sent from the
mailing list because of small changes when the message is resent by the
mailing list with the mailing list footer, etc.  Also, Gmail has recently
been passing messages with SPF and DMARC records in the originating server
DNS and no DKIM signatures.

Relying on authentication from the original author as opposed to the
mailing list forwarder is not correct for DMARC and DKIM and might now
fail unless mailing lists are treated a little more leniently at least
initially.  Library of Congress run mailing lists which I examined a few
days ago did not have DMARC mitigation for correct From header attribution
as from the author via the mailing list address but did have a DMARC DNS
record which is currently missing for the Koha general mailing list. 
Email originating from governments may have their own special rules for
delivery at Gmail, etc.

In my current testing, new subscription confirmation messages for the Koha
general list do not appear in the Gmail, not even in the spam box. 
However, mailing list message delivery to Gmail etc. may continue to have
a free pass from the authentication of the original author's message.

The need for DMARC support authenticating the mailing list as the actual
originator of messages on the mailing list as opposed to the message
author has been raised previously and it has been tested on the Koha-devel
mailing list and may now be implemented on all the Koha mailing lists
except for the general list.


4.  Please be Nice to the People Who Gave Koha to the World.

The Koha general mailing list is a special problem only because Katipo for
which we are all grateful for giving Koha to the world no longer has the
capacity to actively maintain the Koha general mailing list which has been
fine as the existing configuration was good for a very long time and did
not seriously require maintenance until recently.  Even though DMARC
support is a trivial matter of changing three lines and (maybe updating a
DNS serial number for 4 lines), it has not happened for Katipo in the past
few months since I have raised the issue.

It may be possible that people assisting Rachel Hamilton-Williams are not
certain where DNS is configured for lists.katipo.co.nz to update that
record for DMARC support.  DNS configuration could be at the domain
registrar, some intermediate service, some VPS hosting provider, or on the
very system which runs the katipo.co.nz server or lists.katipo.co.nz
server.

If using BIND for DNS line to add would be:

_dmarc.lists.katipo.co.nz. IN TXT "v=DMARC1; p=none"

 or the equivalent in some other system where the leading underscore is
needed and the policy is "p=none" matches the DNS configuration for the
the BibLibre managed Koha mailing lists such as the Koha-devel list.  If
using BIND, the zone file where  lists.katipo.co.nz is configured would
need a serial number update.  The BIND9 daemon would also restart.  Maybe
a change was made without a serial number update or daemon restart. 
There are two equally trivial Mailman configuration changes also needed
but a DNS update for DMARC comes first.

I have sent messages but response discussion was only about having another
party take over running the mailing list for more active maintenance which
would necessarily take time and the Koha community is pursuing a new
system which takes time.


5.  New System Fixes Everything Except New Problems.

Please be patient.

People are working on setting up, configuring, testing etc. a new system
which can give people email for doing everything and a message forum for
people who prefer forums instead of email which is important because
mailing list engagement is declining everywhere.  People need time from
the other more pressing tasks of every day to work on all that is
necessary for a fully working system.

I have weeks of research into various issues for configuration, bugs, some
tests etc.  If we rush things we may have much unhappiness with any of a
variety of common problems: from email not working properly;posts mangled;
message lists jumbled where distinctive content is difficult to find;
database backups silently corrupted and not restoring after a software
update; contented list readers having their accounts deleted after a time;
etc.  People need to take time to do things reasonably well so that
unpleasant surprises are minimised.

I expect that it may take several weeks to a few months for a new system
to be well setup, configured, tested, reconfigured, and retested before we
should have confidence in a new system informed by the problems which
other people had before us.  Meanwhile, fixing DMARC on Mailman 2.1 which
we are running is trivial by comparison.


6.  Reference.

See the message I posted on the Koha-devel mailing list about prospective
delivery problems especially the prospect for the Koha general mailing
list, "[Koha-devel] Feb. 2024 prospects of Koha lists delivery problems
for Gmail, etc." -
https://lists.koha-community.org/pipermail/koha-devel/2023-November/048441.html
.  [I resolved the ARC permissions problem mentioned in the message by
having the startup script change the permissions but with DMARC Gmail has
been satisfied in my testing even when DKIM and ARC are not present.]

Please report verified mailing list receipt problems in "Bug 34927 -
Adding DMARC compatibility to mailing lists" -
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34927 .  [Once
someone has reported that mailing list X messages are not being accepted
by subscribers using big service Y , not even in the spam box, we do not
need the same information again.  Not every user of some big mail service
may have the same experience because such large services do not tend to
implement changes worldwide all at the same time.]


Thomas Dukleth
Agogme
109 E 9th Street, 3D
New York, NY  10003
USA
http://www.agogme.com
+1 212-674-3783