#email

How to log full email messages into *.eml files on filesystem in your Laravel project? We would like to log full message bodies for all messages, no matter if they are sent out directly using Laravel's Mail facade or as Mail Notifications. That can actually be implemented in a super easy way, but it was so hard to find any good documentation or tutorial about this.

How nice would it be, if every email client could be auto-configured and the user would just need to enter his credentials? Entering his email address plus password, hit enter, done. How come nowadays people still need to care about incoming IMAP and outgoing SMTP servers, encryption (SSL/TLS vs. STARTTLS protocols), ports (for a regular end user this is just a number without any meaning, right?), and maybe even wondering what was the username if that differs from the actual email address?

You may tell me: Hey, we're in 2021, email is so old-school! Or you may tell me: Oh, don't you know there is a thing called auto-configuration? – Unfortunately, both statements are wrong. Email today is still widely used for serious communication (and yes, you're right, it should have been replaced long time ago by something better! But it didn't.), and there is such a thing called auto-configuration, but there is just no real standardized protocol that works for every mail client out there. There was no real development in that field during the last 10+ years, sadly.

So, let me present that small Python project Martin and I have been developing for Onlime GmbH back in Feb 2020 and which I have now finally published on GitLab.com: onlime/email-autoconf

At Onlime GmbH we have a mail infrastructure that consists of 3 mail servers: mx1 acts as primary MX server and provides SMTP as outgoing mail gateway for our customers. mx2 acts as secondary MX and fallback incoming mailserver. Incoming email from mx2 is forwarded to mx1 which does spam/antivirus filtering with Rspamd. Finally, the 3rd mailserver which is simply called mail acts as IMAP server and outgoing mailserver.

We allow customers to set up email forwardings on their domains. A forwarding address could have another customer email address as destination, but could also directly or indirectly (through another forwarding) point to an external email address. Spam filtering is done on mx1, but Rspamd actually just flags the email as ham/spam with a spam score and adds the X-Spamd-Result header with all symbols. The actual action is done on the final mailserver mail in the recipients mailbox via Sieve rule. The reason for this is that we want to let the customer define his own spam score (going from "minimal" to "radical" which maps to a spam score treshold) and what should happen with an email that got classified as spam (store it in Spam box or discard it directly).

At Onlime GmbH we have migrated the mail infrastructure in Dec 2019 from good old Spamassassin to Rspamd which greatly improved spam filtering. Rspamd offers a nice way of Bayesian learning in Rspamd statistical module. You can feed emails through rspamc learn_spam or rspamc learn_ham for manual spam/ham training to improve Bayes hit rate.

In the past, we have only internally used Bayesian training. Wouldn't it be nice to let all customers help us improve the Bayes filter / hit rate? Without even asking them to do so?

Years ago I wrote about the extensive sendmail wrapper with sender throttling, a pretty simple Perl script. It reliably provided throttling of the email volume per day by the sender’s original UID (user id). It also logged the pathes of scripts that sent emails directly via sendmail (e.g. via PHP’s mail() function). The main flaw in the original sendmail wrapper was security, though. As in Linux, every executable script must be readable by the user that calls it, the throttle table in MySQL was basically open and every customer could manipulate it. Every customer could raise his own throttling limit and circumvent it.

Today, I’m publishing my new sendmail-wrapper that is going to fix all the flaws of the previous version and add some nice extras. The new sendmail-wrapper is written entirely in PHP and does not require any external libraries. It is a complete rewrite and has pretty much nothing in common with the old Perl version.