cover image

Restrict commands by SSH authorized_keys command option

August 3rd, 2021 by Philip Iezzi 5 min read

SSH authorized_keys allows you to define a command which is executed upon authentication with a specific key by prefixing it with the command="cmd" option.

authorized_keys man page explains this wonderful feature as follows:

cover image

Secure External Backup with ZFS Native Encryption

August 3rd, 2021 by Philip Iezzi 28 min read

Let's improve our Simple and Secure External Backup solution I have published back in 2018. Back then, I was using rsync over SSH to pull backup data, and LUKS encryption as full disk encryption for the external drives. As we all know, transferring data with rsync can get horribly slow and blow up your I/O if you're transferring millions of small files. Also, LUKS encryption may be a bit low level and inflexible. What we want to accomplish: A performant and secure backup solution based on ZFS, using zfs send|recv for efficient data transfer, and ZFS native encryption to secure our external drives. So let's go ahead and built that thing from scratch on a fresh 2021 stack!

cover image

APFS Encryption Status

July 28th, 2021 by Philip Iezzi 2 min read

macOS offers us a super simple way to fully encrypt an APFS formatted external drive: Right-click on the volume in the Finder and choose the option "Encrypt". Carbon Copy Cloner (CCC) also recommends this in its KB article Enabling encryption on a volume that will not contain an installation of macOS.

But macOS is a bit too transparent on that task. It simply starts encrypting the volume without any feedback or progress bar (WTF?). Even Disk Utility just shows the volume as APFS (Encrypted) - as it would have already completed full volume encryption.

cover image

MySQL FEDERATED Storage Engine and Replication

July 24th, 2021 by Philip Iezzi 6 min read

Back in Nov 2020, it got time to rethink the MySQL replication infrastructure for Airpane Controlpanel, our customer dashboard at Onlime GmbH. The whole application runs on a separate server, but an extract of mail account information (mail account credentials, mail mappings/forwardings) needs to get replicated to our 3 mail servers (mailsrv, mx1, mx2). At that time, I was using MySQL MASTER-SLAVE replication for a single database in a 4-node setup (1 master + 3 slaves).

For security reasons, I no longer wanted any mail server to have access to the binlog of the server that hosted our controlpanel (even though I already had that limited to a single database with mailserver related data only). I also wanted to reduce complexity a bit, just using MySQL replication for the 3 mail servers and propagating mailsrv to master.

The mailserver data is extracted from our controlpanel database by MySQL triggers (mainly AFTER INSERT, AFTER UPDATE, and BEFORE DELETE triggers) into a separate database mailsync. How to get whole mailsync data stored on the remote server mailsrv without using MySQL replication? I didn't want to care about this on application level.

That's where MySQL FEDERATED Storage Engine comes into play!

cover image

Recursively fetch dependent rows with mysqldump

July 5th, 2021 by Philip Iezzi 5 min read

How to fetch a row from a MySQL database recursively, going through all foreign key (FK) constraints and fetch all dependent rows as well? That's the question that bothered me during the last 20+ years as MySQL administrator. Isn't there a standard tool like some extended mysqldump that comes with that power? Short answer: No, there is no such tool. I gave up searching. It's just too complex to write a general-purpose tool that works for any kind of database schema.

cover image

Fail2ban persistent banning

May 18th, 2021 by Philip Iezzi 3 min read

If you are using Fail2ban, there is no standard recommended way to persistently ban IPs. Some people recommend to do this outside of Fail2ban, using e.g. iptables-persistent, which is actually super easy to install and configure. But let's say, we don't want to install any extras and want to accomplish the same with Fail2ban, as we already have fail2ban on every single host (which is a must!).

cover image

Block email forwarding spam with Rspamd

April 16th, 2021 by Philip Iezzi 4 min read

At Onlime GmbH we have a mail infrastructure that consists of 3 mail servers: mx1 acts as primary MX server and provides SMTP as outgoing mail gateway for our customers. mx2 acts as secondary MX and fallback incoming mailserver. Incoming email from mx2 is forwarded to mx1 which does spam/antivirus filtering with Rspamd. Finally, the 3rd mailserver which is simply called mail acts as IMAP server and outgoing mailserver.

We allow customers to set up email forwardings on their domains. A forwarding address could have another customer email address as destination, but could also directly or indirectly (through another forwarding) point to an external email address. Spam filtering is done on mx1, but Rspamd actually just flags the email as ham/spam with a spam score and adds the X-Spamd-Result header with all symbols. The actual action is done on the final mailserver mail in the recipients mailbox via Sieve rule. The reason for this is that we want to let the customer define his own spam score (going from "minimal" to "radical" which maps to a spam score treshold) and what should happen with an email that got classified as spam (store it in Spam box or discard it directly).