Need a full-featured toast notification component for your next Vue.js project? Don't look any further: Vue3-Toastify beats it all!

Previously I have tried Vue Toastification («Light, easy and beautiful toasts») and Mosha Vue Toastify («A light weight and fun Vue 3 toast or notification or snack bar») but they are both kind of outdated and abandoned projects with latest releases dating back to prehistoric 2021. Let me show you how to integrate great looking and easy to use toast notifications into your VILT stack!

I first started to build my own toast component but soon ended up struggling with challenging states and animations as those were my (minimal) requirements:

• Support multiple toasts
• Autoclose with progress bar (success)
• No auto-close on warning/error
• Styled for info/success/warning/error notifications, playing nicely with Tailwind
• Close on click
• Pause on hover
• Newest on top
• Bounce transition

Vue3-Toastify offers it all, built-in, fully customizable, and «beautiful by default»! The author is not lying – look at this great feature set and the demo. Even the documentation couldn't be more detailed and simply crystal clear!

But hey, it's just a toast! Does it deserve its own blogpost? Definitely! I am a huge fanboy of that component, how well it is designed and how easy it is to use. 👏

Integrate Vue3-Toastify into VILT

Let me now show you how to integrate Vue3-Toastify into a project built on VILT (Vue3, Inertia.js, Laravel, Tailwind) stack, bootstrapped with Jetstream.

Our goal is to fire flash messages from the backend, potentially more than one per request. The flash messages should be displayed in our frontend and stay open even when navigating to other pages. They should be easy closable, and the success messages should auto-close after some seconds.

Let's start by importing the CSS and loading the Vue3Toastify component in resources/js/app.js (stripped down to the bare metal):

import './bootstrap'
import 'vue3-toastify/dist/index.css'
import '../css/app.css'

import { createApp, h } from 'vue'
import { createInertiaApp } from '@inertiajs/vue3'
import { resolvePageComponent } from 'laravel-vite-plugin/inertia-helpers'
import Vue3Toastify from 'vue3-toastify'

createInertiaApp({
    resolve: (name) => resolvePageComponent(`./Pages/${name}.vue`, import.meta.glob('./Pages/**/*.vue')),
    setup({ el, App, props, plugin }) {
        return createApp({ render: () => h(App, props) })
            .use(plugin)
            .use(Vue3Toastify)
            .mount(el)
    },
})

Import vue3-toastify's CSS before your app.css, so you could override some styles there, e.g.:

:root {
    --toastify-toast-width: 400px;
    --toastify-color-warning: #ec9b00;
}

(see How to style docs for all available --toastify-* CSS variables)

You could now fire a toast message from anywhere in your JS code like this (see Usages):

import { toast } from 'vue3-toastify'

const options = {
  autoClose: 5000,
  type: toast.TYPE.SUCCESS,
  newestOnTop: true,
  theme: toast.THEME.COLORED,
  // and so on ...
} as ToastOptions;

// display toasts
const toastId = toast("Hello", options as ToastOptions);

//shortcut to different types
toast.success("Hello", options as ToastOptions);
toast.info("World", options as ToastOptions);
toast.warn(MyComponent, options as ToastOptions);
toast.error("Error", options as ToastOptions);

But we don't want to care about firing toasts in the frontend – just fire them whenever they pop in! Read on...

ToastNotifications Vue component

Let's now write our global ToastNotifications Vue component. First, create a composable resources/js/Composables/ToastNotifications.js with all the «business logic»:

import { ref, watch } from 'vue'
import { usePage } from '@inertiajs/vue3'
import { toast } from 'vue3-toastify'

const toasts = ref()

const reToasted = ref(false)

function fireToast(notification, sticky = false) {
    toast(notification.message, {
        toastId: notification.id,
        type: notification.type,
        newestOnTop: true,
        theme: toast.THEME.COLORED,
        autoClose: !sticky && (notification.type == 'success' ? 5000 : false),
        closeOnClick: !sticky,
        onClose: () => (reToasted.value = false),
    })
}

function fireToasts(sticky = false) {
    toasts.value?.forEach((notification) => fireToast(notification, sticky))
}

function toastAgain() {
    reToasted.value ? toast.remove() : fireToasts(true)
    reToasted.value = !reToasted.value
}

watch(
    // NOTE: Since Inertia.js 1.0.1, usePage() may return null initially.
    () => usePage()?.props?.flash.toasts,
    (newToasts) => {
        reToasted.value = false
        toasts.value = newToasts
        fireToasts()
    }
)

export function useToasts() {
    return {
        toasts,
        toastAgain,
        reToasted,
    }
}

This fires toast messages whenever the flash.toasts shared prop gets filled. To make sure we don't ever create duplicate toasts, let's give each toast a unique id. Whenever the watched usePage().props.flash.toasts changes, the new toasts get fired.

I am using the following helpers in bootstrap/helpers.php (loaded in composer.json's "autoload": { "files": [ "bootstrap/helpers.php" ], } where the toasts are getting added to the session:

if (! function_exists('toast')) {
    function toast(ToastType $type, string $message, ?RedirectResponse $response = null)
    {
        $toasts = session()->get('toasts', []);
        $toasts[] = [
            'id'      => Str::uuid(),
            'type'    => $type->value,
            'message' => $message,
        ];
        if ($response) {
            return  $response->with('toasts', $toasts);
        } else {
            session()->flash('toasts', $toasts);
        }
    }
}

if (! function_exists('toast_success')) {
    function toast_success(string $message)
    {
        return toast(ToastType::SUCCESS, $message);
    }
}

if (! function_exists('toast_warning')) {
    function toast_warning(string $message)
    {
        return toast(ToastType::WARNING, $message);
    }
}

if (! function_exists('toast_error')) {
    function toast_error(string $message)
    {
        return toast(ToastType::ERROR, $message);
    }
}

... and the related PHP enum – completely unnecessary but I like to favor enums over strings wherever I can:

namespace App\Enums;

enum ToastType: string
{
    case SUCCESS = 'success';
    case WARNING = 'warning';
    case ERROR = 'error';
}

Now, to make the toasts available to your frontend, add them to the Inertia shared data in HandleInertiaRequests middleware:

<?php

namespace App\Http\Middleware;

use Inertia\Middleware;

class HandleInertiaRequests extends Middleware
{
    public function share(Request $request)
    {
        return array_merge(parent::share($request), [
            'flash' => fn () => [
                'toasts'=> $request->session()->get('toasts'),
            ],
            // ...
        ]);
    }
}

And finally, build your ToastNotifications Vue component:

<script setup>
import { BellAlertIcon, BellSlashIcon } from '@heroicons/vue/24/outline'
import { useToasts } from '@/Composables/ToastNotifications'

const { toasts, toastAgain, reToasted } = useToasts()
</script>

<template>
    <div>
        <button
            v-if="toasts"
            class="flex h-10 w-10 items-center justify-center rounded-full bg-gray-100 hover:bg-gray-800 hover:text-white dark:bg-gray-700 dark:text-gray-300 dark:hover:bg-gray-600 dark:hover:text-white"
            :class="{ 'animate-pulse': !reToasted }"
            @click="toastAgain()"
        >
            <component :is="reToasted ? 'BellSlashIcon' : 'BellAlertIcon'" class="h-5 w-5" />
        </button>
    </div>
</template>

You don't need to get that fancy and could also just load the toasts somewhere else without an extra Vue component, or at least without the <template> part. But this component provides you the following:

• Shows nothing when there is no toast message
• Shows you a nice BellAlertIcon button when there are any toasts available (open or already closed but still on the same page)
• You can click on the bell-alert icon to re-open the previous messages again.
• When toasts are re-opened, they are not auto-closed (see sticky flag in ToastNotifications.js function fireToast()), but for easy closing of all toasts, you can just click on the bell-slash icon instead of closing each toast message individually.

Just put this component into your app header or main layout:

<template>
    <!-- ... -->
    <ToastNotifications />

done. 🎉

Oh, and in case you're missing some import statements in my Vue code, I love on-demand component auto-importing with unplugin-vue-components!

Usage in your Controllers

In your backend controllers, you can now simply use:

toast_success('User successfully updated!');

Or how about extending the RedirectResponse in your AppServiceProvider.php?:

class AppServiceProvider extends ServiceProvider
{
    // ...
    private function registerMacros()
    {
        RedirectResponse::macro(
            'withSuccess',
            fn (string $message) => toast(ToastType::SUCCESS, $message, $this)
        );

        RedirectResponse::macro(
            'withError',
            fn (string $message) => toast(ToastType::ERROR, $message, $this)
        );
    }

Like this, you could just append the message to the redirect response:

    public function update(UpdateCustomerRequest $request, Customer $customer)
    {
        $customer->update($request->validated());
        return back()->withSuccess('The customer was fully pimped.');
    }

I wish you a perfectly toasted 🍞 slice of bread that never falls on the wrong side! If it does, just put a ! in front.

Vue v-model is a directive that creates a two-way data binding between a value in our template and a value in our data properties, while v-bind only binds data one way. A common use case for using v-model is when designing forms and inputs.

While the core concept of v-model is quite straightforward, you might struggle with it when building advanced custom form components. Personally, this was my main pain point when learning Vue, especially since there are so many different ways this can be handled and tutorials diverge quite a bit.

Vue's official Components In-Depth > Events > Usage with v-model documentation only covers the most basic usage without providing any advanced examples. So I put together some use cases for you to learn more about v-model. The main question is: How can I make that custom component reusable so that it can be used as a form component with two-way data binding?

v-model directive explained

Vue's v-model directive does the following under the hood:

<MyChild v-model="someRef" />
<!-- same as -->
<MyChild
    :modelValue="someRef"
    @update:modelValue="someRef = $event"
/>

By default, the following applies to v-model:

• default prop name is modelValue
• default event is update:modelValue

This prop and event name will then be available in your child component.

The default prop and event name could be changed by appending the name with a colon: v-model:name, but let's stick with the default! Changing the name would only be needed if you would like to assign multiple independent v-model directives on the same component tag, which is very rarely used.

So, in every custom component which should support two-way data binding you need to implement the following:

• declare modelValue as a prop
• emit update:modelValue event when the input changes

Let's dive into a selection of 3 use cases which all use the recommended Vue 3 Composition API <script setup> syntax.

CustomInput form component

In this example, we are building a CustomInput form component that wraps the native <input> HTML element. It could then be used like this from our parent component:

<script setup>
import { ref } from 'vue'
import CustomInput from './CustomInput.vue'

const fullname = ref('')
</script>

<template>
    <form>
        <CustomInput v-model="fullname" />
    </form>
</template>

Our child component (basic CustomInput form component) looks like this:

<!-- CustomInput.vue -->
<script setup>
defineProps(['modelValue'])
defineEmits(['update:modelValue'])
</script>

<template>
    <input
        :value="modelValue"
        @input="$emit('update:modelValue', $event.target.value)"
    />
</template>

In this example, the data we are binding to our event is $event.target.value which is the actual String value of this input field.

Remember: Native HTML elements always fire Events, so you'll never get a primitive (like e.g. a Boolean or a String) as $event. That's why you need to retrieve the effective input element's value with $event.target.value in this case.

CustomSwitch (Toggle) form component

Let's create a slightly more sophisticated custom Switch (Toggle) form component, using Headless UI Switch component. Please note that the official Headless UI documentation provides us with a basic example where they use an enabled ref to store the current state:

<script setup>
import { ref } from 'vue'
const enabled = ref(false)
</script>

This extra ref is not needed inside our custom component, as the Switch component itself fires an event with a primitive Boolean that represents the current on/off state (see Switch Component API documentation, where v-model supports a Boolean). So, $event is simply the Boolean and we can pass it directly to our parent component with @update:modelValue="$emit('update:modelValue', $event)".

Side-note: In the example below, we are going to use the newer syntax without depending on the "magic" $event variable, so this is exactly the same in Vue 3 (see Component Events > Usage with v-model):

<Switch
    @update:modelValue="$emit('update:modelValue', $event)"
/>
<!-- same as -->
<Switch
    @update:modelValue="(newValue) => $emit('update:modelValue', newValue)"
/>

So our fully working (and minimally styled) CustomSwitch component looks like this:

<!-- CustomSwitch.vue -->
<script setup>
import { Switch } from '@headlessui/vue'

defineProps({
    modelValue: {
        type: Boolean,
        default: false,
    },
    label: {
        type: String,
        default: '',
    },
})

defineEmits(['update:modelValue'])
</script>

<template>
    <Switch
        :class="modelValue ? 'bg-primary-600' : 'bg-gray-300'"
        class="relative inline-flex h-6 w-11 items-center rounded-full"
        :modelValue="modelValue"
        @update:modelValue="(newValue) => $emit('update:modelValue', newValue)"
    >
        <span class="sr-only">{{ label }}</span>
        <span
            :class="modelValue ? 'translate-x-5' : 'translate-x-0'"
            class="inline-block h-5 w-5 transform rounded-full bg-white"
        />
    </Switch>
</template>

Slot props:

Please note, in this example, we use the modelValue prop directly for the current state of the Switch/Toggle. The Headless UI Switch component would offer a checked slot prop for that as well, but this would only be available inside the Switch component tag and not in e.g. any direct property of the Switch tag itself (e.g. class, where we would still depend on either modelValue or some other ref that represents the current state). The following could be done, but just makes our example more complex, as we would use checked slot prop in addition to the actual modelValue:

<template>
    <Switch
        v-slot="{ checked }"
        :class="modelValue ? 'bg-primary-600' : 'bg-gray-300'"
        ...
    >
        <span :class="checked ? 'translate-x-5' : 'translate-x-0'" />
    </Switch>
</template>

Immutable props:

And why can't we directly bind the modelValue to the Switch component using v-model="modelValue" inside our CustomSwitch component? Read this: Props: One-Way Data Flow

All props form a one-way-down binding between the child property and the parent one: when the parent property updates, it will flow down to the child, but not the other way around. This prevents child components from accidentally mutating the parent's state, which can make your app's data flow harder to understand. (...) This means you should not attempt to mutate a prop inside a child component. If you do, Vue will warn you in the console

So, never try to mutate a property in your custom component! Always work around it by only reading the prop value and firing events to pass the changed value back to the parent component, as we did in above example.

CustomListbox (Select) for component

Let's dive into another slightly more complex example of v-model two-way data binding, using Headless UI Listbox component. We can use this component to build custom, accessible select menus.

Options are passed in an object notation as options prop to our custom component:

<template>
    <ListboxSelect
        v-model="form.person"
        label="Person"
        :options="[
            { id: 1, name: 'Danny MacAskill' },
            { id: 2, name: 'Stefano Meloni' },
            { id: 3, name: 'Toni Peperoni' },
        ]"
    />
</template>

In our CustomListbox component, on the top level Listbox component, we store the current option in a selectedOption ref, using v-model. As the current option is an object (e.g. {id: 1, name: 'Danny MacAskill'}), we need to extract the id from it and only pass this Number to the parent component.

So, in below example, we need to store the currently selected option anyway in a new ref selectedOption. Like this, we can make use of v-model on Listbox component and then watch that ref for changes. On every change event, we emit the update:modelValue event and only extract the id from the selected option. Here's our fully working (yet unstyled) CustomListbox component:

<!-- CustomListbox.vue -->
<script setup>
import { ref, watch } from 'vue'
import { Listbox, ListboxLabel, ListboxButton, ListboxOptions, ListboxOption } from '@headlessui/vue'

const props = defineProps({
    modelValue: {
        type: Number,
        default: null,
    },
    label: {
        type: String,
        default: '',
    },
    options: {
        type: Array,
        required: true,
    },
})

const emit = defineEmits(['update:modelValue'])

const selectedOption = ref(props.options[0] ?? null)

watch(selectedOption, (option) => emit('update:modelValue', option.id), { immediate: true })
</script>

<template>
    <Listbox v-model="selectedOption">
        <ListboxLabel>{{ label }}:</ListboxLabel>
        <ListboxButton>{{ selectedOption.name }}</ListboxButton>
        <ListboxOptions>
            <ListboxOption v-for="option in options" :key="option" :value="option">
                {{ option.name }}
            </ListboxOption>
        </ListboxOptions>
    </Listbox>
</template>

I hope these examples gave you a better feeling for v-model.

Now, enjoy the power of magic reactivity by two-way data binding. Happy coding!

I'd like to share my recommended ESLint and Prettier configuration for a Vue.js 3 project. The two are not mutually exclusive – ESLint is for linting and Prettier handles code formatting. Believe me, you're going to use both! As preferred IDE, I am using VS Code. This article is just considered as simple dev notes without digging any deeper, and it's based on my current "best practices".

We're going to install Prettier first and ensure it's running fine. After that, we're going to install ESLint which is a bit trickier and needs some fine-tuning so that it won't conflict with any rules that Prettier already handles.

This is an update of my previous article Configure ESLint and Prettier for Vue/Nuxt.js, adapted to a plain Vue 3 project with recommended setup for a fresh 2022 start. It is mainly based on the following in-depth tutorial: ESLint and Prettier with Vite and Vue.js 3

Prettier

Configure Prettier

Install Prettier into devDependencies using npm (or use yarn add --dev instead):

$ npm install --save-dev prettier

If you also want Prettier to handle PHP files (this is a frontend dev article only, but since Inertia.js is such a great project and motivates us to build single-page apps in the same repo, you might also have such a mix of Vue/PHP files...):

$ npm install --save-dev @prettier/plugin-php

Configure Prettier in .prettierrc (again in your project root):

{
    "semi": false,
    "singleQuote": true,
    "tabWidth": 4,
    "printWidth": 120
}

Also create .prettierignore (optional):

# Ignore artifacts:
build
coverage

VS Code formatOnSave

Configure VS Code Editor: Format on Save to apply auto formatting upon saving, which you can also restrict to specific languages. I recommend only changing your workspace settings for the current project, which are stored in .vscode/settings.json:

{
    "vetur.validation.template": false,
    "vetur.format.defaultFormatter.html": "none",
    // Set the default
    "editor.formatOnSave": false,
    // Enable per-language
    "[javascript]": {
        "editor.formatOnSave": true
    },
    "[vue]": {
        "editor.formatOnSave": true
    },
    "[php]": {
        "editor.formatOnSave": true
    }
}

Now, open any .vue or .js (or .php) file, remove some spacing to make it look ugly, fire Cmd-S to save the file, and see the magic. If it won't yet work for you, consider restarting VSCode.

Don't worry, a file only gets formatted if you explicitely hit Cmd-S. You can still use autoSave for regular saving, which would not invoke auto-formatting of the whole file. That's the relevant global VS Code user settings (can be found in $HOME/Library/Application Support/Code/User/settings.json on macOS):

{
    "files.autoSave": "afterDelay",
    "prettier.printWidth": 81,
}

Configure ESLint

Install ESLint with all recommended plugins into devDependencies using npm:

$ npm install --save-dev eslint eslint-plugin-vue eslint-config-prettier
$ npm install --save-dev eslint-plugin-prettier

Next, configure ESLint by creating an .eslintrc.js file with the following configuration so that it's configured to use the ESLint recommend rules as well as the recommended rules for Vue 3. Minimal setup:

module.exports = {
  env: {
    node: true,
  },
  extends: [
    'eslint:recommended',
    'plugin:vue/vue3-recommended',
    'prettier',
  ],
  rules: {
    // override/add rules settings here, such as:
    // 'vue/no-unused-vars': 'error'
  }
}

NOTE: The really important part here is, that you don't forget to register eslint-config-prettier which turns off all rules that are unnecessary or might conflict with Prettier. Make sure it's the last config defined in the extends array as the order of the configs determine duplicate rules (later rules override previous ones and eslint-config-prettier only turns rules off)!

Visit the eslint-plugin-vue docs to see the other configurations available for use with Vue 3 for less strict options if desired. The different configurations correspond to the 3 different priorities in the Vue.js 3 style guide.

This is my recommended setup which makes use of eslint-plugin-prettier and defines some extra rules:

module.exports = {
    root: true,
    env: {
        browser: true,
        node: true,
    },
    extends: [
        'eslint:recommended',
        'plugin:vue/vue3-recommended',
        'prettier'
    ],
    plugins: ['prettier'],
    rules: {
        'prettier/prettier': ['error'],
        'vue/require-default-prop': 'off',
        'vue/html-indent': ['error', 4],
        'vue/singleline-html-element-content-newline': 0,
        'vue/component-name-in-template-casing': ['error', 'PascalCase'],
    },
    globals: {
        _: true,
    },
}

Feel free to disable some of my rules, e.g. not everybody likes to have component names inside <template> tag in PascalCase.

Happy clean coding!

Review

Above configuration is based on the following versions, added lines to package.json:

{
    "devDependencies": {
        "@prettier/plugin-php": "^0.17.6",
        "eslint": "^8.6.0",
        "eslint-config-prettier": "^8.3.0",
        "eslint-plugin-prettier": "^4.0.0",
        "eslint-plugin-vue": "^8.2.0",
        "prettier": "^2.5.1"
    }
}

Be ready for your configuration to break on the next major release of ESLint! It can be a pain, sometimes.

How to log full email messages into *.eml files on filesystem in your Laravel project? We would like to log full message bodies for all messages, no matter if they are sent out directly using Laravel's Mail facade or as Mail Notifications. That can actually be implemented in a super easy way, but it was so hard to find any good documentation or tutorial about this.

NOTE: Swift Mailer is no longer maintained since Nov 2021. In Laravel 9 LTS, which is going to be released on Jan 25th 2022, Swift Mailer is going to be swapped out with Symfony Mailer, see PR #38481. Below presented solution seems to be compatible with the new Mailer, though. I'll keep this post updated, if any changes are needed for Laravel 9.

Simple log channel implementation (DON'T)

In this Stackoverflow post, I have presented a simple way of logging email messages.

Create logdir logs/emails

Add an extra channel configuration to config/logging.php:

    'channels' => [
        // ...
        'emails' => [
            'driver' => 'single',
            'path' => storage_path('logs/emails/' . \Illuminate\Support\Str::uuid() . '.eml'),
            'level' => 'debug',
        ],

And then simply use the following in your .env:

MAIL_MAILER=log
MAIL_LOG_CHANNEL=emails

That way, you get wonderful logs/emails/<UUID>.eml files. But this solution has several drawbacks:

• There is no easy way of sending out emails both through MAIL_MAILER=log and some other mailer, e.g. the standard MAIL_MAILER=smtp
• Writing more than one message to the email log channel during the same run would not trigger an updated UUID, so the second message would get appended to the same logfile. This happens e.g. if you trigger a notification in Tinker.
• The first line (before Message-ID: ...) of the email header is prefixed with log formatting prefix [_TIMESTAMP_] local.DEBUG: . As a non-standard email header is ignored by all/most email clients, this should be no issue in development. But definitely not cool!
• The *.eml message body is scrambled, somehow. The MIME multipart content type boundary gets lost, so the email client has no idea where the HTML part of the message body starts and which part to treat as plaintext.

So, this solution is rejected! We need some slightly more sophisticated logging!

LogSentMessage event listener

Let's forget about the emails log channel from above solution and create the following disk configuration instead, config/filesystems.php:

return [
    // ...
    'disks' => [
        // ...
        'emails' => [
            'driver'     => 'local',
            'root'       => storage_path('logs/emails'),
        ],

Instead of changing the Mailer in .env (keep whatever MAIL_MAILER you are used to!), we build our LogSentMessage event listener and use Laravel's MessageSent event.

Create app/Listeners/LogSentMessage.php, which is going to write the full message:

<?php

namespace App\Listeners;

use Illuminate\Mail\Events\MessageSent;
use Illuminate\Support\Facades\Storage;
use Illuminate\Support\Str;

class LogSentMessage
{
    public function handle(MessageSent $event)
    {
        $messageId = $event->data['__laravel_notification_id'] ?? Str::uuid();
        Storage::disk('emails')->put(
            sprintf('%s_%s.eml', now()->format('YmdHis'), $messageId),
            $event->message->toString()
        );
    }
}

The LogSentMessage event listener can now be hooked into app/Providers/EventServiceProvider.php

<?php

namespace App\Providers;

use App\Listeners\LogSentMessage;
use Illuminate\Foundation\Support\Providers\EventServiceProvider as ServiceProvider;
use Illuminate\Mail\Events\MessageSent;

class EventServiceProvider extends ServiceProvider
{
    /**
     * The event listener mappings for the application.
     *
     * @var array
     */
    protected $listen = [
        MessageSent::class => [
            LogSentMessage::class,
        ],
    ];
}

To generate the listener class, you could simply add these lines to EventServiceProvider.php:

    protected $listen = [
+        MessageSent::class => [
+            'App\Listeners\LogSentMessage',
+        ],
    ];

... and then fire:

$ php artisan event:generate

This works great! Email notifications/messages are now getting sent out normally and at the same time a copy of the full message is stored here:

storage/logs/emails
├── 20211203131816_385ce6e8-437c-427e-828c-7464a4a48ca2.eml
└── 20211203132656_757c62b6-b6fa-45d0-99e3-6caf5be6a9f3.eml

You can open or even import them with any mail client and they are just 100% valid emails, including all headers (also Bcc). Just make sure this storage path is never publicly exposed.

Thanks @davewood for this simple trick.

Recently, one of our shared hosting webservers at Onlime GmbH got hit by a DoS attack. The attacker started a larger vulnerability scan against common Wordpress security issues. We already had common brute-force attack patterns on Wordpress covered by a custom Fail2Ban jail, which mainly trapped POST requests to xmlrpc.php or wp-login.php (the usual dumb WP brute-force attacks...). But this DoS attack had hundreds of customer sites as target and did not get trapped by our existing rules.

After having blocked the attacker's IP (glad this was no large-scale DDoS!), I wrote an extra Fail2Ban jail which traps such simple DoS attacks. It's a very basic Fail2Ban jail that should cover common attacks and should not cause any false positives as it is only getting triggered by a large amount of failed GET requests.

There are other good articles about setting up such Fail2Ban jails to block simple DoS, but they didn't quite fit our needs:

• Using fail2ban to mitigate simple DOS attacks against apache (or why I am a terrible sysop)
• Install fail2ban to protect your site from DOS attacks

Requirements

What we would like to accomplish:

• Scanning of all Apache access logs
• Ban the attackers IP if there are more than 300 GET requests during a time span of 5 mins resulting in HTTP non-200 (OK) status codes: 401, 403, 404, 503
• Ban the attacker for 1 hour

We're going to scan all customer Apache access.log logfiles which are in a slightly tuned (non-standard) combined log format:

#LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t %I %O \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %V %p" combined

Below failregex also works for default extended/combined or common Apache LogFormat.

Setup Fail2Ban jail

Let's call the new Fail2Ban jail apache-get-dos and create the following two files:

[apache-get-dos]
enabled  = true
port     = http,https
filter   = apache-get-dos
logpath  = /var/www/*/logs/access.log
datepattern = %%d/%%b/%%Y:%%H:%%M:%%S %%z
maxretry = 300
findtime = 5m
bantime  = 1h

In above jail, we didn't specify a banaction, as we're using the global default. You might want to put the following in jail.d/custom.conf:

jail.d/custom.conf

[DEFAULT]
bantime  = 300
findtime = 300
banaction = iptables-allports

# Fail2Ban filter to scan Apache access.log for DoS attacks

[INCLUDES]
before = common.conf

[Definition]
# Option:  failregex
# Notes.:  regex to match GET requests in the logfile resulting in one of the
#          following status codes: 401, 403, 404, 503.
#          The host must be matched by a group named "host". The tag "<HOST>" 
#          can be used for standard IP/hostname matching and is only an alias for
#          (?:::f{4,6}:)?(?P<host>[\w\-.^_]+)
# Values:  TEXT
failregex = ^<HOST> .*"GET (?!\/robots\.txt).*" (401|403|404|503)\s

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
#
ignoreregex =

You might want to tune the failregex to your needs. For our use case, I have created a regex101 with some sample log lines which you can play around with. It will ignore any failing GET /robots.txt requests (as most bots request this) and it will not care about "sane" HTTP status codes like 200 (OK), 301 (Moved Permanently), 302 (Found). So, please be aware, that this jail only protects you from some kind of DoS attack that targets a large amount of non-existing URLs/pathes, which is mostly the case for malicious vulnerability scans.

Monitoring jail

After having restarted Fail2Ban, you might want to monitor the new apache-get-dos jail:

# list status of apache-get-dos jail with monitored Apache logs
$ fail2ban-client status apache-get-dos
Status for the jail: apache-get-dos
|- Filter
|  |- Currently failed: 80
|  |- Total failed: 290517
|  `- File list:    /var/www/<user>/logs/access.log ...
`- Actions
   |- Currently banned: 0
   |- Total banned: 5
   `- Banned IP list:   

# review banning in fail2ban.log
$ grep -Pi 'apache-get-dos\] (un)?ban' /var/log/fail2ban.log
2021-09-14 17:45:47,285 fail2ban.actions        [107006]: NOTICE  [apache-get-dos] Ban 1.2.3.4
2021-09-14 18:45:45,503 fail2ban.actions        [107006]: NOTICE  [apache-get-dos] Unban 1.2.3.4

In case you have designed your failregex too aggressively and have banned too many IPs, remember those commands:

# unban an IP from a specific jail
$ fail2ban-client set <JAIL> unbanip <IP>

# unban an IP (or several) from all jails
$ fail2ban-client unban <IP> ... <IP>

# unbans all IP addresses (in all jails and database)
$ fail2ban-client unban --all

Bonus Commands

I'll provide you with some bonus commands to further investigate such DoS attacks:

# Top 20 list of IPs with a large amount of connections to the server
$ netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head -n20

You may also want to scan all Apache access logs for the IPs with most requests:

# Top 10 IP list for ALL sites for previous hour
$ grep -h "\[$(date -d -1hour +'%d/%b/%Y:%H:')" /var/www/*/logs/access.log | cut -d' ' -f1 | sort | uniq -c | sort -nr | head -n10

# Top 10 IP list for ALL sites for current hour
$ grep -h "\[$(date +'%d/%b/%Y:%H:')" /var/www/*/logs/access.log | cut -d' ' -f1 | sort | uniq -c | sort -nr | head -n10
 
# Top 10 IP list for ALL sites with filename
$ grep "\[$(date +'%d/%b/%Y:%H:')" /var/www/*/logs/access.log | cut -d' ' -f1 | sort | uniq -c | sort -nr | head -n10

I recommend you block such IPs directly on your front-end firewall. But in case you can't, use iptables to quickly block a single IP:

# Block a single IP
$ iptables -A INPUT -s <IP> -j DROP
# Unblock it
$ iptables -D INPUT -s <IP> -j DROP

And now, cross fingers you won't get hit by a brutal DDoS. I plan to cover DDoS attack mitigation in a future article.

Today, shit happened on a larger on-premise GitLab EE instance of one of our Onlime GmbH customers. GitLab's production.log started to fill up with PG::Error (FATAL: the database system is in recovery mode) errors which were somehow related to LFS operations. That definitely didn't sound cool and smelled like data corruption. The customer noticed it by failed CI jobs with 500 Internal Server Errors, and let me know immediately.

As we have that GitLab server running in a LXC container on a ZFS based system (Proxmox VE), it was easy to pull a clone of the full system and play around with PostgreSQL data recovery before working on live data. I decided to go for a full data restore by dumping and loading it from scratch in a freshly initialized PostgreSQL data dir.

So these were the FATAL errors I found in /var/log/gitlab/gitlab-rails/production.log (this is a Omnibus install of GitLab EE):

Processing by Repositories::LfsStorageController#download as HTML
  Parameters: {"repository_path"=>"group/project.git", "oid"=>"38387ae38..."}
PG::Error (FATAL:  the database system is in recovery mode
):

Started GET "/group/project.git/gitlab-lfs/objects/372ea928b2..." for 192.168.x.x at 2021-09-01 09:19:31 +0200
ActiveRecord::ConnectionNotEstablished (FATAL:  the database system is in recovery mode
):

The errors were always preceded by some LFS operation, so that looks correlated.

UPDATE 2021-09-01 23:50 – In the meantime (after the full recovery below - Shame on me ignorant sysadmin!), I have figured out this was actually caused by an OOM issue of that LXC container, where random PostgreSQL processes were killed by the system's oom-killer...

# /var/log/gitlab/postgresql/current
2021-09-01_07:19:31.05398 LOG:  server process (PID 10068) was terminated by signal 9: Killed

# /var/log/syslog
Sep  1 09:19:31 git kernel: [5068062.072130]  oom_kill_process.cold.33+0xb/0x10

But I still keep this article to explain how to do a full PostgreSQL data recovery, even though we never had any data corruption. :)

Check PostgreSQL Tools

First, check your PostgreSQL version and figure out where GitLab Omnibus install put all those commands:

$ gitlab-psql --version
psql (PostgreSQL) 12.6

$ gitlab-psql -c 'select version()'
PostgreSQL 12.6 on x86_64-pc-linux-gnu, ...

Check database sizes:

$ gitlab-psql -c 'SELECT pg_database.datname as "dbname", pg_database_size(pg_database.datname)/1024/1024 AS size_in_mb FROM pg_database ORDER by size_in_mb DESC'
       dbname        | size_in_mb 
---------------------+------------
 gitlabhq_production |      29151
 template0           |          7
 template1           |          7
 postgres            |          7

You'll find the following symlink:

/usr/bin/gitlab-psql -> /opt/gitlab/bin/gitlab-psql

The other commands are here:

/opt/gitlab/embedded/bin/pg_dump
/opt/gitlab/embedded/bin/pg_dumpall

But DON'T try to do a dump as root, always do it under user gitlab-psql:

$ su - gitlab-psql

(gitlab-psql)$ echo $PWD           
/var/opt/gitlab/postgresql
(gitlab-psql)$ which pg_dumpall
/opt/gitlab/embedded/bin/pg_dumpall

# connect over socket
(gitlab-psql)$ pg_dumpall -h /var/opt/gitlab/postgresql
# or connect over port
(gitlab-psql)$ pg_dumpall -h localhost -p 9187

If you're unsure about the port your PostgreSQL is running, check netstat -tulpen (don't need to visit the Netherlands for that!) and grab it from there. If you don't find any running postgres process that binds to some port, try to connect to the socket directly with -h /var/opt/gitlab/postgresql.

For below recovery, consult the following PostgreSQL 12 documentation:

• pg_dumpall
• initdb
• PostgreSQL Backup and Restore

Data Recovery Migration

The Idea

1. Stop all GitLab services, only keep PostgreSQL running
2. Make a full data dump with pg_dumpall
3. Stop PostgreSQL
4. Move current PostgreSQL data dir away, so we can start from scratch
5. Re-initialize PostgreSQL data dir with initdb
6. Start PostgreSQL
7. Restore full data dump by feeding it to gitlab-psql
8. Stop all GitLab services, reconfigure GitLab and restart all services

Dump / Restore

Stop all GitLab services and only keep PostgreSQL running:

$ gitlab-ctl stop
$ gitlab-ctl start postgresql
$ gitlab-ctl status

# You might also want to stop cron, so it won't interfere your migration
$ systemctl stop cron && systemctl disable cron
# Don't forget to start and re-enable it again after migration!

To repeat (I found this quite confusing as PostgreSQL does not always bind to a port):

After starting postgresql, always use netstat -tulpen to check on which port it is running. If it's not running on a port, connect to the socket with -h /var/opt/gitlab/postgresql instead of -h localhost -p <PORT>, see examples below.

Now, make full gzipped dump with pg_dumpall:

$ su - gitlab-psql
(gitlab-psql)$ pg_dumpall -h /var/opt/gitlab/postgresql | gzip > pg_dumpall.sql.gz

Re-create PostgreSQL data dir /var/opt/gitlab/postgresql/data from scratch with initdb:

$ gitlab-ctl stop postgresql

$ su - gitlab-psql
(gitlab-psql)$ mv data data.12.6.BKUP
(gitlab-psql)$ initdb data

$ gitlab-ctl start postgresql

($PWD of gitlab-psql user is already /var/opt/gitlab/postgresql, so you don't need to use full pathes in that context)

Restore dump:

$ su - gitlab-psql
(gitlab-psql)$ zcat pg_dumpall.sql.gz | gitlab-psql -h localhost -p 5432 -d postgres

NOTE:

SQL Dump: Using pg_dumpall: Actually, you can specify any existing database name to start from, but if you are loading into an empty cluster then postgres should usually be used.

Stop postgresql and reconfigure GitLab, then start all GitLab services:

$ gitlab-ctl stop postgresql
$ gitlab-ctl reconfigure
$ gitlab-ctl start

Monitor logs:

$ tail -f /var/log/gitlab/gitlab-rails/production.log /var/log/gitlab/postgresql/current

All good!

This TechBlog is based on Vue.js & Nuxt.js, using nuxt/content as a Git-based headless CMS, Tailwind CSS for styling. Today, we want to talk about full-text search integration. nuxt/content actually has a built-in search which indexes your pages/articles and does full-text lookups that are super easy to integrate. But I had quite a bad experience with it, content search delivering wrong search results (that did not even contain the query string or anything similar at all), and making it hard to extract a snippet of surrounding words/sentences of the search results.

Algolia offers a super powerful and flexible search with ready to use InstantSearch UI components for React and Vue. It was quite easy to write my own AlgoliaSearch component for this blog, but Algolia's InstantSearch defaults were so horribly resource-hungry and it took me a while to figure out how to fine-tune this.

AlgoliaSearch Vue Component

In our AlgoliaSearch component, we're going to use the following Vue InstantSearch UI components:

• ais-instant-search
• ais-configure
• ais-autocomplete
• ais-highlight
• ais-snippet

The following tutorials are great and give you in-depth instructions of how to set up such a search component:

• Create a keyboard navigable site search with Nuxt/Content and Aloglia vue-instantsearch
• How to add Algolia Search to NuxtJS

... so I try to keep it short here and just repeat the basic setup which is based on those articles. If you already have such a component set up, scroll down to the second part, how to fine-tune Algolia for a better search performance without the clutter and waste of resources.

Here's the full code: demo/nuxt-content-blog

And here's the DemoBlog (a stripped-down version of this blog, so you better understand)

The Nuxt project initially was created as follows:

$ yarn create nuxt-app algolia-vue-instantsearch
? Project name: algolia-vue-instantsearch
? Programming language: JavaScript
? Package manager: Yarn
? UI framework: Tailwind CSS
? Nuxt.js modules: Content - Git-based headless CMS
? Linting tools: ESLint, Prettier
? Testing framework: None
? Rendering mode: Universal (SSR / SSG)
? Deployment target: Static (Static/Jamstack hosting)
? Development tools: jsconfig.json (Recommended for VS Code if you're not using typescript)
? Continuous integration: None
? Version control system: Git

Install the required packages:

• algoliasearch, vue-instantsearch: Algolia Vue InstantSearch
• nuxt-content-algolia: Sync Nuxt content to Algolia index during nuxt generate
• remove-markdown: Remove Markdown from content body text for Algolia search index.
• v-click-outside: Vue directive to react on clicks outside an element, used to hide search results.

$ yarn add algoliasearch nuxt-content-algolia vue-instantsearch
$ yarn add remove-markdown v-click-outside

Create custom plugin in plugins/vue-instantsearch.js:

import Vue from 'vue'
import InstantSearch from 'vue-instantsearch'

Vue.use(InstantSearch)

Register custom plugin and set transpile build configuration:

export default {
    // ...
    plugins: [
        '~/plugins/vue-instantsearch'
    ],
    build: {
        transpile: [
            'vue-instantsearch',
            'instantsearch.js/es',
        ],
    },
    // ...
}

Configure nuxt-content-algolia to send index to Algolia during build (nuxt generate):

export default {
    // ...
    buildModules: [
        'nuxt-content-algolia',
    ],
    nuxtContentAlgolia: {
        appId: process.env.ALGOLIA_APP_ID,
        // !IMPORTANT secret key should always be an environment variable
        // this is not your search only key but the key that grants access to modify the index
        apiKey: process.env.ALGOLIA_API_KEY,
        // relative to content directory - each path get's its own index
        paths: [
            {
                name: 'articles',
                index: process.env.ALGOLIA_INDEX || 'articles',
                fields: ['title', 'description', 'bodyPlainText', 'tags'],
            },
        ],
    },
    // ...
}

Markdown formatting/tags are not needed for search. So, we create a bodyPlainText using remove-markdown package, nuxt.config.js:

export default {
    // ...
    hooks: {
        'content:file:beforeInsert': (document) => {
            if (document.extension === '.md') {
                const removeMd = require('remove-markdown')
                document.bodyPlainText = removeMd(document.text)
            }
        },
    },
    // ...
}

Create .env file and set required variables - put your Algolia app ID and search-only key there:

ALGOLIA_INDEX=dev_articles
ALGOLIA_APP_ID=ABCDE12345
#ALGOLIA_API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
ALGOLIA_SEARCH_ONLY_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
ALGOLIA_HITS_PER_PAGE=5
ALGOLIA_QUERY_BUFFER_TIME=300

NOTE: Copy .env.example from my demo/nuxt-content-blog to .env to start with.

WARNING: Make sure, you never store ALGOLIA_API_KEY (the private API key) in your git repo. Only use it in your dev environment's .env and NEVER put it in your production .env. If you deploy to your production host with GitLab CI/CD, you can easily set ALGOLIA_API_KEY as an environment variable only, as this is only needed by nuxt-content-algolia in build step on nuxt generate.

I also recommend setting up 2 separate indices at Algolia (same account), dev_articles for development and prod_articles for production. So you would put ALGOLIA_INDEX=prod_articles into your production .env.

The skeleton of AlgoliaSearch Vue component should then look like this (find full component here: AlgoliaSearch.vue):

<template>
    <ais-instant-search>
        <ais-configure />
        <ais-autocomplete>
            <div slot-scope="{ currentRefinement, indices, refine }">
                <!-- search input field -->
                <input @input="refine($event.currentTarget.value)" />
                <!-- search results -->
                <div v-if="currentRefinement.length">
                    <div v-for="section in indices" :key="section.objectID">
                        <NuxtLink v-for="(hit, index) in section.hits" :key="hit.objectID">
                            <ais-highlight attribute="title" :hit="hit" />
                            <ais-snippet attribute="bodyPlainText" :hit="hit" />
                        </NuxtLink>
                    </div>
                    <ais-powered-by />
                </div>
            </div>
        </ais-autocomplete>
    </ais-instant-search>
</template>

Finally, insert AlgoliaSearch Vue component in your layout:

<template>
  <header>
    <AlgoliaSearch />
  </header>
</template>

full code here: demo/nuxt-content-blog

InstantSearch fine-tuning

If you copied the Search component from the original article, and have signed up for an Algolia Search account, the default search experience is quite bad. Not performance-wise – Algolia keeps its promise and delivers search responses in < 30ms. But look at this waste of resources! ...

• By default, the autocomplete search field fires a POST request to Algolia's servers on every single keystroke!
• By default, InstantSearch sends an initial request to Algolia’s servers with an empty query (see Conditional Requests). Algolia explains it with: "This connection helps speed up later requests." – But for a regular use-case, this will just increase transferred data over the network and does not improve search speed at all (the initial response just contains the last x articles, anyway).
• By default, InstantSearch responds with all attributes, also returning full article content which might be several KB per article. As Algolia even duplicates the whole content in its JSON response for highlighting, it will even double the transferred data.
• If you accidentally wrap the ais-configure tag around your ais-autocomplete component (which works perfectly fine and was even recommended by the original article), this would blow up your initial POST request again, triggering 2 requests instead of just 1.
• If you add your AlgoliaSearch component at multiple place, e.g. in your regular screen header and in your mobile menu, that would double the initial requests to Algolia again.

I managed to fire 4 POST requests to Algolia's servers on initial page load without even touching the search input field. Any consecutive search query fired several requests, one for each keystroke. Like this, I managed to already use up half of my free 10'000 requests/mo in the first two days of integrating the search into this blog, without even publishing it yet. And when I checked Algolia's responses in the developer console, I noticed each response is delivering over 10KB of JSON data. This is insane!

Please, Algolia, stop beeing so greedy and provide us with some sane defaults that are not so resource-hungry! Think about the climate and not just about monetizing your services!

I have managed to fine-tune it to the following:

• No more initial request getting fired to Algolia. If the visitor does not submit a search query, no request!
• Autocomplete search field is now throttling your keystrokes. It will just fire a search query after 300ms, so if you type fast enough, there will be only a single search request fired to Algolia's servers!
• Full-text content bodyPlainText is no longer delivered in Algolia's search response.
• Full-text content bodyPlainText is no longer duplicated for highlighting (which anyway we don't need) in Algolia's search response.

That's how I did:

Avoid Initial Request

Algolia obtrusively makes you send an initial POST request with response data you will probably (in most use cases) never need. There is no way to turn this off by configuration, there is only this hint here: Conditional Requests > Detecting empty search results and for a real workaround you would need to deeply dig into this sample code.

So, this is my solution:

export default {
    computed: {
        searchClient() {
            const algoliaClient = algoliasearch(
                this.$config.algoliaAppId,
                this.$config.algoliaSearchOnlyKey
            )
            return {
                ...algoliaClient,
                search(requests) {
                    if (requests.every(({ params }) => !params.query)) {
                        return Promise.resolve({
                            results: requests.map(() => ({
                                hits: [],
                                nbHits: 0,
                                processingTimeMS: 0,
                            })),
                        })
                    }
                    return algoliaClient.search(requests)
                },
            }
        },
    },
}

I put this into a computed function, as we want to have process.env or this.$config available for ALGOLIA_APP_ID / ALGOLIA_SEARCH_ONLY_KEY env var lookups.

You may then pass searchClient to the ais-instant-search component:

<template>
    <ais-instant-search
        :search-client="searchClient"
        :index-name="$config.algoliaIndex"
        :search-function="searchFunction"
    >
    ...

Throttle Search Submission

In above snippet, you might have noticed I have provided a searchFunction to the ais-instant-search component. This is needed to debounce the search submission so that we're not going to fire a search request on every keystroke.

Please read ais-instant-search prop search-function API documentation. This is my solution:

export default {
    methods: {
        searchFunction(helper) {
            if (helper.state.query) {
                // debounce search queries, so that an Algolia search request is not triggered 
                // if another search query has overwritten the query during the same 300ms
                this.$debounce(300, 'search').then(() => {
                    helper.search()
                    // ensure that search results are always shown, even if they were hidden by previous navigation to
                    // same route that was already active
                    this.showResults = true
                })
            } else {
                // on empty search query, fire search immediately (will send local response, see computed searchClient())
                // console.log('Algolia search called with query: ' + helper.state.query)
                helper.search()
            }
        },
    },
}

UPDATE 2021-09-17: I have updated above searchFunction to use my global debounce function which is injected in Nuxt plugin plugins/helpers.js:

plugins/helpers.js

const debounceStack = []
const debounce = (delay, key = 'global') => {
    return new Promise((resolve) => {
        clearTimeout(debounceStack[key])
        debounceStack[key] = setTimeout(resolve, delay)
    })
}

export default (_context, inject) => {
    inject('debounce', debounce)
}

This is a tech blog, so I guess a regular visitor types fast enough (you're a dev that types fast as hell, right?) to barely ever wait more than 300ms between two keystrokes. The search will be delayed by 300ms, but we can live with that. If you're on a paid Algolia plan and have the money for a lot of extra requests, go ahead and reduce that delay. In my demo/nuxt-content-blog codebase, you can tune this by environment variable in .env:

ALGOLIA_QUERY_BUFFER_TIME=300

You can play around with the search field in my DemoBlog – but make sure you don't type too slow and use up all my free credits at Algolia!

Reduce Response Data

Again, Algolia by default delivers way too much data in every search response JSON. If you did not fine-tune this in Algolia's dashboard, you would get the full article bodyPlainText content duplicated for every article in your response. We don't need those fields:

{
    "results": [
        {
            "hits": [
                {
                    "title": "...",
                    "description": "...",
                    "bodyPlainText": "**REMOVE_THIS**",
                    "tags": ["...", "..."],
                    "objectID": "...",
                    "_highlightResult": {
                        "bodyPlainText": {
                            "value": "**REMOVE_THIS**"
                        }
                    }
                }
            ]
        }
    ]
}

AFAIK, there is no way to configure this client-side as request param. You need to configure this in Algolia dashboard for each (dev_articles, prod_articles) index:

• Relevance Essentials > Searchable attributes: title, description, bodyPlainText, tags (here, you must include bodyPlainText)
• Search Behaviour > Retrieved attributes: description, tags, title (remove default *)
• Pagination and Display > Highlighting > Attributes to highlight: description, title (don't include bodyPlainText)

Your JSON responses will then be below 1KB, while previously they could easily grow over 10KB for just 5 search results, depending on your article sizes.

Other Algolia Settings

I also recommend to enable the following in Algolia Dashboard:

• Search Behaviour > Advanced Syntax

That provides you: exact matching of quoted expressions and exclusion of words preceded by a “-“ sign, see advancedSyntax API parameter

That's it. Please let me know if you have further ideas for improvement.

How nice would it be, if every email client could be auto-configured and the user would just need to enter his credentials? Entering his email address plus password, hit enter, done. How come nowadays people still need to care about incoming IMAP and outgoing SMTP servers, encryption (SSL/TLS vs. STARTTLS protocols), ports (for a regular end user this is just a number without any meaning, right?), and maybe even wondering what was the username if that differs from the actual email address?

You may tell me: Hey, we're in 2021, email is so old-school! Or you may tell me: Oh, don't you know there is a thing called auto-configuration? – Unfortunately, both statements are wrong. Email today is still widely used for serious communication (and yes, you're right, it should have been replaced long time ago by something better! But it didn't.), and there is such a thing called auto-configuration, but there is just no real standardized protocol that works for every mail client out there. There was no real development in that field during the last 10+ years, sadly.

So, let me present that small Python project Martin and I have been developing for Onlime GmbH back in Feb 2020 and which I have now finally published on GitLab.com: onlime/email-autoconf

History: automx vs. email-autoconf

For years, I have been using automx (version 1) by the German Sys4 AG to serve our customers with auto-configuration for common mail clients. But that project was a pain, as it was badly maintained and was still based on ancient Python 2 at that time. It was not available as Debian package, and it required some patching to get it working as expected. Finally, back in Dec 2019, Sys4 has published a completely new version automx2 which was actually a full rewrite and had pretty much nothing in common with the previous product.

We tried to get automx2 running for our environment with the simple requirement of doing database (MySQL) based lookups of email-to-username mappings, as some legacy customers of Onlime GmbH still have email accounts where the username does not match the actual email address. That thing was so easy to configure in automx(1) but got a real pain in automx2 – It even looked like they have just forgotten about it. Overall, automx2 was so over-engineered and hard to install that it did not make our lives any easier. So, we just decided to write our own auto-configuration tool from scratch.

Please don't take me wrong. automx2 seems to have matured in the meantime, is much better documented, and may be easy to deploy. I didn't re-evaluate it since then, as we just didn't need it anymore.

And before you're getting too excited...

Deployment of email-autoconf currently is not that straight-forwarded yet, as up to now we were just using it for our mail services at Onlime and it has just been published (2021-08-26). This is fresh stuff, even though running absolutely wonderful (no single hang-up or performance issue!) since early 2020. I plan to publish an Ansible Galaxy role soon to make deployment easier.

Requirements

Requirements for email-autoconf were quite simple. It should support the following 3 protocols out of the box:

• Mozilla Autoconfig (Thunderbird)
• Microsoft Autodiscover (Outlook)
• Apple Mobileconfig (Apple Mail, iOS)

(... in the order of my preference. Why couldn't they all just adapt Mozilla's protocol which is super-duper simple?!)

It should also support the following:

• Option to run it without a database backend
• Option with a database backend in which it can lookup usernames and display names
• Let it run in the following flavors: Standalone on standard HTTPS/443 port, or with a reverse proxy (Apache/Nginx) in front, running on an internal port or socket, while the SSL-cert is served by the proxy.
• Easy and flexible to configure
• Serving high loads, providing great performance

We chose Flask as Python 3 microframework running on uWSGI as application server. You could use any other application server like Gunicorn, but uWSGI is the only one that has been tested and documented by us.

Installation

NOTE: In below result examples, we use example.org as company infrastructure domain where you run email-autoconf, and example.com as customer domain which is hosted by you.

First, install some required packages, e.g. on Debian Linux:

# Python3 packages for venv
$ apt install python3-venv python3-pip python3-dev
# uWSGI requirements
$ apt install libssl-dev libpcre3-dev

Clone the repo, set up venv, and install required Python modules:

$ git clone https://gitlab.com/onlime/email-autoconf.git
$ cd email-autoconf
$ python3 -m venv venv
$ . venv/bin/activate
$ pip install -r requirements.txt

Copy the example config from instance/settings.py_example to instance/settings.py and edit it to match your needs:

DB_TYPE = 'mysql'
DB_HOST = '127.0.0.1'
DB_PORT = '3306'
DB_NAME = 'email-autoconf'
DB_USER = 'email-autoconf'
DB_PASSWORD = '************'

SELECT_USERNAME = "SELECT `dest` FROM `mailmap` WHERE `address` = '%a'"
SELECT_DISPLAY_NAME = "SELECT `display_name` FROM `mailaccounts` WHERE `username` = '%u'"

IMAP_SERVER_HOSTNAME = 'imap.example.org'
POP3_SERVER_HOSTNAME = 'pop.example.org'
SMTP_SERVER_HOSTNAME = 'smtp.example.org'

EMAIL_PROVIDER = 'example.org'

For all possible options, consult Configuration section below.

Set DB_TYPE = 'none' if you don't need any specific email-to-username lookups. This works in an environment where every mail account uses its email address as username.

Now choose one of the 3 available uWSGI instance configurations:

• uwsgi-http-instance.yml_example: Testing instance / HTTP port 8080
• uwsgi-https-instance.yml_example: HTTPS port 8443
• uwsgi-socket-instance.yml_example: Socket port 3032

Use those configurations as examples and deploy the one that fits your need. We recommend running it as a socket on port 3032 with uwsgi-socket-instance.yml instance config and set up an Apache/Nginx reverse proxy (see "Reverse Proxy" section below) in front of it for proper HTTPS support and optimal performance.

For testing, copy uwsgi-http-instance.yml_example to uwsgi-http-instance.yml and run the instance using uWSGI as application server:

$ uwsgi --yaml uwsgi-http.yml --yaml uwsgi-http-instance.yml

For basic testing, use curl e.g. to retrieve the autoconfig Mozilla scheme:

curl localhost:8080/mail/config-v1.1.xml?emailaddress=demo@example.com

For in-depth testing, use the provided scripts/email_autoconf_test.py, see Testing section below.

Configuration

It is possible to configure email-autoconf within the file instance/settings.py or with environment variables. See instance/settings.py_example for an example configuration.

Environment variables could be handed over if you add them in front of the uwsgi command, like this:

$ SMTP_SERVER=smtp.example.org uwsgi --yaml uwsgi-http.yml --yaml uwsgi-http-instance.yml

Settings for the uWSGI instance could be done in a .yml file which you then use in the start command. We recommend consulting the documentation on Configuring uWSGI for more details about the uWSGI configuration.

For a full list of config parameters, consult: documentation.md

Systemd Service

We recommend running email-autoconf as a Systemd service. The following setup has been proved working on a Debian Linux (tested on Debian 10 Buster).

IMPORTANT: Don’t run as root!

If you use a port number greater than 1024 (I suggest port 3032), the application does not require super user privileges when running. It also does not need to be installed as root. It is recommended that you create a user account specifically for email-autoconf, but other unprivileged users will do as well.

Create /etc/systemd/system/email-autoconf.service:

[Unit]
Description=uWSGI instance to serve email-autoconf
After=network.target

[Service]
Type=simple
User=email-autoconf
Group=email-autoconf
WorkingDirectory=/var/www/email-autoconf
ExecStart=/var/www/email-autoconf/venv/bin/uwsgi --yaml uwsgi-http.yml --yaml uwsgi-socket-instance.yml
Restart=on-failure

[Install]
WantedBy=multi-user.target

After creating this new Systemd service unit, run:

$ systemctl daemon-reload

Then, enable/start the service:

$ systemctl enable email-autoconf.service
$ systemctl start email-autoconf.service
$ systemctl status email-autoconf.service

email-autoconf should now in socket mode on port 3032 (as we have provideduwsgi-socket-instance.yml instance config in Systemd ExecStart command).

DNS Configuration

On your company's infrastructure DNS zone (the domain you usually put into EMAIL_PROVIDER in your instance config), you should configure the following DNS resource records:

email-autoconf  IN  A   X.X.X.X. ; IP of the server running email-autoconf

maildiscovery   IN  CNAME   email-autoconf.example.org.
autoconfig  IN  CNAME   maildiscovery.example.org.
autodiscover    IN  CNAME   maildiscovery.example.org.

On any customer domain, the following records should be configured:

autoconfig  300 IN  CNAME   maildiscovery.example.org.
_autodiscover._tcp  300 IN  SRV 0 0 443 maildiscovery.example.org.

Reverse Proxy

To simplify SSL handling and protect the uWSGI server, we strongly recommend using some sort of reverse proxy in front of email-autoconf. Below you can find config examples for Apache and Nginx.

NOTE: Below examples are based on the recommended uwsgi-socket-instance.yml uWSGI instance configuration for email-autoconf, running it as a socket on port 3032.

Apache

In Apache, the modules proxy and proxy_uwsgi has to be enabled, while proxy should already be enabled by default.

$ a2enmod proxy_uwsgi
$ systemctl restart apache2

Apache vitual hosts to proxy requests to the email-autoconf uWSGI server could look like this:

<VirtualHost *:80>
    ServerName autoconfig.example.org
    ServerAlias autoconfig.*
    ServerAlias maildiscovery.example.org maildiscovery.*
    ServerAlias autodiscover.example.org autodiscover.*
    ErrorLog ${APACHE_LOG_DIR}/email_autoconf_error.log
    CustomLog ${APACHE_LOG_DIR}/email_autoconf_access.log combined
    ServerAdmin hostmaster@example.org
    
    ProxyPass / uwsgi://127.0.0.1:3032/
</VirtualHost>

<VirtualHost *:443>
    ServerName autoconfig.example.org
    ServerAlias autoconfig.*
    ServerAlias maildiscovery.example.org maildiscovery.*
    ServerAlias autodiscover.example.org autodiscover.*
    SSLEngine on
    SSLCertificateKeyFile /etc/letsencrypt/live/maildiscovery.example.org/privkey.pem
    SSLCertificateFile /etc/letsencrypt/live/maildiscovery.example.org/fullchain.pem
    ErrorLog ${APACHE_LOG_DIR}/email_autoconf_error.log
    CustomLog ${APACHE_LOG_DIR}/email_autoconf_access.log combined
    ServerAdmin hostmaster@example.com
    
    ProxyPass / uwsgi://127.0.0.1:3032/
</VirtualHost>

Nginx

For Nginx a config like this should work:

server {
    listen 80;
    server_name autoconfig.example.org autoconfig.* maildiscovery.* autodiscovery.* autodiscover.*;
    location / {
        proxy_pass http://127.0.0.1:3032/;
    }
}

server {
    listen 443 ssl;
    server_name autoconfig.example.org autoconfig.* maildiscovery.* autodiscovery.* autodiscover.*;
    ssl_certificate_key /etc/letsencrypt/live/maildiscovery.example.org/privkey.pem;
    ssl_certificate /etc/letsencrypt/live/maildiscovery.example.org/fullchain.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    location / {
        proxy_pass http://127.0.0.1:3032/;
    }
}

Check if email-autoconf is running (presents a nice splash screen with Onlime logo): https://maildiscovery.example.org/

Testing

email_autoconf_test.py

Use scripts/email_autoconf_test.py helper script to test any self-hosted or external email address. Make sure you execute that script under your venv which is correctly set up as described in above installation section:

$ cd /var/www/email-autoconf
$ . venv/bin/activate
(venv)$ cd scripts
(venv)$ python3 email_autoconf_test.py demo@example.com

Sample result (full XML output for all 3 protocols Mozilla Autoconfig, Microsoft Autodiscover, Apple Mobileconfig):

Testing Autoconfig
Try https://autoconfig.example.org/mail/config-v1.1.xml?emailaddress=demo@example.com
Autoconfig content:

<?xml version="1.0" encoding="UTF-8"?>
<clientConfig version="1.1">
    <emailProvider id="example.org">
      <domain>example.com</domain>
      <displayName>demo@example.com</displayName>
      <incomingServer type="imap">
         <hostname>imap.example.org</hostname>
         <port>993</port>
         <socketType>SSL</socketType>
         <authentication>password-cleartext</authentication>
         <username>demo_example_com</username>
      </incomingServer>
      <incomingServer type="pop3">
         <hostname>pop.example.org</hostname>
         <port>995</port>
         <socketType>SSL</socketType>
         <authentication>password-cleartext</authentication>
         <username>demo_example_com</username>
      </incomingServer>
      <outgoingServer type="smtp">
         <hostname>smtp.example.org</hostname>
         <port>465</port>
         <socketType>SSL</socketType>
         <username>demo_example_com</username>
         <authentication>password-cleartext</authentication>
         <useGlobalPreferredServer>no</useGlobalPreferredServer>
      </outgoingServer>
    </emailProvider>
</clientConfig>

---
Testing Autodiscover
Try https://maildiscovery.example.org/autodiscover/autodiscover.xml
Autodiscover content:

<?xml version="1.0" encoding="UTF-8"?>
<Autodiscover xmlns="http://schemas.microsoft.com/exchange/autodiscover/responseschema/2006">
  <Response xmlns="http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a">
    <User>
      <DisplayName>demo@example.com</DisplayName>
    </User>
    <Account>
      <AccountType>email</AccountType>
      <Action>settings</Action>
      <Protocol>
        <Type>IMAP</Type>
        <Server>imap.example.org</Server>
        <Port>993</Port>
        <SPA>off</SPA>
        <Encryption>Auto</Encryption>
        <SSL>on</SSL>
        <AuthRequired>on</AuthRequired>
        <DomainRequired>off</DomainRequired>
        <LoginName>demo_example_com</LoginName>
      </Protocol>
      <Protocol>
        <Type>POP3</Type>
        <Server>pop.example.org</Server>
        <Port>995</Port>
        <SPA>off</SPA>
        <Encryption>Auto</Encryption>
        <SSL>on</SSL>
        <AuthRequired>on</AuthRequired>
        <DomainRequired>off</DomainRequired>
        <LoginName>demo_example_com</LoginName>
      </Protocol>
      <Protocol>
        <Type>SMTP</Type>
        <Server>smtp.example.org</Server>
        <Port>587</Port>
        <SPA>off</SPA>
        <Encryption>Auto</Encryption>
        <SSL>on</SSL>
        <AuthRequired>on</AuthRequired>
        <UsePOPAuth>on</UsePOPAuth>
        <SMTPLast>off</SMTPLast>
        <DomainRequired>off</DomainRequired>
        <LoginName>demo_example_com</LoginName>
      </Protocol>
    </Account>
  </Response>
</Autodiscover>

---
Testing Mobileconfig
Try https://autoconfig.example.org/email.mobileconfig?email=demo@example.com
(...)

Cool! Now, head over to your email client and do some live testing. If all works well, you should just be required to enter the email address and password, the rest should get auto detected. It works best on Thunderbird. Outlook may struggle with it, depending on your Outlook version.

Continuous deployment is great and a must for every modern web app. Forget about the times when you had to constantly log into your production server over SSH to run some git pull based deployment and cumbersome and error-prone build tasks, or building your project locally and then deploying it with rsync to production, which is not that sexy either. It is all doable and scriptable, but we want to have the whole process automated without any manual work involved. I much prefer to use GitLab CI/CD over GitHub Actions - ok, mainly just because I am more into it and prefer to run a self-hosted GitLab instance. GitLab just gets the job done very well!

We want the whole process to be straightforward without any fancy extras. It should be a matter of 15mins to set it up on every new project and I don't like to introduce any extra dependencies. I am just talking about deployment of a static site / SPA, a Vue.js based frontend that is generated by Nuxt.js. So let's keep it simple here! I am going to present you the solution I am using to deploy this TechBlog.

Requirements

OK, don't do it in the most simple way, we still have some basic requirements aside an apple a day (which keeps the doctor away):

• Building should be done by GitLab CI/CD in Docker executor
• Generated static site is uploaded to production server via SSH/rsync
• We want basic versioning with current symlink, keeping the last 5 releases. No fat dependencies like Capistrano, please!
• Only deploy when pushed to the default (main) branch (or any other branch that fits your workflow, but don't deploy on every branch!)

Manual Deployment

Remember: Building the application, generate every route as a HTML file and statically export to dist/ directory on a Nuxt.js project goes like this (I am preferring yarn over npm, but basically the same):

$ yarn install
$ yarn generate

yarn generate maps to nuxt generate, as we have defined the following in package.json:

{
     "scripts": {
       "dev": "nuxt",
       "build": "nuxt build",
       "start": "nuxt start",
       "generate": "nuxt generate"
     },
     // ...
}

If a local build does not yet work for you (maybe producing some HTML/CSS minification errors...), fix it first locally and study Nuxt.js Commands and Deployment. In this example and on this TechBlog deployment, we are using target: 'static' in nuxt.config.js, that's why we build the site with nuxt generate instead of nuxt build.

On a manual deployment, you would then rsync the whole dist/ content to your production server, e.g.:

$ rsync -aHv --delete dist/ user@example.com:public_html/blog

On production server, you would then point the site root to public_html/blog, but with the automated CD solution below, you would point it to public_html/blog/current, as we're going to create a symlink to the current release.

GitLab CI/CD Setup

• GitLab CI/CD: SSH keys when using the Docker executor
• GitLab: Generate an SSH key pair
• GitLab: Verifying the SSH host keys

Set up CI/CD Variables

Building and deployment is done automatically via GitLab CI/CD, once you have .gitlab-ci.yml (see below) added to your project root. A deployment is always triggered when you push to main branch. But first, set up those CI/CD variables in your GitLab project under Settings > CI/CD > Variables:

• DEPLOY_HOST (Variable): The production host we want to deploy to.
• SSH_KNOWN_HOSTS (Variable): ~/.ssh/known_hosts lines on deployment container (Docker)
• SSH_USER (Variable): The user to connect to the production host.
• SSH_KEY (File): The deployment SSH private key (copy-paste the whole key starting with -----BEGIN OPENSSH PRIVATE KEY-----\n...).

To get the correct line(s) for SSH_KNOWN_HOSTS variable, run this from a trusted host:

$ ssh-keyscan DEPLOY_HOST

Initially, you create a SSH keypair as follows (ED25519 is the first-preference signature algorithm since OpenSSH 8.5 and has been supported since OpenSSH 6.5):

$ ssh-keygen -t ed25519 -f ~/.ssh/deploy_id_ed25519 -N "" -C "gitlab-ci-deployer"
$ cat ~/.ssh/deploy_id_ed25519.pub

Don't store the private key anywhere else than in $SSH_KEY! Copy the public key (deploy_id_ed25519.pub) to the deploy host's authorized_keys:

$ echo "ssh-ed25519 AAAAC3N... gitlab-ci-deployer" >> ~/.ssh/authorized_keys

Create .gitlab-ci.yml

Ready for the real show? We're using official node Docker Image. node:current is based on Debian Buster (as of Aug 2021), so we can also use apt-get inside the container to install our extras, which is actually just rsync, nothing more.

Create the following .gitlab-ci.yml in your project root. It will use the private SSH key which you have provided in SSH_KEY GitLab CI/CD Variables as a File (NOT as a Variable!), so you don't need to know about the actual location of the key, just use ssh -i ${SSH_KEY} in the deploy commands.

image: node:current

variables:
  # In case you did not set up $SSH_KNOWN_HOSTS variable:
  # SSH_OPTS: '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
  SSH_OPTS: ''
  BASE_DIR: public_html/blog
  KEEP: 5

default:
  before_script:
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts
    - chmod 600 ${SSH_KEY}
    - export COMMIT_TIME=$(git show -s --format=%ct $CI_COMMIT_SHA)

cache:
  paths:
    - node_modules

build:
  stage: build
  script:
    - cp .env.prod .env
    - yarn install
    - yarn generate
  artifacts:
    paths:
      - dist
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $RUN_ALWAYS

deploy:
  stage: deploy
  script:
    - apt-get update -y && apt-get install -y rsync
    - ssh -i ${SSH_KEY} ${SSH_OPTS} ${SSH_USER}@${DEPLOY_HOST} "cd ${BASE_DIR} && test ! -h current || cp -r \$(ls -td */ | grep -v current | head -n1) deploy_tmp"
    - rsync -aHv --delete -e "ssh -i ${SSH_KEY} ${SSH_OPTS}" dist/ ${SSH_USER}@${DEPLOY_HOST}:${BASE_DIR}/deploy_tmp
    - ssh -i ${SSH_KEY} ${SSH_OPTS} ${SSH_USER}@${DEPLOY_HOST} "cd ${BASE_DIR} && mv deploy_tmp ${COMMIT_TIME} && rm -f current && ln -s ${COMMIT_TIME} current"
    - ssh -i ${SSH_KEY} ${SSH_OPTS} ${SSH_USER}@${DEPLOY_HOST} "cd ${BASE_DIR} && rm -rf \$(ls -dt */ | grep -v current | tail -n +$(($KEEP+1)))"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $RUN_ALWAYS

UPDATE 2021-08-18: Above .gitlab-ci.yml can be further optimized, so that we no longer need to provide $RUN_ALWAYS variable to force-trigger a pipeline run on a different branch than default (main / master). I have also combined build and deploy stages into a single one, for improved deployment speed, as we don't need to split this up for this simple project. Like this, dist artifacts are no longer needed:

image: node:current

variables:
  # In case you did not set up $SSH_KNOWN_HOSTS variable:
  # SSH_OPTS: '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
  SSH_OPTS: ''
  BASE_DIR: public_html/blog
  KEEP: 5

default:
  before_script:
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts
    - chmod 600 ${SSH_KEY}
    - export COMMIT_TIME=$(git show -s --format=%ct $CI_COMMIT_SHA)

cache:
  paths:
    - node_modules

# combined build and deploy step to speed it up
build_deploy:
  stage: deploy
  script:
    # build
    - cp .env.prod .env
    - yarn install
    - yarn generate
    # deploy
    - apt-get update -y && apt-get install -y rsync
    - ssh -i ${SSH_KEY} ${SSH_OPTS} ${SSH_USER}@${DEPLOY_HOST} "cd ${BASE_DIR} && test ! -h current || cp -r \$(ls -td */ | grep -v current | head -n1) deploy_tmp"
    - rsync -aHv --delete -e "ssh -i ${SSH_KEY} ${SSH_OPTS}" dist/ ${SSH_USER}@${DEPLOY_HOST}:${BASE_DIR}/deploy_tmp
    - ssh -i ${SSH_KEY} ${SSH_OPTS} ${SSH_USER}@${DEPLOY_HOST} "cd ${BASE_DIR} && mv deploy_tmp ${COMMIT_TIME} && rm -f current && ln -s ${COMMIT_TIME} current"
    - ssh -i ${SSH_KEY} ${SSH_OPTS} ${SSH_USER}@${DEPLOY_HOST} "cd ${BASE_DIR} && rm -rf \$(ls -dt */ | grep -v current | tail -n +$(($KEEP+1)))"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_PIPELINE_SOURCE == "web"

From now on, once you push to the default (usually main or master) branch, continuous deployment is triggered.

If you explicitely want to skip CI/CD for a single push, use:

$ git push -o ci.skip

GitLab will respect it!

If you wish to run a manual pipeline on a different branch, just do this from Gitlab's Run pipeline form. The $CI_PIPELINE_SOURCE == "web" rule comes into play!

The deployment task cares about basic versioning, so the final result will be such a structure (the Unix timestamp was taken from the latest git commit), while the oldest releases are getting rotated away (see KEEP variable in .gitlab-ci.yml):

public_html/blog
├── 1629150195
├── 1629154265
├── 1629154699
├── 1629188761
├── 1629188770
└── current -> 1629188770

Honestly, the deploy commands don't look that sexy in our .gitlab-ci.yml. But we are perfectly happy with this, especially since we didn't introduce any new monster dependency like Capistrano or similar deployment tools.

Finally, point your production web server's virtual host to public_html/blog/current. No need to restart anything or flush any caches, as this is a static site / SPA only.

Done! Now, go back coding and no longer care about deployment.

For years now, I am using this little helper script written in Python, which solves a very basic task: Destroy legacy ZFS snapshots. We all know ZFS snapshots are so lightweight and we tend to create so many snapshots, either manually (e.g. before a major system upgrade) or automated (e.g. by our replication or backup jobs). It can somtimes be cumbersome to destroy legacy ZFS snapshots that are no longer needed.

So, here comes help with zfs-destroy-snapshots.py which you could deploy to /usr/local/sbin to have that handy script ready. I know, this is like a really small snipped I could just post to Github Gist (ok, here you go: onlime/zfs-destroy-snapshots.py) or Pastebin, but I think it still deserves a full blown blog post.

Below script supports the following options:

$ zfs-destroy-snapshots.py -h
usage: zfs-destroy-snapshots.py [-h] [--prefix PREFIX] [--exclude EXCLUDE]
                                [--dataset DATASET] [--destroy]
                                age

positional arguments:
  age                List snapshots that are older than age (hours).

optional arguments:
  -h, --help         show this help message and exit
  --prefix PREFIX    Only list snapshot with this prefix.
  --exclude EXCLUDE  Exclude snapshots that contain this string.
  --dataset DATASET  Limit to a specific dataset.
  --destroy          Destroy snapshots that are older than age.

So, basic use would be (run this as root):

# list all snapshots that are older than 48hrs
$ zfs-destroy-snapshots.py 48
# destroy them for good
$ zfs-destroy-snapshots.py 48 --destroy

By default, the script does a dry-run, only listing snapshots it is going to destroy. It will only actually destroy them, if you explicitely use the --destroy argument. For your safety, and your daughter's.

You could further limit the lookup of snapshots by the following:

• --prefix: snapshot prefix like e.g. zrep_ for zrep or rep_ for pve-zsync (Proxmox VE replication manager / backup tool)
• --exclude: exclude snapshots that contain this string
• --dataset: limit by a single dataset, e.g. rpool/zfsdisks/subvol-123-disk-1

Remember to always run this script without --destroy first to verify the snapshots which are going to be deleted.

Here you go, zfs-destroy-snapshots.py / Gist onlime/zfs-destroy-snapshots.py

#!/usr/bin/env python3
"""
Helper script to remove old ZFS snapshots
Copyright (c) Onlime GmbH, https://www.onlime.ch
"""
import argparse
from re import compile
from datetime import datetime, timedelta
from subprocess import check_output
import sys

def list_snapshots(age: int, prefix: str = None, exclude: str = None, dataset: str = None, destroy: bool = False):
    # List all snapshot names of a dataset sorted by creation date; sample command:
    # $ zfs list -H -t snapshot -o name,creation -s creation [<dataset>]
    cmd = ['zfs', 'list', '-H', '-t', 'snapshot', '-o', 'name,creation', '-s', 'creation']
    if dataset:
        cmd.append(dataset)
    snap_list = check_output(cmd).decode(sys.stdout.encoding)
    snap_lines = snap_list.splitlines()
    # reverse sort order (`zfs list` only supports asc sort order), newest snapshots on top
    snap_lines.reverse()

    for line in snap_lines:
        snapshot, creation = line.split('\t')
        if prefix and '@{}'.format(prefix) not in snapshot:
            continue
        if exclude and exclude in snapshot:
            continue
        snap_date = datetime.strptime(creation, '%a %b %d %H:%M %Y')
        if snap_date < datetime.now()-timedelta(hours=age):
            print('{}{} {}'.format(
                'DESTROYING ' if destroy else '',
                snap_date.strftime('%Y-%m-%d %H:%M'), 
                snapshot
            ))
            if destroy:
                check_output(['zfs', 'destroy', snapshot])

if __name__ == "__main__": # Only use the argument parsing if the script was called as script and not imported
    # argument parsing
    parser = argparse.ArgumentParser()
    parser.add_argument("age", type=int, help="List snapshots that are older than age (hours).")
    parser.add_argument("--prefix", help="Only list snapshot with this prefix.")
    parser.add_argument("--exclude", help="Exclude snapshots that contain this string.")
    parser.add_argument("--dataset", help="Limit to a specific dataset.")
    parser.add_argument("--destroy", action='store_true', help="Destroy snapshots that are older than age.")
    args = parser.parse_args()

    list_snapshots(args.age, args.prefix, args.exclude, args.dataset, args.destroy)

some final thoughts...

Besides, Onlime GmbH webhosting services run 100% on Debian Linux, mostly in LXC containers on Proxmox VE (PVE). If you don't know PVE yet - I can just strongly recommend it. They offer stunning support for OpenZFS - formerly known as ZFS-on-Linux, which lately runs super robust (which was definitely not the case 4-5 yrs ago!). So really, if you're not into ZFS yet, look into it. It is so much more than just another file system. Enough promo for today. Cheers!

I'd like to share my recommended ESLint and Prettier configuration for a Vue.js project built with Nuxt.js (loving it!). As preferred IDE, I am using - guess what? - VS Code. This article is just considered as simple dev notes without digging any deeper, and it's based on my current "best practices".

Install ESLint and Prettier with all recommended plugins into devDependencies using yarn (you might as well use npm):

$ yarn add --dev eslint eslint-plugin-nuxt eslint-plugin-vue
$ yarn add --dev eslint-config-prettier prettier eslint-plugin-prettier
$ yarn add --dev @nuxtjs/eslint-module @nuxtjs/eslint-config @babel/eslint-parser

The following lines were added to my package.json (only an extract, showing you the current versions as of Aug 2021):

{
  "devDependencies": {
    "@babel/eslint-parser": "^7.15.0",
    "@nuxtjs/eslint-config": "^6.0.1",
    "@nuxtjs/eslint-module": "^3.0.2",
    "eslint": "^7.32.0",
    "eslint-config-prettier": "^8.3.0",
    "eslint-plugin-nuxt": "^2.0.0",
    "eslint-plugin-prettier": "^3.4.0",
    "eslint-plugin-vue": "^7.15.1",
    "prettier": "^2.3.2"
  }
}

Configure ESLint in .eslintrc.js (in your project root):

module.exports = {
    root: true,
    env: {
        browser: true,
        node: true,
    },
    parserOptions: {
        parser: '@babel/eslint-parser',
        requireConfigFile: false,
    },
    extends: [
        '@nuxtjs',
        'plugin:nuxt/recommended',
        'prettier'
    ],
    plugins: ['prettier'],
    rules: {
        'prettier/prettier': ['error'],
        'vue/html-indent': ['error', 4],
        'vue/singleline-html-element-content-newline': 0,
        'vue/component-name-in-template-casing': ['error', 'PascalCase'],
        'vue/valid-v-slot': [
            'error',
            {
                allowModifiers: true,
            },
        ],
    },
    globals: {
        _: true,
    },
}

Configure Prettier in .prettierrc (again in your project root):

{
    "semi": false,
    "singleQuote": true,
    "tabWidth": 4,
    "printWidth": 120
}

Also create .prettierignore:

# Ignore artifacts:
build
coverage

Configure VS Code Editor: Format on Save to apply auto formatting upon saving, which you can also restrict to specific languages, .vscode/settings.json:

{
    "vetur.format.defaultFormatter.html": "none",
    // Set the default
    "editor.formatOnSave": false,
    // Enable per-language
    "[javascript]": {
        "editor.formatOnSave": true
    },
    "[vue]": {
        "editor.formatOnSave": true
    }
}

Now, open any .vue or .js file, fire Cmd-S to save the file and see the magic. If it won't yet work for you, consider restarting VSCode.

Don't worry, a file only gets formatted if you explicitely hit Cmd-S. You can still use autoSave for regular saving, which would not invoke auto-formatting of the whole file. That's the relevant global VS Code user settings (can be found in $HOME/Library/Application Support/Code/User/settings.json on macOS):

{
    "files.autoSave": "afterDelay",
    "prettier.printWidth": 81,
}

Done. Happy clean coding!

SSH authorized_keys allows you to define a command which is executed upon authentication with a specific key by prefixing it with the command="cmd" option.

authorized_keys man page explains this wonderful feature as follows:

command="command"

Specifies that the command is executed whenever this key is used for authentication. The command supplied by the user (if any) is ignored. (...) This option might be useful to restrict certain public keys to perform just a specific operation. An example might be a key that permits remote backups but nothing else. (...) The command originally supplied by the client is available in the SSH_ORIGINAL_COMMAND environment variable.

The syntax goes like this, put command="cmd" and all other options before a pubkey in authorized_keys:

command="cmd",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-rsa AAAAB3N...

A detailed description of these options can be found here:

• Configuring authorized_keys for OpenSSH

You might have missed the two use cases from my other blog posts, as they are quite long and complex:

• 2018-03-06 Simple and Secure External Backup: Restricting SSH access to rsync only from a specific directory
• 2021-08-03 Secure External Backup with ZFS Native Encryption: Restricting SSH access to a list of specific command patterns

So here's the two use cases in short:

Use Case 1: Limit rsync to a directory (rrsync)

Let's suppose we need full root permissions to pull data from a backup server, but its access should be limited to rsync from a specific directory and not executing any other commands on the remote backup server. The host that pulls data is going to be called extbackup.

There is a simple rsync wrapper for this which comes shipped with Debian's rsync package but first needs to be unpacked/installed:

backup$ zcat /usr/share/doc/rsync/scripts/rrsync.gz > /usr/local/bin/rrsync
backup$ chmod 0755 /usr/local/bin/rrsync
backup$ ln -s /usr/local/bin/rrsync /usr/bin/rrsync

We then add this to /etc/sudoers.d/extbackup on backup:

%backuppers ALL= NOPASSWD:SETENV: /usr/bin/rrsync

Let's now create a specific user extbackup and add him to group backuppers:

backup$ adduser extbackup
backup$ adduser extbackup backuppers

Add the public key of extbackup (found in extbackup:~/.ssh/id_rsa.pub) to backup:/home/extbackup/.ssh/authorized_keys, prefixed by the following options:

command="sudo -E /usr/bin/rrsync -ro /backups/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-rsa AAAABxyz...

This limits extbackup to only pull data from backup:/backups/ directory over rsync. Test-run:

extbackup$ rsync -aH --numeric-ids --delete --delete-before extbackup@backup: /mnt/backups

Explanation: extbackup's root user rsyncs data from backup:/backups/ to /mnt/backups but only needs to provide the relative path to backup as rsyncing is enforced over rrsync to /backups/ directory on the remote side. If we want to pull all data from backup:/backups/, simply provide an empty relative path after the colon.

Isn't that cool? We can pull data with root rights from our backup server but are limited to one specific directory and are not able to execute any other commands except rsync!

Use Case 2: Restrict to specific command patterns

As pointed out here, OpenSSH's authorized_keys command option enforces a single command (by overriding the original command). So the question was:

The authorized_keys has a command="..." option that restricts a key to a single command. Is there a way to restrict a key to multiple commands? E.g. by having a regex there, or by editing some other configuration file?

This can be simply accomplished by a wrapper script, as OpenSSH makes the original command supplied by the client available in the SSH_ORIGINAL_COMMAND environment variable.

In Secure External Backup with ZFS Native Encryption our requirement was to limit root access over SSH to only specific command patterns, so that the client (in this case extbackup which pulls data from backup server) was only able to execute commands which were needed for PVE-zsync backup/replication.

To accomplish this specific restrictions, we need to use a script in command section before the extbackup's pubkey in .ssh/authorized_keys:

command="/root/.ssh/allowed-commands.sh",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-rsa AAAAB3N... root@extbackup

allowed-commands.sh then does pattern matching to verify the commands - using $SSH_ORIGINAL_COMMAND to get the original command:

For numeric ranges in Bash regexes I have used Regex Numeric Range Generator.

#!/bin/bash

## CONFIGURATION ##############################################
veid_pattern='(19[4-9]|2[0-4][0-9]|25[0-4])'
backup_srcdir='/backup'
pool_name=dpool
###############################################################

# Regex patterns
# WARNING: '\w' is not supported in Bash, so we need to use '[:alnum:]' instead
ds_pattern="${pool_name}/zfsdisks/subvol-${veid_pattern}-disk-1"
ts_pattern='20[2-9][0-9]-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])_(0[0-9]|1[0-9]|2[0-3])(:(0[0-9]|[1-5][0-9])){2}'
snap_prefix_pattern='rep_extbackup_([1-9]a_)?'
snap_pattern="@${snap_prefix_pattern}${ts_pattern}"
ds_snap_pattern="${ds_pattern}${snap_pattern}"

patt1='zfs list -r( -)?t snapshot -Ho name( -S creation)? '"$ds_pattern($snap_pattern)?"
patt2='zfs (snapshot|destroy) '"$ds_snap_pattern"
patt3='zfs send '"(-i $ds_snap_pattern )?-- $ds_snap_pattern"
patt4="zfs rename $ds_snap_pattern $ds_snap_pattern" # only used in extbackup-migrate-zfs-snaps.sh

cmd="$SSH_ORIGINAL_COMMAND"

if [[ "$cmd" == 'list-datasets' ]]; then
    # special command to get a server list of all datasets and mountpoints
    zfs list -H -o name,mountpoint,usedds | grep $backup_srcdir
    exit
elif [[ $cmd =~ ^$patt1$ || $cmd =~ ^$patt2$ || $cmd =~ ^$patt3$ || $cmd =~ ^$patt4$ ]]; then
    $SSH_ORIGINAL_COMMAND
    exit
else
    logger "$(basename $0) violation: $cmd"
    echo "Access denied"
    exit 1
fi

We can then test the magic list-datasets command to get a server list of all datasets and mountpoints:

extbackup$ ssh backup list-datasets

And pve-zsync is able to execute remote commands like such:

$ zfs list -r -t snapshot -Ho name -S creation dpool/zfsdisks/subvol-181-disk-1
$ zfs snapshot dpool/zfsdisks/subvol-181-disk-1@rep_extbackup_2021-07-12_23:15:53
$ zfs send -- dpool/zfsdisks/subvol-181-disk-1@rep_extbackup_2021-07-12_23:15:53
$ zfs destroy dpool/zfsdisks/subvol-181-disk-1@rep_extbackup_2021-07-12_23:13:13

Whenever we try to fire another command, we get an Access denied and the command is logged to syslog.

Sidenote: I trust extbackup as a client, especially since its SSH private key is encrypted and is only getting decrypted to volatile memory. But I still wanted to restrict it to a small subset of commands, and also to a subset of ZFS datasets it can pull data from, for improved security.

OpenSSH's authorized_keys command option gives us full flexibility with such a wrapper script!

Let's improve our Simple and Secure External Backup solution I have published back in 2018. Back then, I was using rsync over SSH to pull backup data, and LUKS encryption as full disk encryption for the external drives. As we all know, transferring data with rsync can get horribly slow and blow up your I/O if you're transferring millions of small files. Also, LUKS encryption may be a bit low level and inflexible. What we want to accomplish: A performant and secure backup solution based on ZFS, using zfs send|recv for efficient data transfer, and ZFS native encryption to secure our external drives. So let's go ahead and built that thing from scratch on a fresh 2021 stack!

Let's assume you already have an existing backup server that is connected to the internet 24/7 and runs daily/weekly/monthly backup jobs. Backup data is stored in ZFS datasets, ideally as individual datasets per full-system backup for each host. We would now like to set up a second offsite backup server that just cares about storing data to an encrypted external drive and after each backup run, you are going to physically detach that drive.

So, we are talking about offline backups in addition to the fact having this server offsite - at a different location than your main backup server.

Preferably, your main backup server would also be offsite. But as it needs to pull data frequently, its storage is always available and not getting detached.

Let's call your main backup server backup and the one we are going to set up here extbackup.

Requirements

We have the following – from a security point of view – rather strict requirements:

• Hardware: Payable but reliable fanless system that could be mounted to any office desk.
• System: Proxmox VE for well-maintained OpenZFS and PVE-zsync support
• No backup data should ever be stored (not even temporarily!) on rpool (system partition), only on encrypted external drives.
• System is installed on internal SSD, external drives attached over USB 3 or eSATA
• External drives fully encrypted using ZFS native encryption (preferred over LUKS for better handling and easier setup)
• ZFS encryption key not stored anywhere on disk, only generated at runtime.
• Snapshot-based backup for optimal performance: Backup data is pulled from backup over SSH, using zfs send|recv. We choose PVE-zsync as simple backup tool / replication manager
• SSH private key to connect to backup is not stored anywhere on extbackup in plaintext, only getting generated at runtime.
• SSH access on remote backup server is strictly limited to pve-zsync operations, without providing a full shell.
• Both ZFS encryption key and SSH private key should not be required to be stored in any password manager or anywhere else.
• Unlock password for those keys should only be used as a 2nd factor / seed, so encryption security does not depend on its length.
• External disks should only get decrypted during a backup run and getting encrypted/unmounted right afterwards.
• Backup data is directly streamed to external disks without temporary storage on extbackup server.
• Ideally, the external drive should be a plain SATA drive without external casing, which could be plugged directly into a server via onboard SATA or eSATA (in case of an emergency when we lack time to transfer the data for recovery). Performance should also be decent in case we ever run from such a drive in production, so SSD is preferred!

Phew! This sounds like a lot of tight requirements! But it is all doable in a very simple slick and sexy setup. Keep on reading!

Hardware Setup

I do recommend the following hardware for a small robust fanless system that is built for 24/7 operation:

This hardware currently (Aug 2021) is available for ~ $900 in total. You can go much cheaper with some Intel NUC system that comes delivered with a basic RAM / SSD setup. But we do recommend that Shuttle XPC line as it is an industry-grade platform and we never had any issues with its predecessors DS57U / DS77U. It's also 100% quiet, producing no noise at all.

And maybe you want to save some bucks and don't get the powerful DS10U5 with an i5, but rather a DS10U3 with an i3 or even the base model with a Celeron 4205U.

As external 8TB SSD (we're up for speed!), I do recommend the following:

• Samsung 870 QVO 8TB (model MZ-77Q8T0BW), or go for 4/2/1 TB variant, if that is enough for you
• StarTech USB 3.1 to SATA adapter (model USB312SAT3CB)

Do NOT use any USB3-to-SATA adapter that emulates its own device, e.g. this shitty piece of crap: Manhattan USB 3.0 to SATA (model 130424). The adapter should support direct access to your drive. The StarTech adapter supports UASP (USB Attached SCSI) which is what we want!

This whole hardware setup might sound luxury in your eyes. It all depends of how important the (customer) data is and how far you want to optimize disaster recovery time. You can go much cheaper if that all doesn't matter to you.

These are my recommendations and they have been very well tested at Onlime GmbH webhosting.

System Setup

Install Proxmox VE 7 (Debian Bullseye based) from an USB stick which you create as follows:

$ wget https://www.proxmox.com/images/download/pve/iso/proxmox-ve_7.0-1.iso
$ cat proxmox-ve_7.0-1.iso > /dev/sdx

On macOS, use diskutil list to identify the correct device name of your USB stick and run diskutil unmountDisk /dev/disk2 prior to writing to it. Something like:

$ diskutil list
$ sudo su -
$ diskutil unmountDisk /dev/disk2
$ cat proxmox-ve_7.0-1.iso > /dev/disk2

I am not going to explain how to set up Proxmox VE here as its installer is super slick and I am sure you are going to manage that by yourself.

Build encryption key

As promised, the encryption key for ZFS native encryption (and a second LUKS encryption key, if we're ever going to use that again) is only going to be stored at runtime in ramfs (volatile memory aka. RAM). It is built upon first login as root. So let's hook below script into /root/.profile:

echo "/usr/local/sbin/build-encryption-key.sh" >> /root/.profile

Deploy the following script to /usr/local/sbin/build-encryption-key.sh:

#!/bin/bash
#
# This script usually is called on the first login and asks for a password
# to build the LUKS + ZFS encryption keys which then are stored only in volatile
# memory /mnt/ramfs (ramfs).
# This script is added to your /root/.profile in order you won't forget to
# build the encryption key each time you reboot the server.
#
# We are using ramfs instead of tmpfs as there is no swapping support in 
# ramfs which is good in a security perspective.
# see: http://www.thegeekstuff.com/2008/11/overview-of-ramfs-and-tmpfs-on-linux
#

################### CONFIGURATION #######################
RAMFS_PATH='/mnt/ramfs'
RAMFS_SIZE='20M'
LUKS_KEYFILE=$RAMFS_PATH/luks_pw
ZFS_KEYFILE=$RAMFS_PATH/zfs_enc_key
SALT='**************************************************'
SHA1_CHECK='fb9e740efe20f541349d37eff7aa34efd4ac823d'
#########################################################

printinfo() {
    echo "[INFO] $1"
}

printwarn() {
    echo "[WARNING] $1" | grep --color "WARNING"
}

if [[ -f "$LUKS_KEYFILE" && -f "$ZFS_KEYFILE" ]]; then
    # exit silently, as the key file already exists
    exit
else
    printwarn "The LUKS ($LUKS_KEYFILE) and ZFS ($ZFS_KEYFILE) key files don't yet exist!"
fi

# set up RAM disk if it is not yet mounted
if ! mountpoint -q "$RAMFS_PATH"; then
    echo "Setting up ramfs on $RAMFS_PATH (size=$RAMFS_SIZE) ..."
    mkdir -p $RAMFS_PATH
    mount -t ramfs -o size=$RAMFS_SIZE ramfs $RAMFS_PATH
    if ! grep -q "$RAMFS_PATH" /etc/fstab; then
        echo "Adding line to /etc/fstab to persist mounting of $RAMFS_PATH ..."
        echo "ramfs   $RAMFS_PATH              ramfs   defaults,size=$RAMFS_SIZE        0 0" >> /etc/fstab
    fi
fi

# make this script start on each login
SCRIPT=$(readlink -f $0)
if ! grep -q "$SCRIPT" /root/.profile; then
    echo "$SCRIPT" >> /root/.profile
fi

# get password from interactive user input
while read -s -p 'Unlock encryption keys: ' PASS && [[ $(echo -n "$PASS" | wc --chars) -lt 8 ]] ; do
    echo
    echo "Your password must be at least 8 characters long!"
done
echo

# calculate encryption key (SHA-512 hash of salt.password concatenation)
KEY=$(echo -n "$SALT.$PASS" | sha512sum | cut -d' ' -f1)

# store LUKS key file to ramfs
touch $LUKS_KEYFILE && chmod 600 $LUKS_KEYFILE
echo -n "$KEY" > $LUKS_KEYFILE

# SHA-1 check of the key - assure you have correctly built it by entering the correct password
KEY_SHA1=`cat $LUKS_KEYFILE | sha1sum | cut -d' ' -f1`
if [ "$KEY_SHA1" != "$SHA1_CHECK" ]; then
    printwarn "Your key does not seem to be correct. You might have entered the wrong password. Please run `basename $SCRIPT` again!"
    printinfo "If you are sure you have entered the right password, try to set SHA1_CHECK='$KEY_SHA1' in `basename $SCRIPT`."
    rm -f $LUKS_KEYFILE
    exit 1
fi

printinfo "The LUKS key file was successfully stored in $LUKS_KEYFILE."

# ZFS raw key needs to be 32 characters long
head -c 32 $LUKS_KEYFILE > $ZFS_KEYFILE
printinfo "The ZFS key file was successfully stored in $ZFS_KEYFILE."

############################
# SSH private key encryption
############################
#
# INITIAL SETUP:
#
# Initially, we need to create an encrypted version of id_rsa and destroy the plaintext private key:
#   $ cat ~/.ssh/id_rsa | openssl enc -e -aes-256-cbc -pbkdf2 -a -pass file:/mnt/ramfs/luks_pw > ~/.ssh/id_rsa.encrypted
#
# or you could also copy the password from /mnt/ramfs/luks_pw and enter it interactively:
#   $ cat ~/.ssh/id_rsa | openssl enc -e -aes-256-cbc -pbkdf2 -a > ~/.ssh/id_rsa.encrypted
#   $ enter aes-256-cbc encryption password: (...)
#   $ chmod 600 ~/.ssh/id_rsa.encrypted
#   $ shred -u ~/.ssh/id_rsa
#
# Updating the encrypted version of id_rsa:
#   $ cat /mnt/ramfs/id_rsa.decrypted | openssl enc -e -aes-256-cbc -pbkdf2 -a -pass file:/mnt/ramfs/luks_pw > ~/.ssh/id_rsa.encrypted
# Testing encryption and decryption:
#   $ cat /mnt/ramfs/id_rsa.decrypted | openssl enc -e -aes-256-cbc -pbkdf2 -a -pass file:/mnt/ramfs/luks_pw > ~/.ssh/id_rsa.encrypted-testing
#   $ cat ~/.ssh/id_rsa.encrypted-testing | openssl base64 -d | openssl enc -d -aes-256-cbc -pbkdf2 -pass file:/mnt/ramfs/luks_pw > /mnt/ramfs/id_rsa.testing
#   $ diff /mnt/ramfs/id_rsa.decrypted /mnt/ramfs/id_rsa.testing
#   $ shred -u /mnt/ramfs/id_rsa.testing
#
if [ -f ~/.ssh/id_rsa.encrypted ]; then
    touch $RAMFS_PATH/id_rsa.decrypted && chmod 600 $RAMFS_PATH/id_rsa.decrypted
    cat ~/.ssh/id_rsa.encrypted | openssl base64 -d | openssl enc -d -aes-256-cbc -pbkdf2 -pass file:$LUKS_KEYFILE > $RAMFS_PATH/id_rsa.decrypted
    ln -sf $RAMFS_PATH/id_rsa.decrypted ~/.ssh/id_rsa
    printinfo "SSH private key was successfully decrypted to $RAMFS_PATH/id_rsa.decrypted."
else
    printwarn "Please encrypt your SSH private key to ~/.ssh/id_rsa.encrypted so I can decrypt it to ramfs."
fi

But before using this script in production, ensure you have set your own SALT in the CONFIGURATION section. Don't yet care about SHA1_CHECK - this is only used to verify that the key got correctly built. Upon first run, the script will provide you with the right information. The unlock password can be freely chosen, just please remember it!:

$ build-encryption-key.sh
Unlock encryption keys: (choose a new unlock password)
[WARNING] Your key does not seem to be correct. You might have entered the wrong password. Please run build-encryption-key.sh again!
[INFO] If you are sure you have entered the right password, try to set SHA1_CHECK='fb9e740efe20f541349d37eff7aa34efd4ac823d' in build-encryption-key.sh.

The unlock password is only needed to unlock/generate your LUKS + ZFS encryption keys and is only used as a second factor. For ZFS native encryption, we are then only going to use the generated /mnt/ramfs/zfs_enc_key which you should not store anywhere else!!!

Set correct SHA1_CHECK (from above output) in /usr/local/sbin/build-encryption-key.sh and test if the script gets correctly invoked upon first login as root:

extbackup$ sudo su -
[WARNING] The LUKS (/mnt/ramfs/luks_pw) and ZFS (/mnt/ramfs/zfs_enc_key) key files don't yet exist!
Unlock encryption keys: 
[INFO] The LUKS key file was successfully stored in /mnt/ramfs/luks_pw.
[INFO] The ZFS key file was successfully stored in /mnt/ramfs/zfs_enc_key.

Set up SSH keypair with encrypted private key

You may have noticed that our build-encryption-key.sh script also tries to decrypt your SSH private key to ramfs upon first login as root. We are going to use the default identity file ~/.ssh/id_rsa to access remote backup server over SSH (using pve-zsync to pull data). But we don't want the private key to get stored plaintext anywhere on persistent storage.

build-encryption-key.sh cares about the following:

• Asks for encryption key password
• Builds LUKS encryption key /mnt/ramfs/luks_pw using a seed and the provided password
• Derives ZFS encryption key /mnt/ramfs/zfs_enc_key from LUKS encryption key (first 32 chars)
• Decrypts the encrypted SSH private key of user root ~/.ssh/id_rsa.encrypted to /mnt/ramfs/id_rsa.decrypted

Initially, you need to do this in two steps, as we did not yet encrypt id_rsa private key yet.

Also see setup instructions in comments inside build-encryption-key.sh!

$ ssh extbackup
$ sudo su -
[WARNING] The LUKS (/mnt/ramfs/luks_pw) and ZFS (/mnt/ramfs/zfs_enc_key) key files don't yet exist!
Unlock encryption keys: 
[INFO] The LUKS key file was successfully stored in /mnt/ramfs/luks_pw.
[INFO] The ZFS key file was successfully stored in /mnt/ramfs/zfs_enc_key.
[WARNING] Please encrypt your SSH private key to ~/.ssh/id_rsa.encrypted so I can decrypt it to ramfs.

# regenerate RSA/4096 keypair, if not already done
$ ssh-keygen -t rsa -b 4096 -P ''

# encrypt SSH private key
$ cd /root/.ssh/
$ cat id_rsa | openssl enc -e -aes-256-cbc -pbkdf2 -a -pass file:/mnt/ramfs/luks_pw > id_rsa.encrypted
$ chmod 600 id_rsa.encrypted

Let's now secure erase our plaintext private key:

$ shred -u /root/.ssh/id_rsa

Testing, logout and login again:

$ rm -f /mnt/ramfs/*
$ logout
$ sudo su -
[WARNING] The LUKS key file (/mnt/ramfs/luks_pw) does not yet exist!
Unlock LUKS encryption key: **********
[INFO] The LUKS key file was successfully stored in /mnt/ramfs/luks_pw.
[INFO] The ZFS key file was successfully stored in /mnt/ramfs/zfs_enc_key.
[INFO] SSH private key was successfully decrypted to /mnt/ramfs/id_rsa.decrypted.

Your keypair is now ready. You can now copy-paste your public key to remote backup server:

$ cat /root/.ssh/id_rsa.pub

Append it to /root/.ssh/authorized_keys on backup server.

Encrypt external drive: Setup dpool

• Encrypting ZFS File Systems
  • Examples of Encrypting ZFS File Systems
  • How-To: Using ZFS Encryption at Rest in OpenZFS (ZFS on Linux, ZFS on FreeBSD, …)

NOTE: We only need to set up the first external drive. To set up further external drives, use mirroring as described in the final chapter below.

QUICK HOWTO: Initialize a new external drive /dev/sdb

# encrypt disk / set up dpool (use -f to force, if dpool already exists)
$ zpool create -O acltype=posixacl -O encryption=on -O keylocation=file:///mnt/ramfs/zfs_enc_key -O keyformat=raw dpool sdb
# check status
$ zfs get encryption,keylocation,keystatus,keyformat dpool

# set up LABEL and dataset(s)
$ echo -n "1a" > /dpool/LABEL
$ zfs create dpool/zfsdisks

# unmount and export (only disconnect disk, after exporting dpool!)
$ zfs unmount dpool && zfs unload-key dpool
$ zpool export dpool

In more detail...

Initialize a new external drive with ZFS native encryption, putting everything under encrypted pool dpool (name stands for "data-pool", while rpool is the default system "root-pool" in Proxmox VE):

The aes-256-ccm algorithm is used by default (don't be confused about misleading ZFS documentation which says aes-128-ccm)

# use lsblk to check device names
$ lsblk

# create dpool on /dev/sdb
# NOTE: key must be exactly 32 characters long, so use /mnt/ramfs/zfs_enc_key (first 32 chars of luks_pw)
$ zpool create -O acltype=posixacl -O encryption=on -O keylocation=file:///mnt/ramfs/zfs_enc_key -O keyformat=raw dpool sdb

# check pool and encrytion status
$ zpool status dpool
  pool: dpool
 state: ONLINE
config:
    NAME        STATE     READ WRITE CKSUM
    dpool       ONLINE       0     0     0
      sdb       ONLINE       0     0     0

$ zfs get encryption,keylocation,keystatus,keyformat dpool
NAME   PROPERTY     VALUE                          SOURCE
dpool  encryption   aes-256-gcm                    -
dpool  keylocation  file:///mnt/ramfs/zfs_enc_key  local
dpool  keystatus    available                      -
dpool  keyformat    raw                            -

Unmount/encrypt dpool like this:

$ zfs unmount dpool
$ zfs unload-key dpool
$ zpool export dpool

WARNING: Never forget to export dpool before you detach the external drive or before you reboot!

Mount/decrypt dpool like this:

$ zpool import dpool
$ zfs load-key dpool
$ zfs mount -l -a

man zfs:

zfs mount [-Olv] [-o options] -a | filesystem
 -a  Mount all available ZFS file systems.  Invoked automatically as
     part of the boot process if configured.
 -l  Load keys for encrypted filesystems as they are being mounted. This
     is equivalent to executing zfs load-key on each encryption root be‐
     fore mounting it. Note that if a filesystem has a keylocation of
     prompt this will cause the terminal to interactively block after
     asking for the key.

Finally, create a dataset on dpool - it will also be encrypted:

$ zfs create dpool/zfsdisks

$ zfs get encryption,keystatus dpool/zfsdisks
NAME            PROPERTY    VALUE        SOURCE
dpool/zfsdisks  encryption  aes-256-gcm  -
dpool/zfsdisks  keystatus   available    -

Set up mounting aliases

Let's set up aliases for the common used mounting commands in /etc/profile.d/zfs-encryption.sh:

alias mount-extbackup='zpool import dpool && zfs load-key dpool && zfs mount -l -a'
alias umount-extbackup='zfs unmount dpool && zfs unload-key dpool && zpool export dpool'

We can now use those aliases to mount/unmount our external drive to /dpool:

$ mount-extbackup
$ cat /dpool/LABEL
$ umount-extbackup

Limit SSH access to pve-zsync operations

For security reasons, we want to limit SSH access on remote backup server only to the very specific commands which pve-zsync fires on remote side.

In below extbackup-zfs.sh backup script, pve-zsync is executed as follows, e.g. for VMID 181:

$ pve-zsync sync --source fatboy:dpool/zfsdisks/subvol-181-disk-1 --dest rpool/zfsdisks/subvol-181-disk-1 --maxsnap 12 --name extbackup

This produces the following remote commands...

Sample pve-zsync remote commands on first run:

$ zfs list -r -t snapshot -Ho name -S creation dpool/zfsdisks/subvol-181-disk-1
$ zfs snapshot dpool/zfsdisks/subvol-181-disk-1@rep_extbackup_2021-07-12_23:15:53
$ zfs send -- dpool/zfsdisks/subvol-181-disk-1@rep_extbackup_2021-07-12_23:15:53

Sample pve-zsnyc remote commands on the following runs:

$ zfs list -rt snapshot -Ho name dpool/zfsdisks/subvol-181-disk-1@rep_extbackup_2021-07-12_23:15:53
$ zfs send -i dpool/zfsdisks/subvol-181-disk-1@rep_extbackup_2021-07-12_23:15:53 -- dpool/zfsdisks/subvol-181-disk-1@rep_extbackup_2021-07-12_23:48:52
$ zfs destroy dpool/zfsdisks/subvol-181-disk-1@rep_extbackup_2021-07-12_23:13:13

To accomplish this specific restrictions, we need to use a script in command section before the extbackup's pubkey in .ssh/authorized_keys:

command="/root/.ssh/allowed-commands-onlime.sh",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-rsa AAAAB3N... root@extbackup

allowed-commands-onlime.sh then does pattern matching to verify the commands - use $SSH_ORIGINAL_COMMAND to get the original command:

#!/bin/bash
#
# https://serverfault.com/a/803873/299863
# You can have only one forced command in ~/.ssh/authorized_keys. Use this wrapper to allow several commands.
# And use https://3widgets.com/ - Regex Numeric Range Generator
#

## CONFIGURATION ##############################################
# host range 194-254
veid_pattern='(19[4-9]|2[0-4][0-9]|25[0-4])'
backup_srcdir='/backup'
pool_name=dpool
###############################################################

# Regex patterns
ds_pattern="${pool_name}/zfsdisks/subvol-${veid_pattern}-disk-1"
ts_pattern='20[2-9][0-9]-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])_(0[0-9]|1[0-9]|2[0-3])(:(0[0-9]|[1-5][0-9])){2}'
snap_prefix_pattern='rep_extbackup_([1-9]a_)?'
snap_pattern="@${snap_prefix_pattern}${ts_pattern}"
ds_snap_pattern="${ds_pattern}${snap_pattern}"

patt1='zfs list -r( -)?t snapshot -Ho name( -S creation)? '"$ds_pattern($snap_pattern)?"
patt2='zfs (snapshot|destroy) '"$ds_snap_pattern"
patt3='zfs send '"(-i $ds_snap_pattern )?-- $ds_snap_pattern"
patt4="zfs rename $ds_snap_pattern $ds_snap_pattern" # only used in extbackup-migrate-zfs-snaps.sh

cmd="$SSH_ORIGINAL_COMMAND"

if [[ "$cmd" == 'list-datasets' ]]; then
    # special command to get a server list of all datasets and mountpoints
    zfs list -H -o name,mountpoint,usedds | grep $backup_srcdir
    exit
elif [[ $cmd =~ ^$patt1$ || $cmd =~ ^$patt2$ || $cmd =~ ^$patt3$ || $cmd =~ ^$patt4$ ]]; then
    $SSH_ORIGINAL_COMMAND
    exit
else
    logger "$(basename $0) violation: $cmd"
    echo "Access denied"
    exit 1
fi

We can then test the magic list-datasets command to get a server list of all datasets and mountpoints:

extbackup$ ssh backup list-datasets

This magic command is going to be used by our extbackup script to loop over ZFS datasets that need to be backupped.

Final extbackup script

The following script basically decrypts and mounts the external USB device, then runs pve-zsync to pull the whole data from backup for each dataset, and unmounts / closes the encrypted device right afterwards.

extbackup-zfs.sh:

#!/bin/bash

######### CONFIGURATION ############
USB_DEVICE=/dev/sdb
DISKS=3
DATAPOOL=dpool
BKUP_SERVER="backup"
BKUP_SRCDIR="/backup"
MAXSNAP=1
KEYFILE="/mnt/ramfs/zfs_enc_key"
LOGFILE="/var/log/extbackup.log"
EXCLUDES=() # datasets that should be excluded from backup
SCRIPTNAME=$(basename $0)
CREATE_RERUN=1 # default to creating rerun on failure
RERUN_FILE="/tmp/$SCRIPTNAME.rerun"
####################################

usage() {
    echo "Usage:"
    echo "$(basename $0) [-m|--maxsnap N] [-r|--rerun]"
    echo
    echo "Where:"
    echo "  -m|--maxsnap N  The number of snapshots to keep until older ones are erased (default: 1)"
    echo "  -r|--rerun      Rerun mode: rerun only if previous run has failed"
    exit 1
}

# Get options
while :
do
    case $1 in
        -m | --maxsnap)
            MAXSNAP="${2:-$MAXSNAP}"
            shift
            shift
            ;;
        -r | --rerun)
            RERUN="true"
            shift
            ;;
        -h | --help)
            usage
            ;;
        *)  # no more options. Stop while loop
            break
            ;;
    esac
done

# include common functions
source /usr/local/sbin/common-functions.sh
# ensure this is the only running instance
assert_single_instance
# append STDOUT and STDERR to logfile and output both at the same time
logall_output_stderr_stdout $LOGFILE

# zfs command convienence aliases
zfsgetval='zfs get -H -o value'

# Rerun if `--rerun` was provided and previous run has failed
if [ -n "$RERUN" ]; then
    if [[ ! -f $RERUN_FILE ]]; then
        # printinfo "did not find rerun file $RERUN_FILE; exiting OK"
        exit
    fi
elif [[ $CREATE_RERUN -eq 1 ]]; then
    touch $RERUN_FILE
fi

# check if key file exists
if [ ! -f $KEYFILE ]; then
    errquit "The key file $KEYFILE does not exist yet. Please run build-encryption-key.sh first!"
fi

# First, make sure /dev/sdb will wake up and actually exists (using parted as a simple workaround)
parted -s $USB_DEVICE print > /dev/null
if [ $? -ne 0 ]; then
    errquit "External USB disk does not seem to show up as $USB_DEVICE. Backup script aborted!"
fi

# Import dpool, load encrytion key and mount all pools
printinfo "Importing ZFS pool $DATAPOOL ..."
zpool import $DATAPOOL
if [ $? -ne 0 ]; then
    errquit "Could not import $DATAPOOL. Backup script aborted!"
fi
printinfo "Loading key for ZFS pool $DATAPOOL ..."
zfs load-key $DATAPOOL
printinfo "Mouting/decrypting all ZFS datasets ..."
zfs mount -l -a

# Check if decryption/mounting of dpool was done
if [ $? -ne 0 ]; then
    errquit "Could not mount/decrypt ZFS filesystems. Backup script aborted!"
fi
if [ "$($zfsgetval keystatus $DATAPOOL)" != "available" ]; then
    errquit "ZFS dataset $DATAPOOL does not seem to be correctly decrypted. Backup script aborted!"
fi

# abort on missing /dpool/LABEL or wrongly formatted label
if [ ! -f /$DATAPOOL/LABEL ]; then
    errquit "Disk label (/$DATAPOOL/LABEL) does not exist. Backup script aborted!!!"
fi
label=$(cat /$DATAPOOL/LABEL)
if [[ ! $label =~ ^[1-$DISKS]a$ ]]; then
    errquit "Invalid label in /$DATAPOOL/LABEL: $label"
fi

printinfo "=== STARTING EXTBACKUP TO DISK WITH LABEL $label ==="

# loop over all datasets and run pve-zsync
# `list-datasets` is a magic command that is resolved to `zfs list -H -o name,mountpoint,usedds | grep /backup` on remote
ssh $BKUP_SERVER list-datasets | while read dataset mountpoint usedds; do
    zsync="pve-zsync sync --source $BKUP_SERVER:$dataset --dest $(dirname $dataset) --maxsnap $MAXSNAP --name extbackup_$label"
    # check if the dataset should be excluded from extbackup
    if [[ ${EXCLUDES[*]} =~ $dataset ]]; then
        printwarn "(SKIPPED) $zsync"
        continue
    fi

    # check if dataset exists
    if zfs list $dataset &>/dev/null; then
        # unmount dataset prior to pve-zsync run to avoid "dataset is busy"
        zfs unmount $dataset
    fi
    printinfo "$zsync # $usedds"
    # use `< /dev/null` as workaround to avoid pve-zsync breaking out of while loop (as it somehow expects input)
    $zsync < /dev/null

    # set mountpoint to the same as on remote backup server
    if [ "$($zfsgetval mountpoint $dataset)" != "$mountpoint" ]; then
        cmd="zfs set mountpoint=$mountpoint $dataset"
        printinfo "$cmd"
        $cmd
        sleep 1
    else
        zfs mount $dataset
    fi
done

printinfo "Unmounting/encrypting ZFS pool $DATAPOOL ..."
zfs unmount $DATAPOOL
printinfo "Unloading key for ZFS pool $DATAPOOL ..."
zfs unload-key $DATAPOOL
printinfo "Exporting ZFS pool $DATAPOOL ..."
zpool export $DATAPOOL
if [ $? -ne 0 ]; then
    errquit "Could not export $DATAPOOL. Please fix!"
fi

# Remove rerun file on successful run
rm -f $RERUN_FILE

sync
printinfo "DONE."

common-functions.sh:

#!/bin/bash

####### CONFIGURATION #################
TSFORMAT="%Y-%m-%d %H:%M:%S"
EXECUTING_SCRIPT=$(basename $0)
#######################################

printinfo() {
    echo "[`date +"$TSFORMAT"`] $1"
}

printwarn() {
    echo "[`date +"$TSFORMAT"`] WARNING: $1" | grep --color "WARNING"
}

printerr() {
    >&2 echo "[`date +"$TSFORMAT"`] ERROR: $1"
}

errquit() {
    if [ -n "$1" ]; then
        printerr "$1"
    fi
    if [ -n "$LOCKFILE" ]; then
        rm -f $LOCKFILE
    fi
    exit 1
}

assert_single_instance() {
    # Check if another instance of this script is already running
    # https://stackoverflow.com/a/16807995/5982842
    for pid in $(pidof -x $EXECUTING_SCRIPT); do
        if [ $pid != $$ ]; then
            errquit "$EXECUTING_SCRIPT process is already running with PID $pid"
        fi
    done
}

logall_output_stderr_stdout() {
    logfile=${1:-${EXECUTING_SCRIPT}.log}
    # Append STDOUT and STDERR to logfile and output both at the same time
    exec >  >(tee -ia $logfile)
    exec 2> >(tee -ia $logfile >&2)
}

Run this as a weekly cronjob, e.g. via /etc/cron.d/extbackup, starting Sat morning:

# weekly extbackup on encrypted USB drive (Sat 07:30AM), rerun on failure the next 3 days
30 07   * * 6   root    extbackup-zfs.sh
30 07   * * 0-2 root    extbackup-zfs.sh --rerun    

On first run (e.g. in a screen session instead of via cronjob), make sure you follow the log:

$ tail -f /var/log/extbackup.log

Set up another encrypted external drive

We want to rotate several external drives, storing them in different physical locations. Setting up an additional drive should not require us to transfer all data again from backup server (which in our case was > 4TB of data that needed to get transferred between two datacenters). We want to start from the latest external drive and replicate its data instead and then only continue with incremental backups.

Idea

Problem: ZFS does not support duplicating/copying of existing snapshots on the same dataset (with a new snapshot name, obviously). So, the initial idea was to set up encryption on new disk first, then connect first disk and import secondary as dpool2, then copying all datasets over using zfs send|receive. That would force us to write another complex script. Way to complicated! Let's use mirroring to replicate the full disk instead!

Solution:

• Part 1) Replicate Disk
  1. Connect existing disk with LABEL 1a and import dpool: zpool import dpool
  2. Connect new (empty) disk and attach it to dpool, creating a mirror: zpool attach dpool sdb sdd
  3. Wait until resilvering has completed
  4. Split mirror again to have new disk in a separate dpool2: zpool split dpool dpool2
  5. Export first dpool, disconnect first disk, and import dpool2 by renaming it to dpool: zpool import dpool2 dpool
  6. Export dpool (now the new final replicated disk), never forget exporting before detaching a disk!
• Part 2) Renaming LABEL / snapshots
  1. Rember, we have a full copy of initial disk with LABEL 1a on new disk which should be renamed to 2a
  2. mount/decrypt new disk and check label (still must be the same as the previous disk!), unmount/encrypt
  3. run extbackup-zfs.sh --maxsnap 2 with old LABEL, to create new snapshots that can be renamed (and keeping first ones alive on backup for first disk)
  4. change LABEL
  5. run extbackup-migrate-zfs-snaps.sh to rename all local and remote (backup) snapshots to new LABEL
  6. purge local snapshots with old label by running extbackup-migrate-zfs-snaps.sh --purge
  7. unmount/encrypt disk

Part 1) Replicate a disk

Replicate a full disk by mirorring dpool :

# On extbackup with 1st disk (LABEL 1a) connected
$ zpool import dpool
$ zpool status dpool
  pool: dpool
 state: ONLINE
config:
    NAME        STATE     READ WRITE CKSUM
    dpool       ONLINE       0     0     0
      sdb       ONLINE       0     0     0

# Connect 2nd disk (already partitioned or not should not matter)
$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sdb      8:16   0   7.3T  0 disk 
├─sdb1   8:17   0   7.3T  0 part 
└─sdb9   8:25   0    64M  0 part 
sdd      8:48   0   7.3T  0 disk 

# Attach new device to existing zpool, creating a mirror
$ zpool attach dpool sdb sdd
# (maybe) need to use --force if the disk was already initialized before
$ zpool attach dpool sdb sdd -f
# Resilvering starts immediately.

# Wait until fully resilvered ...
$ zpool status dpool
  pool: dpool
 state: ONLINE
  scan: resilvered 4.73T in 09:15:53 with 0 errors on ...
config:
    NAME        STATE     READ WRITE CKSUM
    dpool       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        sdb     ONLINE       0     0     0
        sdd     ONLINE       0     0     0

You can use the zpool split command to detach disks from a mirrored ZFS storage pool to create a new pool with one of the detached disks. The new pool will have identical contents to the original mirrored ZFS storage pool:

$ zpool split dpool dpool2
# dpool2 is not imported automatically, which is what we want!

# now, export dpool to free this pools name
$ zpool export dpool

# rename new dpool2 to dpool and export it again
$ zpool import dpool2 dpool
$ zpool export dpool

# disconnect both disks and list importable pools after reconnecting the second disk (should then show up as /dev/sdb)
$ zpool import
   pool: dpool
     id: 9764679279839179262
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:
    dpool                                          ONLINE
      ata-Samsung_SSD_870_QVO_8TB_S5SSNF0R201266B  ONLINE

# import it to check status
$ zpool import dpool
$ zpool status dpool
  pool: dpool
 state: ONLINE
  scan: resilvered 4.61T in 08:45:42 with 0 errors on ...
config:
    NAME                                           STATE     READ WRITE CKSUM
    dpool                                          ONLINE       0     0     0
      ata-Samsung_SSD_870_QVO_8TB_S5SSNF0R201266B  ONLINE       0     0     0

$ zpool export dpool

All cool! We now have two identical disks, both still having label set to 1a in /dpool/LABEL.

Part 2) Renaming LABEL / snapshots

Now, set up new replicated disk:

1. mount/decrypt new disk and check label (still must be the same as the previous disk!), unmount/encrypt
2. run extbackup-zfs.sh --maxsnap 2 with old LABEL, to create new snapshots that can be renamed (and keeping first ones alive on backup for first disk).
3. change LABEL to 2a, 3a,...
4. run extbackup-migrate-zfs-snaps.sh to rename all local and remote (backup) snapshots to new LABEL
5. purge local snapshots with old label by running extbackup-migrate-zfs-snaps.sh --purge
6. unmount/encrypt disk

For mounting/dencrytion and encryption/unmounting, we're using the following aliases in below commands:

alias mount-extbackup='zpool import dpool && zfs load-key dpool && zfs mount -l -a'
alias umount-extbackup='zfs unmount dpool && zfs unload-key dpool && zpool export dpool'

$ mount-extbackup
# check LABEL (must match previous disk, DON'T CHANGE IT YET!) and review snapshots
$ cat /dpool/LABEL
1a
$ zfs list -H -t snapshot -o name

# ... and unmount it again (to be ready for extbackup-zfs.sh)
$ umount-extbackup

# run extbackup by keeping 2 snapshots (and still pretending we're 1a, keeping old LABEL !!)
$ screen
$ extbackup-zfs.sh --maxsnap 2

# after extbackup completion change LABEL
$ mount-extbackup
$ echo -n "2a" > /dpool/LABEL

# Rename latest @rep_extbackup_1a_* snapshots to @rep_extbackup_2a_* (new LABEL)
$ extbackup-migrate-zfs-snaps.sh --dryrun
# This will rename them both locally and on remote backup server
$ extbackup-migrate-zfs-snaps.sh

# destroy old snapshots (only on extbackup, KEEP THEM ON backup !!!)
$ extbackup-migrate-zfs-snaps.sh --purge --dryrun
$ extbackup-migrate-zfs-snaps.sh --purge
# review (there should now only be @rep_extbackup_2a_* snapshots)
$ zfs list -H -t snapshot -o name | grep -v @rep_extbackup_2a

# unmount/encrypt disk
$ umount-extbackup

The disk can now be unplugged and is ready for the next backup run.

Helper script extbackup-migrate-zfs-snaps.sh:

#!/bin/bash
######### CONFIGURATION ############
DISKS=3
DATAPOOL=dpool
OLDLABEL=1a
BKUP_SERVER="backup"
####################################

usage() {
    echo "Usage:"
    echo "$(basename $0) [-l|--label OLDLABEL]"
    echo
    echo "Where:"
    echo "  -l|--label OLDLABEL  The label of the disks where the previous snapshots have been performed (default: 1a)"
    echo "  -s|--dryrun          Dry-run / simulation mode: Don't rename any snapshots"
    echo "  -p|-purge            Purge all snapshots that don't match current disk label (only run this as a final migration!)"
    exit 1
}

# Get options
while :
do
    case $1 in
        -l | --label)
            OLDLABEL="${2:-$OLDLABEL}"
            shift
            shift
            ;;
        -s | --dryrun)
            DRYRUN="true"
            shift
            ;;
        -p | --purge)
            PURGE="true"
            shift
            ;;
        -h | --help)
            usage
            ;;
        *)  # no more options. Stop while loop
            break
            ;;
    esac
done

# include common functions
source /usr/local/sbin/common-functions.sh

# zfs command convienence aliases
zfslistname='zfs list -H -o name'

# abort on missing label
if [ ! -f /$DATAPOOL/LABEL ]; then
    errquit "Disk label (/$DATAPOOL/LABEL) does not exist. Backup script aborted!!!"
fi
label=$(cat /$DATAPOOL/LABEL)
if [[ ! $label =~ ^[1-$DISKS]a$ ]]; then
    errquit "Invalid label in /$DATAPOOL/LABEL: $label"
fi

dspattern="$DATAPOOL/zfsdisks/[[:alnum:]-]+"
for dataset in $($zfslistname -t filesystem -d 2 $DATAPOOL); do
    if [[ $dataset =~ ^$dspattern$ ]]; then
        # local purge run (as a final migration)
        if [ -n "$PURGE" ]; then
            for snapshot in $($zfslistname -t snapshot -s creation $dataset | grep @rep_extbackup_${OLDLABEL}_); do
                cmd="zfs destroy $snapshot"
                if [ -z "$DRYRUN" ]; then
                    printinfo "$cmd"
                    $cmd
                else
                    printinfo "(DRYRUN) $cmd"
                fi
            done
            continue # don't proceed with snapshot renaming below, as that is part of first migration
        fi

        # snapshot renaming (first migration)
        snapold=$($zfslistname -t snapshot -s creation $dataset | grep @rep_extbackup | tail -n1)
        if [[ -z "$snapold" ]]; then
            printwarn "Skipping $dataset: no extbackup snapshot found."
            continue
        fi
        snapnew="${snapold/@rep_extbackup_$OLDLABEL/@rep_extbackup_$label}"
        cmd="zfs rename $snapold $snapnew"
        if [ -z "$DRYRUN" ]; then
            printinfo "$cmd"
            $cmd
            if [ $? -ne 0 ]; then
                errquit "Failed to locally rename snapshot. Migration script aborted!"
            fi
            ssh $BKUP_SERVER $cmd
            if [ $? -ne 0 ]; then
                errquit "Failed to remotely rename snapshot on $BKUP_SERVER. Migration script aborted!"
            fi
        else
            printinfo "(DRYRUN) $cmd"
        fi
    fi
done