Proxmox VE 4.x OpenVZ to LXC Migration
At Onlime Webhosting we chose ProxmoxVE as our favorite virtualization platform and are running a bunch of OpenVZ containers for many years now, with almost zero issues. We very much welcome the small overhead and simplicity of container based virtualization and wouldn’t want to move to anything else. ProxmoxVE added ZFS support by integrating ZFSonLinux back in Feb 2015 with the great ProxmoxVE 3.4 release – which actually would have deserved to bump its major version because of this killer feature.
OpenVZ on PVE
Previously we have been running our OpenVZ containers on plain Debian Linux boxes. But with Debian Wheezy (7.0) the Debian community decided to no longer include a kernel which has been patched with the OpenVZ extensions. Switching to ProxmoxVE at that time was a piece of cake as ProxmoxVE was just a plain Debian Linux with a RHEL kernel, OpenVZ and KVM support, and some nice web interface to manage the containers and VMs. Actually, we never really needed the ProxmoxVE web interface (GUIs always suck!), but that one is quite lightweight and very well integrated. If you don’t like it, use the provided CLI tools – ProxmoxVE does not force you to use the web interface at all.
In 2014 I did a review for Proxmox High Availability (by PACKT publishing). But at Onlime, we never employed any cluster technologies to date. I was always looking for simple solutions without unnecessary complexity. ProxmoxVE with OpenVZ on ZFS pools was the perfect match. We are replicating our containers from one host node to another every 10mins with simple and super fast
zfs send|receive via a modified version of ZREP.
In ProxmoxVE 4, OpenVZ support was removed in favor of LXC. Both, KVM and LXC are built into any newer Linux kernel, so in the long term (and due to the nature of OpenVZ as a huge kernel patch which is hard to maintain) it was clear we had to give up OpenVZ. Even though the OpenVZ developers are currently focusing on merging the OpenVZ and Virtuozzo source codebase and have just released OpenVZ 7.0 I don’t give OpenVZ a long future any more. In comparison to LXC, the tools for OpenVZ are still much more mature and powerful, but lately LXC made a lot of progress and having it perfectly integrated into ProxmoxVE is a great step forward. So, we have decided to migrate to LXC.
(Sorry about the long preface. You probably don’t care about history reading this blog post. I am getting to the point now…)
The clumsy (official) migration path
What I cannot believe and what prevented me to look into LXC on ProxmoxVE up to now is the clumsy migration path that ProxmoxVE suggests on ProxmoxVE Wiki: Convert OpenVZ to LXC. Basically they suggest to dump and restore a full container as follows:
pve1$ vzctl stop 100 && vzdump 100 -storage local pve1$ scp /var/lib/vz/dump/vzdump-openvz-100.tar pve2:/var/lib/vz/dump/ pve2$ pct restore 100 /var/lib/vz/dump/vzdump-openvz-100.tar
Srsly? Do you guys only have containers with sizes below 1GB and don’t care about downtime at all? Real world looks different and I would not even go this migration path with a 5GB container. I don’t even want to think about those containers with ~ 1TB of data.
Recommended migration to LXC
If you are running OpenVZ containers on ProxmoxVE, sure you have some easy solution to migrate your containers from one host node to another. As I said, we are using ZREP for container replication and have written a wrapper script that migrates a container to its failover host node. That’s what I suggest and please forget about live migration – it never really was working in OpenVZ (well, it was, but I am talking about a really stable solution that never fails) and LXC lacks any kind of hot/live migration. Doing a ZREP presync, stopping the container on the source host node, doing the main ZREP sync, finally starting the container on the destination host node is just a matter of seconds, usually causes a downtime of 10s up to 30s max (on a larger container).
Here’s the migration path I suggest for any ProxmoxVE setup with containers running on ZFS pools:
- move all OpenVZ containers to the primary host node
- upgrade the secondary host node to ProxmoxVE 4.x
- prepare the new LXC container on the secondary host node, don’t start it yet
- replace the newly created ZFS pool with the existing (which is still in sync with ZREP replication)
- do a
vzctl stop $VEID && zrep failover rpool/ROOT/pve-$VEIDto activate (rw) the ZFS pool on the secondary host node
- start the LXC container on the secondary host node
- once you have migrated all containers to LXC on the secondary host node, upgrade the primary host node to ProxmoxVE 4.x
On a new or upgraded ProxmoxVE host node, first configure ZFS storage:
pve2$ zfs create rpool/zfsdisks # turn on posixacl by default for all containers (inherited) pve2$ zfs set acltype=posixacl rpool/zfsdisks pve2$ pvesm add zfspool zfsvols -pool rpool/zfsdisks -content images,rootdir -sparse
Check the storage configuration,
dir: local path /var/lib/vz maxfiles 0 content rootdir,iso,images,vztmpl zfspool: zfsvols pool rpool/zfsdisks content images,rootdir sparse
Creating a new LXC container can be done via ProxmoxVE web interface or (my preference) via CLI. We are using the new zfsvols storage –
pct takes care of creating the ZFS pool for you:
pve2$ pct create 100 local:vztmpl/debian-8.0-standard_8.4-1_amd64.tar.gz \ -hostname demo.onlime.ch \ -cpulimit 2 \ -cpuunits 1024 \ -memory 4096 \ -swap 1024 \ -ostype debian \ -rootfs zfsvols:subvol-100-disk-1,acl=1 \ -net0 name=eth0,bridge=vmbr0,gw=X.X.X.X,ip=X.X.X.100/25 \ -storage zfsvols
You probably want to write a short helper script to simplify this step as most parameters will stay the same for all containers.
Do not try to change the subvol-VMID-disk-1 naming scheme of the LXC container, see Dietmar Maurer’s comment:
I guess we hardcoded that naming scheme a several places, so it would be hard to change that. We also use disk names to encode some informations, so we cannot allow arbitrary name changes.
We now have both ZFS pools on the secondary host node:
pve2$ zfs list NAME USED AVAIL REFER MOUNTPOINT rpool/ROOT/pve-100 560M 436G 529M /var/lib/vz/private/184 rpool/zfsdisks/subvol-100-disk-1 560M 436G 529M /rpool/zfsdisks/subvol-100-disk-1
Migration is now that simple, just replace the ZFS pool with your existing data (without dumping/backupping/restoring anything!):
pve2$ VMID=100 pve2$ zfs destroy -r rpool/zfsdisks/subvol-$VMID-disk-1 pve2$ zfs rename rpool/ROOT/pve-$VEID rpool/zfsdisks/subvol-$VMID-disk-1 pve2$ zfs set mountpoint=/rpool/zfsdisks/subvol-$VMID-disk-1 rpool/zfsdisks/subvol-$VMID-disk-1
After adjusting the
zrep:dest-fs ZFS property on both source and destination host, you can stop the OpenVZ container, replicate it over, and start it as LXC container on the destination side:
pve1$ zfs set zrep:dest-fs=rpool/zfsdisks/subvol-100-disk-1 rpool/ROOT/pve-100 pve2$ zfs set zrep:dest-fs=rpool/zfsdisks/subvol-100-disk-1 rpool/zfsdisks/subvol-100-disk-1 pve1$ vzctl stop 100 && zrep failover rpool/ROOT/pve-100 && ssh pve2 pct start 100
That’s it. We didn’t have to copy a single bit of data and we just had the regular (very minimal) downtime of a container failover like we did it before in our OpenVZ-to-OpenVZ environment.
Author: Philip Iezzi (Pipo)
Owner of Onlime GmbH - providing quality webhosting with love. All into system engineering, Linux sysadmin, security, full stack web development, mountain biking, slacklining, dancing & feeling connected to nature.