Process hiding in LXC using hidepid capabilities of procfs

August 6th, 2018 by Philip Iezzi 7 min read
cover image

Back in 2013, I wrote about Linux process hiding using hidepid capabilities of procfs. On shared webhosting servers at Onlime GmbH, I have used the hidepid=2 mount option for procfs (/proc filesystem) for improved security. Like this, a regular system user (which could potentially be an evil customer that has gained SSH access and tries to spy on other's processes) does only see his own processes, all other processes are hidden.

This is great and super simple to enable, as it is part of the official Linux kernel for quite a while now. But things start to get a little trickier when we try to set up hidepid procfs mount option inside an LXC container. Enabling the mount option on the host system will not do! Inside an LXC container, a regular system user is still able to see all processes. Before LXC 2.1 (released in Sept 2017), this was also quite doable, as we just had to create a new AppArmor profile on the host system to allow the LXC container to set the /proc mount options. But since LXC 2.1 it got super tricky. I will present both solutions below, in case you have struggled with this hard one in newer LXC versions.

Recap: procfs hidepid mount option

man proc explains the hidepid capability of procfs in depth:

PROC(5)
   Mount options
       The proc filesystem supports the following mount options:

       hidepid=n (since Linux 3.3)
              This option controls who can access the information in /proc/[pid] directories.  The  ar‐
              gument, n, is one of the following values:

              0   Everybody  may access all /proc/[pid] directories.  This is the traditional behavior,
                  and the default if this mount option is not specified.

              1   Users may not access files and subdirectories inside any /proc/[pid] directories  but
                  their  own  (the /proc/[pid] directories themselves remain visible).  Sensitive files
                  such as /proc/[pid]/cmdline and /proc/[pid]/status are now  protected  against  other
                  users.  This makes it impossible to learn whether any user is running a specific pro‐
                  gram (so long as the program doesn't otherwise reveal itself by its behavior).

              2   As for mode 1, but in addition the /proc/[pid] directories belonging to  other  users
                  become  invisible.  This means that /proc/[pid] entries can no longer be used to dis‐
                  cover the PIDs on the system.  This doesn't hide the fact that a process with a  spe‐
                  cific  PID  value  exists (it can be learned by other means, for example, by "kill -0
                  $PID"), but it hides a process's UID and GID, which could otherwise be learned by em‐
                  ploying  stat(2)  on a /proc/[pid] directory.  This greatly complicates an attacker's
                  task of gathering information about running processes (e.g., discovering whether some
                  daemon is running with elevated privileges, whether another user is running some sen‐
                  sitive program, whether other users are running any program at all, and so on).

So, hidepid=2 is what we want! On a standard Linux (we're using stable Debian Linux everywhere), you can enable this procfs mount option by a simple remount:

$ mount -o remount,hidepid=2 /proc

You may also add this mount option directly to /etc/fstab in order to make it persistent over reboots:

proc            /proc           proc    defaults,hidepid=2        0       0

A regular system user will then only see his own processes, e.g.:

testuser@web:~$ ps
  PID TTY          TIME CMD
17486 pts/0    00:00:00 bash
24806 pts/0    00:00:00 ps

This also works for other commands like pstree, top, htop,...

All good so far, but this article is about LXC containers with its super strict AppArmor policies that make things a little bit harder...

Procfs mount options in LXC before v2.1

On an host system with an LXC version before 2.1, in our case Proxmox VE up until 5.0 (LXC 2.1 was introduced in Proxmox VE 5.1 in Oct 2017), you could allow the container to set procfs mount options by creating a new AppArmor profile. I have described this in the following Proxmox VE forum post:

On the host system (running Proxmox VE or any system that offers LXC < 2.1), copy the default AppArmor profile /etc/apparmor.d/lxc/lxc-default-cgns to /etc/apparmor.d/lxc/lxc-default-cgns-with-proc-remount with :

lxc-default-cgns-with-proc-remount
# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default-cgns-with-proc-remount flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>

  # the container may never be allowed to mount devpts.  If it does, it
  # will remount the host's devpts.  We could allow it to do it with
  # the newinstance option (but, right now, we don't).
  deny mount fstype=devpts,
  mount fstype=cgroup -> /sys/fs/cgroup/**,

  # This will allow remounting /proc, e.g. to add hidepid=2 mount option
  # The hidepid option doesn't need to be included here as it's not known by AppArmor.
  mount options=(rw,nosuid,nodev,noexec,remount,relatime) -> /proc/,
}

Note: We have added the last section, see the diff:

--- lxc-default-cgns
+++ lxc-default-cgns-with-proc-remount
@@ -1,7 +1,7 @@
 # Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
 # will source all profiles under /etc/apparmor.d/lxc
 
-profile lxc-container-default-cgns flags=(attach_disconnected,mediate_deleted) {
+profile lxc-container-default-cgns-with-proc-remount flags=(attach_disconnected,mediate_deleted) {
   #include <abstractions/lxc/container-base>
 
   # the container may never be allowed to mount devpts.  If it does, it
@@ -10,4 +10,8 @@
   deny mount fstype=devpts,
   mount fstype=cgroup -> /sys/fs/cgroup/**,
   mount fstype=cgroup2 -> /sys/fs/cgroup/**,
+
+  # This will allow remounting /proc, e.g. to add hidepid=2 mount option
+  # The hidepid option doesn't need to be included here as it's not known by AppArmor.
+  mount options=(rw, nosuid, nodev, noexec, remount, relatime) -> /proc/,
 }

You can then load the new AppArmor profile:

$ apparmor_parser -r -W -T /etc/apparmor.d/lxc-containers

Add the new AppArmor profile to the container config, using the new lxc.apparmor.profile key (on a Proxmox VE host, the container config can also be found here: /etc/pve/lxc/$VMID.conf):

$ pct config $VMID | grep lxc
lxc.apparmor.profile: lxc-container-default-cgns-with-proc-remount

Then, restart the container (again, we're on a Proxmox VE host system, so we use the fancy pct command instead of native low-level lxc commands):

$ pct shutdown $VMID
$ pct start $VMID

You can now set the procfs mount option in the container's /etc/fstab, as you used to do on a plain Linux system:

proc  /proc  proc  defaults,noexec,nosuid,nodev,relatime,hidepid=2 0  0

And it all works (check this inside the container):

$ mount | grep '/proc '
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

But that will no longer work since LXC 2.1, so read on...

Procfs mount options in LXC 2.1+

As above solution with a customized AppArmor profile did no longer work since LXC 2.1, I have posted this in LXC forums:

a month later, a smart guy (thanks so much, @brauner!) came up with this workaround. I was super happy about that solution, even though in my eyes it was still a workaround and until now I didn't find a cleaner solution than that.

So, this is the final solution: Add the following lines to your LXC container configuration, e.g. /etc/pve/lxc/$VMID.conf on a Proxmox VE host system):

/etc/pve/lxc/$VMID.conf
# ...
lxc.mount.auto: 
lxc.mount.auto: sys:mixed cgroup:mixed
lxc.mount.entry: proc proc proc rw,remount,nodev,nosuid,noexec,relatime,hidepid=2 0 0
lxc.mount.entry: proc/sys proc/sys proc ro,bind,relative 0 0
lxc.mount.entry: proc/sys/net proc/sys/net proc rw,bind,relative 0 0
lxc.mount.entry: proc/sysrq-trigger proc/sysrq-trigger proc ro,bind,relative 0 0
lxc.mount.entry: /var/lib/lxcfs/proc/cpuinfo proc/cpuinfo none bind,optional 0 0
lxc.mount.entry: /var/lib/lxcfs/proc/diskstats proc/diskstats none bind,optional 0 0
lxc.mount.entry: /var/lib/lxcfs/proc/meminfo proc/meminfo none bind,optional 0 0
lxc.mount.entry: /var/lib/lxcfs/proc/stat proc/stat none bind,optional 0 0
lxc.mount.entry: /var/lib/lxcfs/proc/swaps proc/swaps none bind,optional 0 0
lxc.mount.entry: /var/lib/lxcfs/proc/uptime proc/uptime none bind,optional 0 0

You will no longer need the custom AppArmor profile I have described above. If your LXC config still contains a lxc.apparmor.profile line, remove it!

Once you restart your LXC container with those lxc.mount.* options, you don't even need to add the hidepid=2 mount option to the /etc/fstab inside your container, as this is already done in above LXC configuration.

UPDATE 2021-08-20:

In latest Proxmox VE 7.0 / LXC 4.0.x, I am still using this solution. Confirmed working. I will keep you posted if I find a simpler solution.