LXC on Debian

19 minute read , Mar 08, 2016

Install packages:

root@debian01:~# aptitude install bridge-utils vlan debconf-utils

Setup the network bridge br0:

root@debian01:~# vi /etc/network/interfaces 
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

auto br0
iface br0 inet static
  bridge_ports eth0
  bridge_fd 0
  address 192.168.100.88 
  netmask 255.255.255.0
  gateway 192.168.100.1
  dns-nameservers 192.168.100.1

And restart networking:

root@debian01:~# service networking restart

Next we install LXC:

root@debian01:~# aptitude install lxc

and setup CGroups:

root@debian01:~# mkdir /cgroup
root@debian01:~# vi /etc/fstab
...
# LXC cgroup
cgroup        /cgroup        cgroup        defaults    0    0

NOTE: default is to mount under /sys/fs/cgroup

root@debian01:~# mount -a

Now run lxc-checkconfig utility to confirm if all requirements are satisfied:

root@debian01:~# lxc-checkconfig
Kernel config /proc/config.gz not found, looking in other places...
Found kernel config file /boot/config-3.2.0-4-amd64
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

If Cgroup memory controller is not enabled and we want to have memory management in the containers we need to recompile the kernel and enable this feature:

root@debian01:~# aptitude install build-essential kernel-package linux-source-3.2 libncurses5-dev libc6-dev zlib1g-dev
root@debian01:~# cd /usr/src
root@debian01:/usr/src# tar -xjvf linux-source-3.2.tar.bz2
root@debian01:/usr/src# ln -sf linux-source-3.2 linux
root@debian01:/usr/src# ls -l
root@debian01:/usr/src# cd linux
root@debian01:/usr/src/linux# make clean
root@debian01:/usr/src/linux# make mrproper
root@debian01:/usr/src/linux# cp /boot/config-$(uname -r) .config
root@debian01:/usr/src/linux# make menuconfig

Navigate to “Load Alternate Configuration File”.
Type in “.config” (without “) and hit enter.
Go to “General Setup —>” (enter)
Go to “Control Group support —>” (enter)
Go to “Resource Counters”, enable it (space)
A subentry arises, enable: “Memory Resource Controller for Control Groups”
Another subentry, enable: “Memory Resource Controller Swap” (this is optional and not “so” important.. if you are scared off by the experimental remark: dont activate it).
Now go to Exit (three or four times) and say “Yes” when it asks you whether to save the configuration.

root@debian01:/usr/src/linux# make-kpkg clean
exec make kpkg_version=12.036+nmu3 -f /usr/share/kernel-package/ruleset/minimal.mk clean 
====== making target minimal_clean [new prereqs: ]======
This is kernel package version 12.036+nmu3.
test ! -f .config || cp -pf .config config.precious
test ! -e stamp-building || rm -f stamp-building
test ! -f Makefile || \
            make    ARCH=x86_64 distclean
make[1]: Entering directory `/usr/src/linux-source-3.2'
  CLEAN   scripts/basic
  CLEAN   scripts/kconfig
  CLEAN   include/config include/generated
  CLEAN   .config .config.old
make[1]: Leaving directory `/usr/src/linux-source-3.2'
test ! -f config.precious || mv -f config.precious .config
rm -f modules/modversions.h modules/ksyms.ver scripts/cramfs/cramfsck scripts/cramfs/mkcramfs

root@debian01:/usr/src/linux# make-kpkg --append-to-version "-cgroup-memcap" --revision 1 --us --uc --initrd kernel_image kernel_headers
root@debian01:/usr/src/linux# dpkg -i linux-image-3.2.32-cgroup-memcap_1_i386.deb
root@debian01:/usr/src/linux# mkinitramfs -o /boot/initrd.img-3.2.32-cgroup-memcap 3.2.32-cgroup-memcap
root@debian01:/usr/src/linux# update-grub2
root@debian01:/usr/src/linux# reboot

IMPORTANT: Debian Wheezy LXC template is broken and the above procedure will NOT work

Download the one from http://freedomboxblog.nl/wp-content/uploads/lxc-debian-wheezy.gz and unpack it under /usr/share/lxc/templates/ (lxc-debian-wheezy file). Then create the containers as follows:

$ sudo chmod +x /usr/share/lxc/templates/lxc-debian-wheezy
$ sudo lxc-create -n wheezy01 -t debian-wheezy
$ sudo lxc-create -n ubuntu01 -t ubuntu --fssize 8G --fstype xfs
$ sudo lxc-start -n wheezy01		# login root/root
$ sudo lxc-start -n ubuntu01		# login ubuntu/ubuntu

Configure the first container

root@debian01:~# vi /var/lib/lxc/vm0/config
...
## Limits
lxc.cgroup.cpu.shares                  = 512
lxc.cgroup.cpuset.cpus                 = 0
lxc.cgroup.memory.limit_in_bytes       = 256M
lxc.cgroup.memory.memsw.limit_in_bytes = 512M
...
# MY ADDED CONFIG
# networking
lxc.utsname = wheezy01 
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.name = eth0
lxc.network.veth.pair = vethvm1_0
lxc.network.hwaddr = 00:FF:12:34:56:78
lxc.network.ipv4 = 192.168.100.89/24

lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br1
lxc.network.name = eth1
lxc.network.veth.pair = vethvm1_1
lxc.network.hwaddr = 00:FF:FF:11:22:33
lxc.network.ipv4 = 10.10.1.10/24

root@debian01:~# lxc-start -n vm0 -d
root@debian01:~# lxc-info -n vm0
state:   RUNNING
pid:     28068

root@debian01:~# lxc-console -n vm0

To generate random MAC address that we can use in the above config:

$ printf 'DE:AD:BE:EF:%02X:%02X\n' $((RANDOM%256)) $((RANDOM%256))

Setup private network between containers

Static IP’s

root@debian01:~# aptitude install uml-utilities
root@debian01:~# tunctl -t tap0
root@debian01:~# brctl addbr br1
root@debian01:~# brctl addif br1 tap0
root@debian01:~# brctl show
bridge name	bridge id		STP enabled	interfaces
br0		8000.5254001d993e	no		eth0
br1		8000.528a275a4139	no		tap0

root@debian01:~# vi /etc/network/interfaces
...
# Containers external network bridge
auto br0
iface br0 inet static
  bridge_ports eth0
  bridge_fd 0
  address 192.168.100.88 
  netmask 255.255.255.0
  gateway 192.168.100.1
  dns-nameservers 192.168.100.1 

# Containers virtual private network bridge
auto br1
iface br1 inet static
  address 10.10.1.1
  netmask 255.255.255.0
  bridge_ports tap0
  bridge_fd 0
  pre-up /usr/sbin/tunctl -t tap0
  pre-up /sbin/ifconfig tap0 up
  post-down /sbin/ifconfig tap0 down

Another option for setting the tap interface is using the “vde2” driver. In that case the config would be:

$ sudo aptitude install vde2

Then setup in /etc/network/interfaces:

# Virtual interface
auto tap0
iface tap0 inet manual
  vde2-switch -t tap0

# Containers virtual private network bridge
auto br1
iface br1 inet static
  bridge-ports tap0
  bridge_fd 0
  address 10.10.1.1
  netmask 255.255.255.0

Restart networking:

root@debian01:~# service networking restart

and add a new interface in the container:

# MY ADDED CONFIG
## Network
lxc.utsname = wheezy01
lxc.network.type = veth
lxc.network.veth.pair = vethvm1_0
lxc.network.flags = up
lxc.network.link = br0
lxc.network.name = eth0
lxc.network.hwaddr = 00:FF:12:34:56:78
lxc.network.ipv4 = 192.168.100.89/24

lxc.network.type = veth
lxc.network.veth.pair = vethvm1_1
lxc.network.flags = up
lxc.network.link = br1
lxc.network.name = eth1
lxc.network.hwaddr = 00:FF:FF:11:22:33
lxc.network.ipv4 = 10.10.1.10/24

Then in the containers we have to set the default gateway to get Internet access:

root@wheezy01:~# ip route add default via 192.168.100.1 dev eth0
root@wheezy02:~# ip route add default via 192.168.100.1 dev eth0

in case the default route is missing.

DHCP

In case we want to have a DHCP service on the private network instead of static IP’s, we can use dnsmasq as DHCP server and DNS forwarder.

First we install dnsmasq and stop the service and disable its autostart:

root@debian01:~# aptitude install dnsmasq
root@debian01:~# vi /etc/default/dnsmasq
..
START=no

root@debian01:~# service dnsmasq stop

then we create the user the service will run under and the pid file directory:

root@debian01:~# useradd -c "LXC dnsmasq" -s /bin/false -d /var/lib/lxc -m lxc-dnsmasq
root@debian01:~# mkdir /var/run/lxc

and start the particular dnsmasq service:

root@debian01:~# dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/lxc/dnsmasq.pid --conf-file= --listen-address 10.10.1.1 --dhcp-range 10.10.1.2,10.10.1.49 --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface=br1

in this way we can have many dnsmasq processes listening on different ports.

Now we can remove the lxc.network.ipv4 = 10.10.1.10/24 from the containers config, example for wheezy02, and set its eth1 in its /etc/network/interfaces for DHCP:

auto eth1
iface eth1 inet dhcp

After restarting the container we can see in the host syslog:

Jan 31 00:38:50 debian01 dnsmasq-dhcp[17625]: DHCPREQUEST(br1) 10.10.1.6 00:ff:ff:11:22:44 
Jan 31 00:38:50 debian01 dnsmasq-dhcp[17625]: DHCPACK(br1) 10.10.1.6 00:ff:ff:11:22:44 wheezy02

the wheezy02 container asking for IP and getting 10.10.1.6 in response which can be confirmed on the container it self:

root@wheezy02:~# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 00:ff:ff:11:22:44
          inet addr:10.10.1.6  Bcast:10.10.1.255  Mask:255.255.255.0
          inet6 addr: fe80::2ff:ffff:fe11:2244/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1564 (1.5 KiB)  TX bytes:1244 (1.2 KiB)

One unwanted effect of having eth0 and eth1 in the container to DHCP is that both interfaces will set default gateway thus braking the Internet access. To avoid that we do:

# aptitude install ifmetric

and set /etc/network/interfaces as:

auto eth0
iface eth0 inet dhcp
  metric 0

auto eth1
iface eth1 inet dhcp
  metric 100
  #post-up /sbin/route del default gw 10.10.1.1

and then even though two default gateways are going to get set-up on boot time:

root@lxc02:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.3.1        0.0.0.0         UG    0      0        0 eth0
0.0.0.0         10.10.10.1      0.0.0.0         UG    100    0        0 eth1
10.0.3.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
10.10.10.0      0.0.0.0         255.255.255.0   U     0      0        0 eth1

the traffic will always have priority routing through eth0.

NOTE: Uncomment the post-up line for eth1 in case routing is still broken or DNS resolution is too slow

In case routing for the eth1 still fails to get created we set:

auto eth0
iface eth0 inet dhcp

auto eth1
iface eth1 inet dhcp
  metric 1
  post-up /sbin/route del default gw 10.10.10.1 || true
  post-up /sbin/route add -net 10.10.10.0/24 dev eth1 || true
  post-down /sbin/route del -net 10.10.10.0/24 dev eth1 || true

and reboot.

At the end, to auto-start and auto-stop dnsmasq service on the host every time the bridge br1 gets brought up/down we do:

# Containers virtual private network bridge
auto br1
iface br1 inet static
  address 10.10.1.1
  netmask 255.255.255.0
  bridge_ports tap0
  bridge_stp off
  bridge_maxwait 0
  bridge_fd 0
  pre-up /usr/sbin/tunctl -t tap0
  pre-up /sbin/ifconfig tap0 up
  post-down /sbin/ifconfig tap0 down
  post-down /bin/kill -KILL `/bin/cat /var/run/lxc/dnsmasq-br1.pid`
  post-up /usr/sbin/dnsmasq -u lxc-dnsmasq \
          --strict-order --bind-interfaces \
          --pid-file=/var/run/lxc/dnsmasq-br1.pid \
          --conf-file= \
          --listen-address 10.10.1.1 \
          --dhcp-range 10.10.1.2,10.10.1.49 \
          --dhcp-lease-max=253 --dhcp-no-override \
          --except-interface=lo --interface=br1

NOTE: To assign an IP to a linux bridge it has to have at least one interface associated hence the need of the tap0 in the bridge.

Then we disable dnsmasq from autostart on the host:

root@igor-laptop:~# update-rc.d dnsmasq disable

Add support for loop device control in the containers

In the containers config:

root@debian01:~# vi /var/lib/lxc/wheezy[01]/config
...
lxc.cgroup.devices.allow = c 10:236 rwm
lxc.cgroup.devices.allow = c 10:237 rwm
lxc.cgroup.devices.allow = b 7:* rwm

Then create the loop-control node for the containers:

root@debian01:~# mknod -m 0600 /var/lib/lxc/wheezy02/rootfs/dev/loop-control c 10 137
root@debian01:~# mknod -m 0600 /var/lib/lxc/wheezy01/rootfs/dev/loop-control c 10 137
root@debian01:~# chmod +t /var/lib/lxc/wheezy01/rootfs/dev/loop-control 
root@debian01:~# chmod +t /var/lib/lxc/wheezy02/rootfs/dev/loop-control

This is needed so when LVM2 gets installed it creates the necessary /dev/mapper/control device.

Setting up the second container

root@debian01:~# cd /var/lib/lxc/wheezy01/
root@debian01:/var/lib/lxc/wheezy01# ls -l
total 12
-rw-r--r--  1 root root 1262 Jan 29 00:45 config
-rw-r--r--  1 root root  996 Jan 29 00:41 config.bkp
drwxr-xr-x 22 root root 4096 Jan 29 00:35 rootfs
 
root@debian01:/var/lib/lxc/wheezy01# tar -czf ../template.tar.gz *
root@debian01:/var/lib/lxc/wheezy01# cd ../
root@debian01:/var/lib/lxc# ls -l
total 241760
-rw-r--r-- 1 root root 247557259 Jan 29 01:09 template.tar.gz
drwxr-xr-x 3 root root      4096 Jan 29 00:45 wheezy01

root@debian01:/var/lib/lxc# mkdir wheezy02
root@debian01:/var/lib/lxc# cd wheezy02/
root@debian01:/var/lib/lxc/wheezy02# tar -xzf ../template.tar.gz 
root@debian01:/var/lib/lxc/wheezy02# ls -l
total 12
-rw-r--r--  1 root root 1262 Jan 29 00:45 config
-rw-r--r--  1 root root  996 Jan 29 00:41 config.bkp
drwxr-xr-x 22 root root 4096 Jan 29 00:35 rootfs

and change the name, IP’s, MAC’s etc in the config file:

root@debian01:/var/lib/lxc/wheezy02# vi config

# networking
lxc.utsname = wheezy02
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.name = eth0
lxc.network.veth.pair = vethvm2_0
lxc.network.hwaddr = 00:FF:12:34:56:79
lxc.network.ipv4 = 192.168.100.90/24

lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br1
lxc.network.name = eth1
lxc.network.veth.pair = vethvm2_1
lxc.network.hwaddr = 00:FF:FF:11:22:44
lxc.network.ipv4 = 10.10.1.11/2

Check the container list:

root@debian01:/var/lib/lxc/wheezy02# lxc-list
RUNNING

FROZEN

STOPPED
  wheezy01
  wheezy02

Now we can start both containers as:

root@debian01:/var/lib/lxc/wheezy02# lxc-start -n wheezy01 -d
root@debian01:/var/lib/lxc/wheezy02# lxc-start -n wheezy02 -d
root@debian01:/var/lib/lxc/wheezy02# lxc-list
RUNNING
  wheezy01
  wheezy02

FROZEN

STOPPED

root@debian01:/var/lib/lxc/wheezy02#
root@debian01:/var/lib/lxc/wheezy02# lxc-console -n wheezy02

We can also use cloning to setup the second container:

# lxc-freeze -n wheezy01
# lxc-clone -o trusty01 -n trusty02
Created container trusty02 as copy of trusty01
# lxc-unfreeze -n wheezy01

NOTE: Actually the source container has to be stopped first for this to work

# lxc-stop -n trusty01
# lxc-clone -o trusty01 -n trusty02
Created container trusty02 as copy of trusty01
# lxc-start -n trusty01

Then edit the trusty02 container config file and replace the name, IP’s and MAC’s of the interfaces. Use:

$ printf '00:16:3e:%02x:%02x:%02x\n' $((RANDOM%256)) $((RANDOM%256))

for example to generate random MAC addresses and then start the second container:

# lxc-start -n trusty02 -d
# lxc-ls --fancy
NAME      STATE    IPV4                          IPV6  AUTOSTART
----------------------------------------------------------------
trusty01  RUNNING  192.168.16.11, 10.10.1.17     -     NO
trusty02  RUNNING  192.168.16.12, 10.10.1.15  	 -     NO
wheezy01  STOPPED  -                             -     NO

Then attach to the container and edit the /etc/network/interfaces to reflect what we have set in the config file. Then stop and start the container again.

Autostart

It is often the case that we’ll want the containers to autostart after a reboot, particularly if they are hosting services. By default, containers will not be started after a reboot, even if they were running prior to the shutdown.

To make a container autostart, we simply need to symlink its config file into the /etc/lxc/auto directory:

$ sudo ln -s /var/lib/lxc/test-container/config /etc/lxc/auto/test-container.conf

Container storage

LXC gives us awesome flexibility for storage. The container is a standard folder on our host, and can be moved easily across any Linux host. This is a new level of mobility for our server. Our server is now portable and can be simply zipped or rsynced anywhere, and be back up in seconds. Tasks like cloning, snapshotting and backups become fast and simple.

The container file system is in the default LXC container folder usually /var/lib/lxc. The individual container file systems are in the ‘containername’ folder rootfs directory.

To access host data from inside a container we can simply bind-mount a folder or drive in the host in the container’s fstab file located in the container’s directory.

/var/www var/www none bind,create=dir

This will mount the /var/www folder from the host in the container at /var/www. We can mount the same location in multiple containers to share data. We can also mount a shared folder in a separate container so it acts as a sort of shared storage outside our container. For instance:

/var/lib/lxc/mysqlvol/rootfs/var/lib/mysql var/lib/mysql

will mount mysqlvol’s /var/lib/mysql folder in mysql container so we can separate the application or container and it’s data. This storage container can also be shared across containers if required.

LXC passthrough devices

LXC can pass through devices from the host to the container. This is like using VT/d of Intel processors in virtualized systems to pass though hardware devices to the VM, but LXC does not need VTd

By default LXC prevents any such access using the devices cgroup as a filtering mechanism. The default LXC config allows certain devices to pass through. This is per container and is set in the container config file.

We can edit the individual container configuration to allow the additional devices and then restart the container. For one-off things, there’s a very convenient tool called ‘lxc-device’. With it, we can simply run:

$ sudo lxc-device add -n p1 /dev/ttyUSB0 /dev/ttyS0

Which will add (mknod) /dev/ttyS0 in the container with the same type/major/minor as /dev/ttyUSB0 and then add the matching cgroup entry allowing access from the container.

The same tool also allows moving network devices from the host to within the container.

Adding the containers to libvirt

# vi add-lxc-container.xml

<domain type="lxc">
<name>NAME</name>
<memory>332768</memory>
<os>
<type>exe</type>
<init>/sbin/init</init>
</os>
<vcpu>1</vcpu>2:8.3.13-2
<clock offset="utc"/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/lib/libvirt/libvirt_lxc</emulator>
<filesystem type="mount">
<source dir="/var/lib/lxc/NAME/rootfs"/>
<target dir="/"/>
</filesystem>
<interface type="network">
<source network="default"/>
</interface>
<console type="pty"/>
</devices>
</domain>

# sed 's/NAME/wheezy01/g' add-lxc-container.xml > wheezy01.xml
# sed 's/NAME/wheezy02/g' add-lxc-container.xml > wheezy02.xml

# virsh -c lxc:// define wheezy01.xml
# virsh -c lxc:// start wheezy01
# virsh -c lxc:// console wheezy01

# virsh -c lxc:// define wheezy02.xml
# virsh -c lxc:// start wheezy02
# virsh -c lxc:// console wheezy02

Press Ctrl+] to exit the containers console.

We can also go to virt-manager GUI and add new Connection as type LXC to interact with these containers as regular VM’s.

Create Ubuntu-14.04 (trusty) container

For Ubuntu-12.04 we need to install from backports:

root@igor-laptop:~# aptitude install -t precise-backports lxc

It has testing version of LXC 1.0.0~alpha1-0ubuntu14~ubuntu12.04.1 which supports setting default gateway in the container config so we don’t need to do that manually:

lxc.network.ipv4.gateway = 192.168.16.1

However, the /usr/share/lxc/templates/lxc-ubuntu-cloud template that comes with Wheezy looks for ubuntu-cloudimg-query which is nowhere to be found. Workaround is to comment out every line mentioning ubuntu-cloudimg-query and create the container like this:

root@debian01:~# lxc-create -t ubuntu-cloud -n trusty01 -- -r trusty -T http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-root.tar.gz

On Ubuntu:

root@igor-laptop:~# lxc-create -t ubuntu -n trusty01 -- -d ubuntu -r trusty -a amd64

OR if we want to limit disk space we can put the container into LVM we create for it, for example:

$ sudo lxc-create -t ubuntu -B lvm -n trusty01 --vgname=VG --lvname=LVNAME --fssize 8G --fstype xfs

root@igor-laptop:~# cat /var/lib/lxc/trusty01/config 
lxc.tty = 4
lxc.pts = 1024
lxc.rootfs = /var/lib/lxc/trusty01/rootfs
lxc.mount  = /var/lib/lxc/trusty01/fstab
lxc.arch = amd64
lxc.cap.drop = sys_module mac_admin
lxc.pivotdir = lxc_putold

# uncomment the next line to run the container unconfined:
#lxc.aa_profile = unconfined

lxc.cgroup.devices.deny = a
# Allow any mknod (but not using the node)
lxc.cgroup.devices.allow = c *:* m
lxc.cgroup.devices.allow = b *:* m
# /dev/null and zero
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
# consoles
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
#lxc.cgroup.devices.allow = c 4:0 rwm
#lxc.cgroup.devices.allow = c 4:1 rwm
# /dev/{,u}random
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
# rtc
lxc.cgroup.devices.allow = c 254:0 rwm
#fuse
lxc.cgroup.devices.allow = c 10:229 rwm
#tun
lxc.cgroup.devices.allow = c 10:200 rwm
#full
lxc.cgroup.devices.allow = c 1:7 rwm
#hpet
lxc.cgroup.devices.allow = c 10:228 rwm
#kvm
lxc.cgroup.devices.allow = c 10:232 rwm
# loop devices
lxc.cgroup.devices.allow = c 10:236 rwm
lxc.cgroup.devices.allow = c 10:237 rwm
lxc.cgroup.devices.allow = b 7:* rwm
# DRBD devices
lxc.cgroup.devices.allow = b 147:* rwm

# mounts point
lxc.mount.entry=proc /var/lib/lxc/trusty01/rootfs/proc proc nodev,noexec,nosuid 0 0
lxc.mount.entry=devpts /var/lib/lxc/trusty01/rootfs/dev/pts devpts defaults 0 0
lxc.mount.entry=sysfs /var/lib/lxc/trusty01/rootfs/sys sysfs defaults  0 0

# limits
lxc.cgroup.cpu.shares = 1024
lxc.cgroup.cpuset.cpus = 0
lxc.cgroup.memory.limit_in_bytes = 512M
lxc.cgroup.memory.memsw.limit_in_bytes = 2G

# autostart
#lxc.start.auto = 1
#lxc.start.delay = 5
#lxc.start.order = 0

# networking
lxc.utsname = trusty01 

lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = lxcbr0 
lxc.network.name = eth0
lxc.network.veth.pair = vethlxc1_0
lxc.network.hwaddr = 00:16:3e:38:e1:31 
lxc.network.ipv4 = 192.168.16.11/24

lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = lxcbr1 
lxc.network.name = eth1
lxc.network.veth.pair = vethlxc1_1
lxc.network.hwaddr = 00:16:3e:38:a7:75
#lxc.network.ipv4 = 10.10.1.11/24

NOTE: To be able to use the Memory cgroups limit we need to enable this in the kernel on startup by adding:

GRUB_CMDLINE_LINUX_DEFAULT="cgroup_enable=memory swapaccount=1"

to /etc/default/grub file OR add the options if the line already exists.

root@igor-laptop:~# cat /etc/network/interfaces
...
#
# LXC
#
# Containers public network bridge
auto lxcbr0
iface lxcbr0 inet static
  bridge_ports none 
  bridge_fd 0
  bridge_stp off
  address 192.168.16.1 
  netmask 255.255.255.0

# Containers private network bridge
auto lxcbr1
iface lxcbr1 inet static
  address 10.10.1.1
  netmask 255.255.255.0
  bridge_ports lxctap0 
  bridge_fd 0
  pre-up /usr/sbin/tunctl -t lxctap0
  pre-up /sbin/ifconfig lxctap0 up
  post-down /sbin/ifconfig lxctap0 down
  post-up dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces \
  --pid-file=/var/run/lxc/lxcbr1.pid --conf-file= \
  --except-interface lo --listen-address 10.10.1.1 \
  --dhcp-range 10.10.1.10,10.10.1.20 \
  --dhcp-leasefile=/var/run/lxc/lxcbr1.leases \
  --dhcp-lease-max=11 --dhcp-no-override

Upon startup if we see the following error:

root@igor-laptop:~# lxc-start -n trusty01 -l debug
lxc-start: Device or resource busy - failed to mount 'proc' on '/usr/lib/lxc/root//proc'
lxc-start: failed to setup the mount entries for 'trusty01'
lxc-start: failed to setup the container
lxc-start: invalid sequence number 1. expected 2
lxc-start: failed to spawn 'trusty01'
lxc-start: Device or resource busy - failed to remove cgroup '/sys/fs/cgroup//lxc/trusty01'

and the trusty01 cgroup is left behind, we can manually remove it like this:

root@igor-laptop:~# find /sys/fs/cgroup/lxc/trusty01/ -type d | tac | xargs rmdir

Finally we need to apply the following firewall rules on the host to filter traffic and protect from mixing with other virtual networks on the host:

root@igor-laptop:~# iptables -A INPUT -i lxcbr0 -p udp -m udp --dport 53 -j ACCEPT
root@igor-laptop:~# iptables -A INPUT -i lxcbr0 -p tcp -m tcp --dport 53 -j ACCEPT
root@igor-laptop:~# iptables -A INPUT -i lxcbr0 -p udp -m udp --dport 67 -j ACCEPT
root@igor-laptop:~# iptables -A INPUT -i lxcbr0 -p tcp -m tcp --dport 67 -j ACCEPT
root@igor-laptop:~# iptables -A INPUT -i lxcbr1 -p udp -m udp --dport 67 -j ACCEPT
root@igor-laptop:~# iptables -A INPUT -i lxcbr1 -p tcp -m tcp --dport 67 -j ACCEPT
root@igor-laptop:~# iptables -A FORWARD -d 192.168.16.0/24 -i eth0 -o lxcbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
root@igor-laptop:~# iptables -A FORWARD -s 192.168.16.0/24 -i lxcbr0 -o eth0 -j ACCEPT
root@igor-laptop:~# iptables -A FORWARD -i lxcbr0 -o lxcbr0 -j ACCEPT
root@igor-laptop:~# iptables -A FORWARD -o lxcbr0 -j REJECT --reject-with icmp-port-unreachable
root@igor-laptop:~# iptables -A FORWARD -i lxcbr0 -j REJECT --reject-with icmp-port-unreachable
# next one not needed
root@igor-laptop:~# iptables -t nat -A POSTROUTING -o eth0 -s 192.168.16.0/24 ! -d 192.168.16.0/24 -j MASQUERADE

Now start the container and add the lxcbr0 as default gateway:

root@igor-laptop:~# lxc-start -n trusty01 -d

root@igor-laptop:~# lxc-ls --fancy
NAME      STATE    IPV4           IPV6  AUTOSTART
-------------------------------------------------
trusty01  RUNNING  192.168.16.11  -     NO
wheezy01  STOPPED  -              -     NO

root@igor-laptop:~# lxc-attach -n trusty01

root@trusty01:~# route add default gw 192.168.16.1 dev eth0
 
root@trusty01:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.16.1    0.0.0.0         UG    0      0        0 eth0
192.168.16.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0

root@trusty01:~# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:16:3e:38:e1:31
          inet6 addr: fe80::216:3eff:fe38:e131/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3431 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2017 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:5104305 (5.1 MB)  TX bytes:152183 (152.1 KB)

eth1      Link encap:Ethernet  HWaddr 00:16:3e:38:a7:75
          inet addr:10.10.1.17  Bcast:10.10.1.255  Mask:255.255.255.0
          inet6 addr: fe80::216:3eff:fe38:a775/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:36 errors:0 dropped:0 overruns:0 frame:0
          TX packets:32 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:7965 (7.9 KB)  TX bytes:2900 (2.9 KB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:1 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:112 (112.0 B)  TX bytes:112 (112.0 B)

From now on we can do everything we do with any other server install, run apt-get etc.

Connecting LXC on multiple hosts

GRE tunnel

A GRE tunnel is a simple point to point IP tunnel connecting 2 public IPs and networks. But it’s not encrypted. For private networks across the public Internet, encryption is often standard but if your traffic does not have high security requirements a GRE tunnel is a simple just works solution.

Connecting 2 LXC hosts with a GRE tunnel will enable your LXC containers to access each other. LXC containers will be using their default NAT bridge lxcbr0.

We are going to change the subnet provided by lxcbr0 on one side which is normally 10.0.3.0/24 to 10.0.4.0/24 so there is no clash of IPs on either side and we can route both ways.

So here is the network map. We are going to call our GRE tunnel ‘neta’ on Host A and ‘netb’ on Host B (You can call this anything you want)

Host A has public IP 1.2.3.4 Host B has public IP 2.3.4.5 Containers in Host A are on subnet 10.0.4.0/24 via default lxcbr0 nat bridge Containers in Host B are on subnet 10.0.3.0/24 via default lxcbr0 nat bridge

To change the subnet edit the /etc/init.d/lxc-net script. Change the subnet entries from 10.0.3.0/24 to 10.0.4.0/24.

Create the GRE tunnel on Host A and B:

sudo ip tunnel add neta mode gre remote 2.3.4.5 local 1.2.3.4 ttl 255

sudo ip tunnel add netb mode gre remote 1.2.3.4 local 2.3.4.5 ttl 255

On Host A:

sudo ip link set neta up
sudo ip addr add 10.0.4.254 neta
sudo ip route add 10.0.3.0/24 dev neta

On Host B:

sudo ip link set netb up
sudo ip addr add 10.0.3.254 netb
sudo ip route add 10.0.4.0/24 dev netb

The IP address we added to the tunnel on each side 10.0.3/4.254 is being used as the gateway to reach either side. This is a random link IP, you can use anything 10.0.0.1/2 for instance. To remove the tunnel run:

sudo ip link set netb down
sudo ip tunnel del netb

Share on

Twitter Facebook Google+ LinkedIn

Igor Cicimov