Building and Managing Thick Jails with VNET on FreeBSD

🔗Introduction

Jails have been part of FreeBSD since 2000, and they remain one of the most practical tools in the system. The concept is straightforward: isolate a process tree and its filesystem from the rest of the host. What makes jails interesting to dig into is how much you can do once you pair them with VNET, which gives each jail its own independent network stack.

This guide builds a working thick jail with VNET entirely by hand, without iocage, ezjail, or any other management tool. Every configuration file and command is explained. The aim is to understand what is actually happening, not just have something running.

Useful references for going deeper:

FreeBSD Jails using VNETs by rderik
FreeBSD Mastery: Jails by Michael W. Lucas and Allan Jude
FreeBSD Handbook: Jails chapter

🔗Why Thick Jails with VNET and No Tooling

A thick jail contains a complete FreeBSD userland inside its own directory. It behaves like a standalone installation and can be configured, updated, and patched independently of the host.

VNET (Virtual Network Stack) assigns the jail its own TCP/IP stack. Without VNET, a jail shares the host's network stack and is limited to IP aliases. With VNET, each jail has its own interfaces, routing table, and can run its own firewall or DHCP client.

Skipping management tooling means you configure everything through jail.conf, rc.conf, and standard network commands. Nothing is hidden. Every setting is explicit, and you decide how the jail behaves.

🔗Advantages

Full control: Every configuration is visible and adjustable. Nothing is hidden behind tooling defaults.
True network isolation: Each jail gets its own TCP/IP stack, preventing address conflicts and enabling realistic network simulation.
Security: Process, filesystem, and network isolation reduce the attack surface for each service.
Low overhead: Jails share the host kernel. There is no hardware emulation or per-instance kernel memory overhead.
Predictable behavior: No tooling updates can change your configuration unexpectedly.

🔗Disadvantages

Manual setup: Initial provisioning takes more time and requires a solid understanding of FreeBSD networking and storage.
No tooling shortcuts: You maintain all configurations yourself. Good documentation habits matter.
Version drift risk: Without automation, different jails can end up on different patch levels if updates are not tracked consistently.
More disk space: Thick jails duplicate the userland for each jail. Thin jails, which mount a shared read-only base, are more storage-efficient.

🔗Use Cases

Network simulation and topology testing without extra hardware
Running multiple independent services on one physical server
Isolated development and QA environments
Lightweight service hosting with per-service network control
Security research with contained, network-isolated environments

🔗When to Avoid This Approach

Your workflow requires rapid, automated provisioning. In that case, iocage or bastille will save time.
Your applications require Linux-specific system calls. Jails run the FreeBSD userland; Linux compatibility requires the Linux ABI layer and is not suitable for all applications.
Your team is small and cannot dedicate time to maintaining manual configurations.
You need live migration between hosts. FreeBSD jails do not have built-in live migration.

🔗Available Tooling (for Reference)

This guide uses no tooling, but the following options exist if you prefer automation later:

iocage: ZFS-centric jail manager with a Python-based CLI.
bastille: Lightweight template-driven jail framework.
appjail: Template and service-oriented jail deployment.
ezjail: Older script-based tool, largely superseded by the above.

🔗Prerequisites

Before starting, confirm the following on your host:

FreeBSD 8.0 or later with VNET support compiled in. The default GENERIC kernel includes it. Confirm with:

sysctl kern.features.vimage

Expected output: kern.features.vimage: 1

If the output is 0 or the sysctl does not exist, you need a kernel built with options VIMAGE.

You know your host's primary network interface name. Find it with:

ifconfig -l

This guide uses em0 as the example. Replace it with your actual interface.

🔗Step 1: Prepare the Jail Root Directory

Create the directory that will hold the jail's complete userland:

mkdir -p /usr/local/jails/myjail

🔗Step 2: Obtain the FreeBSD Distribution Sets

You need base.txz for a working jail. lib32.txz adds 32-bit library compatibility (optional on amd64). src.txz adds the FreeBSD source tree (optional).

Place the files in /usr/freebsd-dist/ before extracting. Match the release version to your host:

uname -r

Method A: Download from official mirrors

REL=$(uname -r | cut -d- -f1,2)   # e.g. 12.2-RELEASE
ARCH=$(uname -m)                   # e.g. amd64

mkdir -p /usr/freebsd-dist
cd /usr/freebsd-dist

fetch https://download.freebsd.org/releases/${ARCH}/${ARCH}/${REL}/base.txz
fetch https://download.freebsd.org/releases/${ARCH}/${ARCH}/${REL}/lib32.txz  # amd64 only, optional

Method B: Copy from installer media

If you have a FreeBSD installer ISO mounted or a physical disc:

mkdir -p /mnt
mount -t cd9660 /dev/cd0 /mnt

mkdir -p /usr/freebsd-dist
cp /mnt/usr/freebsd-dist/base.txz /usr/freebsd-dist/
cp /mnt/usr/freebsd-dist/lib32.txz /usr/freebsd-dist/   # optional

umount /mnt

Method C: Use bsdinstall to fetch sets interactively

mkdir -p /usr/freebsd-dist
bsdinstall distfetch

Follow the prompts to select a mirror and download the sets you need.

🔗Step 3: Extract the Userland into the Jail

cd /usr/local/jails/myjail
tar -xpf /usr/freebsd-dist/base.txz
tar -xpf /usr/freebsd-dist/lib32.txz   # optional

The -p flag preserves file permissions. Do not omit it.

🔗Step 4: Enable VNET Kernel Modules

Add the following to /boot/loader.conf to load the required modules at boot:

if_bridge_load="YES"
if_epair_load="YES"

Apply the changes now without rebooting:

kldload if_bridge
kldload if_epair

🔗Step 5: Create and Configure the Bridge Interface

Create a persistent bridge interface on the host and attach your physical NIC to it:

ifconfig bridge0 create
ifconfig bridge0 addm em0 up

To make this persistent across reboots, add to /etc/rc.conf:

cloned_interfaces="bridge0"
ifconfig_bridge0="addm em0 up"

The bridge allows the jail's virtual interface to communicate on the same network segment as the host.

🔗Step 6: Create the epair Interface

An epair creates two linked virtual Ethernet interfaces (epair0a and epair0b). One end stays on the host and is added to the bridge; the other end goes into the jail.

ifconfig epair0 create
ifconfig epair0a up
ifconfig bridge0 addm epair0a

For multiple jails, create additional epairs: epair1, epair2, and so on. Each pair provides one dedicated virtual link per jail.

🔗Step 7: Configure the Jail in /etc/jail.conf

Add the following block to /etc/jail.conf. Create the file if it does not exist:

myjail {
    host.hostname = "myjail.local";
    path = "/usr/local/jails/myjail";
    persist;
    vnet;
    vnet.interface = "epair0b";
    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
    exec.clean;
    allow.raw_sockets;
}

Key parameters:

vnet: enables the per-jail network stack
vnet.interface = "epair0b": moves epair0b into the jail's network namespace when the jail starts
persist: allows the jail to run without any processes, useful during setup
allow.raw_sockets: permits ping and other tools that use raw sockets inside the jail

🔗Step 8: Configure the Jail's Base System

Before starting the jail, write a minimal /etc/rc.conf and /etc/resolv.conf inside the jail directory:

echo 'ifconfig_epair0b="inet 192.168.1.50 netmask 255.255.255.0"' \
    >> /usr/local/jails/myjail/etc/rc.conf

echo 'defaultrouter="192.168.1.1"' \
    >> /usr/local/jails/myjail/etc/rc.conf

echo 'nameserver 1.1.1.1' \
    > /usr/local/jails/myjail/etc/resolv.conf

Adjust the IP address, subnet mask, gateway, and nameserver to match your network.

🔗Step 9: Start the Jail

service jail onestart myjail

To start all jails defined in /etc/jail.conf at boot, add to /etc/rc.conf:

jail_enable="YES"

Then use:

service jail start

🔗Step 10: Verify Networking

From the host, run a connectivity test inside the jail:

jexec myjail ping -c 3 1.1.1.1

If ping succeeds, the jail has a working network stack and outbound routing. If it fails, check:

That epair0a is a member of bridge0: ifconfig bridge0
That the jail's IP and default gateway are correct: jexec myjail ifconfig and jexec myjail netstat -rn
That the host has IP forwarding enabled if routing between subnets: sysctl net.inet.ip.forwarding

🔗Step 11: Access the Jail Shell

jexec myjail /bin/sh

From inside the jail you can install packages with pkg, run services, and configure the environment as you would on any FreeBSD system.

🔗Managing Multiple Jails

For each additional jail, repeat the epair creation and jail.conf block with incremented names and unique IPs:

ifconfig epair1 create
ifconfig epair1a up
ifconfig bridge0 addm epair1a

myjail2 {
    host.hostname = "myjail2.local";
    path = "/usr/local/jails/myjail2";
    persist;
    vnet;
    vnet.interface = "epair1b";
    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
    exec.clean;
    allow.raw_sockets;
}

🔗Network Diagram

                    +-------------------+
                    |     LAN / WAN     |
                    +-------------------+
                             |
                           em0
                             |
+--------------------------------------------------------------+
|                        FreeBSD Host                          |
|                                                              |
|   +---------------------------------------------------+      |
|   |  bridge0  (members: em0, epair0a, epair1a, epair2a) |    |
|   +---------------------------------------------------+      |
|         |            |            |                          |
|      epair0a      epair1a      epair2a                       |
+---------|------------|------------|-------------------------+
          |            |            |
       epair0b      epair1b      epair2b
          |            |            |
    +----------+ +----------+ +----------+
    |  Jail A  | |  Jail B  | |  Jail C  |
    |  VNET    | |  VNET    | |  VNET    |
    +----------+ +----------+ +----------+

Each jail owns one side of an epair. The host side (epairNa) is a member of the bridge. The bridge connects all jail interfaces and the physical NIC at Layer 2.

🔗Useful Jail Commands

Task	Command
List running jails	`jls`
Open a shell in a jail	`jexec myjail /bin/sh`
Stop a jail	`service jail stop myjail`
Run a single command in a jail	`jexec myjail pkg update`
Show jail network interfaces	`jexec myjail ifconfig`

🔗Resource Efficiency

Jails share the host kernel, so there is no per-jail kernel memory overhead. With ZFS, you can snapshot and clone jail datasets for fast provisioning and easy rollback:

zfs snapshot zroot/jails/myjail@before-upgrade
zfs rollback zroot/jails/myjail@before-upgrade

Thick jails use more disk space than thin jails because each one carries a full userland copy. The trade-off is configuration independence: updating one jail's base does not affect any other jail.

🔗Scalability

FreeBSD can run dozens to hundreds of jails on a single host with low overhead, provided you monitor CPU, memory, and network load. VNET jails scale well because each jail has its own network presence and can be moved or reconfigured independently. Combining ZFS datasets with scripted epair creation makes it straightforward to add jails programmatically.

🔗Community Support

FreeBSD's jail ecosystem has decades of history. Support is available through:

FreeBSD Forums: General troubleshooting and discussion
FreeBSD mailing lists: In-depth technical exchanges (freebsd-jail@)
IRC and Matrix: Real-time help from experienced admins (#freebsd on Libera.Chat)
man pages: jail(8), epair(4), bridge(4), and vnet(9) are thorough references

🔗Closing Thoughts

What I find satisfying about this setup is that nothing is hidden. Every interface, every address, every routing decision is something you put there deliberately. Once the jail is running and ping comes back clean, it is genuinely yours: a complete isolated system sitting inside your host, connected to the network on its own terms. From there, the interesting part begins.