Debug School

Cover image for System Jails
Suyash Sambhare
Suyash Sambhare

Posted on

System Jails

The jail utility can build new jails, as well as modify or remove existing ones. It can also output a list of configured jails and their parameters. A jail is specified using command-line options or the jail.conf file.

About

The jail utility debuted in FreeBSD 4.0. FreeBSD 8.0 added hierarchical/extensible jails. The configuration file was introduced with FreeBSD 9.1. Poul-Henning Kamp wrote the jail feature for R&D Associates, who then contributed it to FreeBSD. Robert Watson produced the updated documentation, discovered a few problems, introduced some new functionality, and tidied up the userland jail environment. Bjoern A. Zeeb implemented multi-IP jail support for IPv4 and IPv6, building on a patch previously created by Pawel Jakub Dawidek for IPv4. James Gritton contributed the extensible jail parameters, hierarchical jails, and configuration files.

Command details:

SYNOPSIS
 From Configuration File
  jail [-cm] [-dqv] [-f conf_file] [-p limit] [jail]
  jail [-r] [-qv] [-f conf_file] [-p limit] [* | jail ...]

 Without Configuration File
  jail [-cm] [-dhilqv] [-J jid_file] [-u username] [-U username]
  param=value ... [command=command ...]
  jail [-rR] [-qv] [* | jail ...]

 Show Parameters
  jail [-f conf_file] -e separator

 Backward Compatibility
  jail [-dhilqv] [-J jid_file] [-u username] [-U username] [-n jail name]
  [-s secure level] path hostname ip[,...] command ...
Enter fullscreen mode Exit fullscreen mode

Description

The jail utility creates new jails, modifies, or removes existing jails. It can also print a list of configured jails and their parameters. A jail is specified via parameters on the command line, or in the jail.conf file.

It could be a good idea to include an address alias flag so that daemons listening on all IPs do not bind to that address, allowing for the creation of a safe host environment in which host daemons do not impose on services provided by jails. Currently, the simplest solution is to limit the services offered on the host, to those provided by inetd, which is easily adjustable.

When controlling viewable directories within the jail, exercise extreme caution. For example, if an imjailed process's current working directory is set to a directory outside of the jail's chroot, the process may be able to access file space outside of the jail. It is advised that folders be copied rather than relocated from a jail. Furthermore, there are various ways for an unprivileged user outside the jail to collaborate with a privileged user inside the jail and thereby get elevated access in the host environment. Many of these attacks can be prevented by ensuring that the jail root is not available to unprivileged users in the host environment. Regardless, in general, untrustworthy users with privileged access to a jail should not be given access to the host environment.

At least one of the options -c, -e, -m or -r must be specified. These options are used alone or in combination to describe the operation to perform:

-c Create a new jail. The jail jid and name parameters, if specified on the command line, must not refer to an existing jail.
-e separator Exhibit a list of all configured non-wildcard jails and their parameters. No jail creation, modification or removal performed if this option is used. The separator string is used to separate parameters. Use jls utility to list running jails.
-m Modify an existing jail. One of the jid or name parameters must exist and refer to an existing jail. Some parameters may not be changed on a running jail.
-r Remove the jail specified by jid or name. All jailed processes are killed, and all jails that are children of this jail are also removed.
-rc Restart an existing jail. The jail is first removed and then re-created, as if jail -r and jail -c were run in succession.
-cm Create a jail if it does not exist or modify the jail if it does exist.
-mr Modify an existing jail. The jail may be restarted if necessary to modify parameters than could not otherwise be changed.
-cmr Create a jail if it does not exist, or modify, and restart, the jail if it does exist.
-d Allow making changes to a dying jail, equivalent to the allow.dying parameter.
-f conf_file Use configuration file conf_file instead of the default /etc/jail.conf.
-h Resolve the host.hostname parameter and add all IP addresses returned by the resolver to the list of addresses for this jail. This is equivalent to the ip_hostname parameter.
-i Output the jail identifier of the newly created jails. This implies the -q option.
-J jid_file Write a jid_file file, containing the parameters used to start the jail.
-l Run commands in a clean environment. This is deprecated and is equivalent to the exec.clean parameter.
-n jailname Set the jail's name. This is deprecated and is equivalent to the name parameter.
-p limit Limit the number of commands from exec.* that can run simultaneously.
-q Suppress the message printed whenever a jail is created, modified, or removed. Only error messages will be printed.
-R A variation of the -r option that removes an existing jail without using the configuration file. No removal-related parameters for this jail will be used -- the jail will simply be removed.
-s securelevel Set the kern.securelevel MIB entry to the specified value inside the newly created jail. This is deprecated and is equivalent to the securelevel parameter.
-u username The username from host environment as whom jailed commands should run. This is deprecated and is equivalent to the exec.jail_user and exec.system_jail_user parameters.
-U username The username from the jailed environment as whom jailed commands should run. This is deprecated and is equivalent to the exec.jail_user parameter.
-v Print a message on every operation, such as running commands and mounting filesystems.

All jails listed in the jail.conf file will be subject to the operation, apart from removal, if no arguments are provided after the options. Only the designated jail will be affected by a single argument for the jail name. Running jails that are not listed in the jail.conf file can also be eliminated with the -r and -R options, either by name or jid.

This is the most reliable method for -r to delete all jails: an argument of * is a wildcard that will work on any jail, regardless of whether it appears in jail.conf. A partial matching wildcard specification may be supplied if hierarchical jails are present. For instance, jails with names like foo.bar and foo.bar.baz would fall under the foo.* parameter.

In addition, a jail can be explicitly defined via parameters in the name=value format on the command line, excluding the contents of jail.conf. Path, hostname, IP, and command are the other four fixed parameters that may be present in the command line for backward compatibility without names.

Jail Parameters

The format of parameters on the command line or in the jail.conf file is typically name=value. Certain parameters, such persist or nopersist, are Boolean in nature, meaning they can be set just by their name, with or without a no prefix. The values true and false can also be assigned to them. Other parameters can have multiple values; these can be entered in the configuration file using += or as a list separated by commas.

The jail utility distinguishes between two argument kinds. When a jail is created, the kernel receives the true jail parameters, which are viewable with jls and may typically be modified with jail -m. Furthermore, there exist pseudo-parameters that are exclusive to the jail system.

Jails have a set of basic settings, and kernel modules can provide additional jail parameters. The current set of accessible options can be accessed with sysctl -d security.jail.param. Any parameters that are not set will be assigned default values, frequently based on the existing environment.

Kernel modules may define their own parameters, which are only available when the module is loaded. These are often grouped under a parameter named after the module, with values such as inherit to give the jail full usage of the module, new to encapsulate the jail in a module-specific form, and disable to make the module unavailable to it. There may also be additional parameters that specify jail behaviour within the module.

There are pseudo-parameters that are not supplied to the kernel but are utilised by jail to configure the jail environment, often by running specific commands when jails are formed or removed. The exec.* command parameters are sh commands that can be executed in either the system or jail environment. They can be provided multiple values, which will execute the specified commands in order. All commands must be successful (return a zero-exit status), or the jail will not be formed or deleted, as necessary.

Jail

How it Works

Jails are often configured in one of two ways: to restrict a certain application (potentially running with privileges) or to construct a virtual system image that runs a range of daemons and services. In both circumstances, a full FreeBSD file system installation is required to provide the appropriate command line tools, daemons, libraries, application configuration files, and so on. However, replacing the boot process requires some additional work in a virtual server configuration. This manual page describes the configuration steps required to support either of these stages, though they may need to be tweaked depending on local circumstances.

Setting up a Jail Directory Tree

To set up a jail directory tree containing an entire FreeBSD distribution, the following sh command script can be used:

  D=/here/is/the/jail
  cd /usr/src
  mkdir -p $D
  make world DESTDIR=$D
  make distribution DESTDIR=$D
Enter fullscreen mode Exit fullscreen mode

In many circumstances, this example would put many more people in jail than necessary. In the other extreme, a jail may simply contain one file: the executable to be run within the jail.
Experiment options to start with a fat jail and remove items until it stops working than to start with a thin jail and add things until it works.

Set Up Jail

To create the jail directory tree, follow the steps outlined in Setting Up a Jail Directory Tree. For the purposes of this example, we will suppose you created it in '/data/jail/testjail' for a jail called testjail. Substitute your own directory, IP address, and host name where applicable.

Set up Host Environment

First, configure the environment of the real machine to be jail-friendly. We will refer to the imjailed virtual machine as the jail environment and the parent computer as the host environment to maintain consistency. Disabling IP services on the host system that listen on all local IP addresses for a service is one of the first things to perform because jails are established using IP aliases. Service requests may be delivered to jail IP addresses if the jail did not bind the port if a network service that binds all available IP addresses rather than just certain IP addresses is present in the host environment. This entails configuring inetd to only listen on the relevant IP address, among other things. In /etc/rc.conf, add the following in host environment:

  sendmail_enable=`NO`
  inetd_flags=`-wW -a 10.12.113.47`
  rpcbind_enable=`NO`
Enter fullscreen mode Exit fullscreen mode

In this example, the host system's native IP address is 10.12.113.47. Daemons that operate on inetd can be easily set to use just the specified host IP address. Other daemons must be explicitly configured; for some, this is done via rc.conf flags entries; for others, it is necessary to alter per-application configuration files or recompile the application. The following widely deployed services' configuration files must be updated to restrict the application to listening to a specified IP address. In addition, a number of services must be recompiled before they may run in the host environment. This comprises most applications that provide rpc services.
In general, programs that do not allow you to specify which IP address to bind should not be launched in the host environment unless they also send service requests to jail IP addresses. Attempting to serve NFS from the host environment can also be confusing, since certain NFS services are hosted directly from the kernel, making it difficult to adjust to use only specific IPs. Any third-party network software running in the host environment should also be examined and configured to ensure that it does not bind all IP addresses, causing such services to seem to be supplied by the jail environments.

Once these daemons have been stopped or corrected in the host environment, reboot to ensure that all daemons are in a known condition, reducing the possibility of confusion later.

Configure Jail

Starting any jail for the first time without establishing the network interface allows you to tidy it up and set up accounts. As with any system (virtual or not), you will need to configure a root password, time zone, and so on. Some of these actions are only necessary if you wish to run a full virtual server within the jail; others are applicable to both limiting a specific application and running a virtual server.

Start a shell in the jail: jail -c path=/data/jail/testjail mount.devfs host.hostname=testhostname ip4.addr=192.0.2.100 command=/bin/sh

Assuming no errors, you will end up with a shell prompt within the jail. You can now run bsdconfig and do the post-install configuration to set various configuration options, or perform these actions manually by editing /etc/rc.conf, etc.

  • Configure /etc/resolv.conf so that name resolution within the jail will work correctly.
  • Run newaliase to quell sendmail warnings.
  • Set a root password, different from the real host system.
  • Set the time zone.
  • Add accounts for users in the jail environment.
  • Install any packages the environment requires.

You may also wish to execute any package-specific settings (web servers, SSH servers, etc.), modify /etc/syslog.conf to log as desired, and so on. If you are not using a virtual server, you may want to configure syslogd in the host environment to listen on the syslog socket in the jail environment; in this case, the syslog socket would be in /data/jail/testjail/var/run/log.
Exit the shell, and the jail will be shut down.

Start Jail

You are now ready to restart the jail and bring up the environment with all its daemons and other programs. Create an entry for the jail in /etc/jail.conf

testjail {
 path = /tmp/jail/testjail;
 mount.devfs;
 host.hostname = testhostname;
 ip4.addr = 192.0.2.100;
 interface = em0;
 exec.start = `/bin/sh /etc/rc`;
 exec.stop = `/bin/sh /etc/rc.shutdown jail`;
}
Enter fullscreen mode Exit fullscreen mode

To begin a virtual server environment, execute /etc/rc to launch various daemons and services, then /etc/rc.shutdown to terminate them when the jail is removed. If you are only running one application in the jail, replace the command used to start it with /bin/sh /etc/rc; there may be a script available to cleanly shut down the application, or it may be sufficient to go without a stop command and have jail send SIGTERM to the application.

Start the jail by running: jail -c testjail

A few warnings may be generated; however, everything should work perfectly. Using PS, you should be able to observe inetd, syslogd, and other processes operating within the jail, along with the 'J' flag next to imjailed processes. jls displays an active list of jails. If sshd is enabled in the jail, you should be able to ssh to the imjailed environment's hostname or IP address and log in with the previously generated accounts.

It is possible to have jails started at boot time. Please refer to the jail_* variables in rc.conf for more information.

Manage Jail

Normal machine shutdown commands, such as halt, reboot, and shutdown, cannot be used successfully within the jail. To kill all processes from within a jail, you may use one of the following commands, depending on what you want to accomplish:

kill -TERM -1
kill -KILL -1

This will send the SIGTERM or SIGKILL signals to all processes in the jail -- be careful not to run this from the host environment! Once all the jail's processes have died, unless the jail was created with the persist parameter, the jail will be removed. Depending on the intended use of the jail, you may also want to run /etc/rc.shutdown from within the jail.

To shut down the jail from the outside, simply remove it with: jail -r which will run any commands specified by exec.stop, and then send SIGTERM and eventually SIGKILL to any remaining jailed processes.

The /proc/pid/status file contains, as its last field, the name of the jail in which the process runs, or - to indicate that the process is not running within a jail. The PS command also shows a `J' flag for processes in a jail.

You can also list/kill processes based on their jail ID. To show processes and their jail ID, use the following command: ps ax -o pid,jid,args

To show and then kill processes in jail number 3 use the following commands: pgrep -lfj 3 & pkill -j 3 or killall -j 3

Jails and File Systems

It is not feasible to mount any file system within a jail unless the file system is declared as jail-friendly, which the jails allow. Mount parameter is set, and the jail's enforce_statfs parameter is less than two.

Multiple jails that use the same file system can impact one another. For example, a user in one jail can exhaust the file system, leaving no room for processes in the other. Attempting to block this with quota would also fail, because file system quotas are not aware of jails and only look at user and group IDs. This means that users with the same ID in two jails share a single file system quota. To make this work, each jail would require its own file system.

Hierarchical Jails

By setting the children.max option in a jail, processes within it may be able to build their own jails. These kid jails are organised in a hierarchy, with jails only able to see and/or change the jails they produced (or their children). Each jail includes a read-only parent parameter that contains the jid of the jail that generated it; a jid of 0 indicates that the jail is a child of the current jail (or a top-level jail if the current process is not jailed).

Jailed processes are not permitted to provide more rights than they have been granted; for example, if a jail is formed with allow.nomount, it cannot construct a jail with allow.mount enabled. Similarly, such constraints as ip4.address and securelevel may not be by- passed in child jails.

If a child jail has its own children, it may also establish additional child jails. Remember that the max parameter is set to zero by default. Their dad and all ancestors can see these jails and change them.

This structure is reflected in jail names, where a complete name is a MIB-style string divided into dots. For instance, if a process in the base system establishes a jail called foo, and another process within that jail builds a second jail called bar, then in the base system, the second jail will be recognised as foo.bar (although it is only recognised as bar to any processes inside jail foo). Conversely, jids are singular entities that exist in a single space, and every jail needs its own jid.

Like the names, a child jail's path appears relative to its creator's own path. This is by virtue of the child jail being created in the chrooted environment of the first jail.

Ref: https://man.freebsd.org/cgi/man.cgi?query=jail

Top comments (0)