Jailkit howto - debugging a chroot jail

Objectives

Often when adding new functionality to a jail, things are not working as expected. Some file is probably missing in the jail. But how to find which file?

Debugging the jail

One of the most useful programs to find files that are required in a jail is strace. Strace is available on most Linux distributions and *BSD operating systems. The source is is available from http://www.liacs.nl/~wichert/strace/. Strace will print all system calls and their arguments, and will therefore print any file that is opened.

Another very useful program is makejail. Makejail is available from http://www.floc.net/makejail/. Makejail uses short config files to copy some required files, and makes heavy use of strace to find any additional requirements. Makejail copies sometimes too much. Check your jail carefully after makejail is finished if everything there is really required.

Finding requirements outside the jail.

If you have some process running on the real system, but you want to move it into a jail, a good way to start is to use strace on the real system and look for the required files. Lets try to find anything required to use the editor joe. Let's start with a strace. strace joe The first lines of output are interesting already execve("/usr/bin/joe", ["joe"], [/* 25 vars */]) = 0 uname({sys="Linux", node="aria", ...}) = 0 brk(0) = 0x8096000 old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40017000 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=65190, ...}) = 0 What we see is that it immediately tries to open 3 files, but only the third file exists: /etc/ld.so.cache. This is a file that tells the executable where to find all libraries on this system. It is not required, but it might speedup the start of many programs.

In the following lines there is a lot of useless information (for now), but also some interesting lines, here is a summary: open("/lib/libncurses.so.5", O_RDONLY) = 3 open("/lib/tls/libutil.so.1", O_RDONLY) = 3 open("/lib/tls/libc.so.6", O_RDONLY) = 3 These are obviuously libraries used by joe. As you see all files that are used can be found by tracing the open system call. There is another one to find required files: the access system call. If you use grep to search trough the strace output, you can easily get a list of required files this way.

Using grep, strace and a temporary log file: strace joe > /tmp/strace.log 2>&1 egrep '^(access|open)' /tmp/strace.log | grep -v ENOENT we get the following output: open("/etc/ld.so.cache", O_RDONLY) = 3 open("/lib/libncurses.so.5", O_RDONLY) = 3 open("/lib/tls/libutil.so.1", O_RDONLY) = 3 open("/lib/tls/libc.so.6", O_RDONLY) = 3 open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3 open("/usr/share/locale/locale.alias", O_RDONLY) = 3 open("/etc/mtab", O_RDONLY) = 3 open("/proc/meminfo", O_RDONLY) = 3 access("/etc/terminfo/x/xterm", R_OK) = 0 open("/etc/terminfo/x/xterm", O_RDONLY) = 3 open("/etc/joe/joerc", O_RDONLY) = 3 open("/etc/joe/joerc", O_RDONLY) = 3 open("/etc/localtime", O_RDONLY) = 3 These are all successful file access and open calls executed by joe on the real system. Not every file will be really required, but if things do not work as expected, you know where to look.

Debugging a user account in a jail

Debugging inside the jail is very similar. But if you don't want to have the strace utility itself inside the jail you need a slight variation. First lookup the PID of the shell of the jailed user using jk_list, or simply ps: # jk_list Pid User Jail Command 30450 john /home/downjail /home/downjail/bin/bash Now trace this process (using -p), and it's children (using -f): strace -f -p 30450 If you have a lot of trace information it can be useful to store the traces into a file (-o). Strace can store the trace for each child process in a different file (-ff): strace -ff -p 30450 -o trace

Debugging a user account without shell in a jail (sftp, cvs, etc.)

The above example works for user accounts that have a shell. But how to trace sftp or cvs sessions? You can trace all children of the sshd deamon process, but that will result in an enormous amount of trace information.

For such debugging I use the following strategy. First I start a sftp/cvs session to the jailed user, BUT I DO NOT YET LOG IN sftp jailuser@localhost then, in a separate terminal, as root, I look at the process ID from the ssh process that is accepting the connection ps axu|grep sshd|grep jailuser this will show something like # ps axu|grep sshd|grep jailuser root 30752 0.2 0.2 6460 2124 ? Ss 10:40 0:00 sshd: jailuser [priv] sshd 30753 0.0 0.1 6272 1428 ? S 10:40 0:00 sshd: jailuser [net] root 30754 0.0 0.0 6460 792 ? S 10:40 0:00 sshd: jailuser [pam] Now connect the strace to the PID of the sshd [priv] process. This process will eventually start jk_chrootsh, which in it's turn will start jk_lsh (or another shell), which in it's turn will start the sftp-server process: strace -p 30752 -ff -o /tmp/tracefile now continue to log in in the first terminal as jailuser and the trace logs will be generated.