• Story16

Don't require ld-linux.so.2 to be inside ChrootSetuidJail

Status: in progress

Stop putting ld-linux.so.2 (the dynamic linker) inside the fixed chroot jail that is part of ChrootSetuidJail. This executable currently needs to be present inside the chroot so that we have an executable to invoke via execve(), because Linux does not provide an fexecve() system call that takes a file descriptor instead of a filename. Having the dynamic linker inside the chroot is problematic because it is part of PlashGlibc. It makes it hard to use different versions of PlashGlibc.

We could use a generic ELF loader instead. The loader can take a file descriptor for ld-linux.so.2 instead of a filename. It would not need to use the PlashObjectCapabilityProtocol, because that would defeat the object of making ChrootSetuidJail independent of changes to the comms protocol. The ELF loader could load the executable as well, but that is not essential because ld-linux.so.2 already knows how to load the executable.

The ELF loader could be based on rtldi.

This could have a performance impact. We should try to measure that, and set up ContinuousIntegration to measure this continuously.

See also UserModeExec.

See the ELF ABI documentation:

• ELF ABI (System V ABI Edition 4.1)

• Supplement for i386

• Supplement for amd64 (here is a much older version)

• Other specifications

Background

Normally doing exec on a normal dynamically-linked ELF executable works like this:

• Caller does the syscall exec([executable, arg1, arg2, ...]).

• The kernel opens the executable and looks for the INTERP field which gives the filename of the dynamic linker.
• The kernel clears out the process's memory mappings and maps the executable and the dynamic linker.
• The kernel passes control to the dynamic linker, which loads libraries (doing open() syscalls) and passes control to the executable.

With Plash, before this story, this becomes:

• Caller does the syscall exec(["/special/ld-linux.so.2", executable, arg1, arg2, ...]).

• The kernel opens the dynamic linker, special/ld-linux.so.2, which is the main executable from its point of view. It has no INTERP field. The pathname is interpreted inside the chroot.

• The kernel clears out the process's memory mappings, maps the dynamic linker, and passes control to it.
• The dynamic linker loads and maps the executable and its libraries, doing open() via RPC.
• The dynamic linker passes control to the executable.

After this story, this becomes:

• Caller opens the dynamic linker (the file descriptor is ldso_fd).

• Caller does the syscall exec(["/chainloader", ldso_fd, executable, arg1, arg2, ...]).

• The kernel opens /chainloader (pathname inside the chroot), which is the main executable from its point of view.

• The kernel clears out the process's memory mappings, maps the chainloader, and passes control to it.
• The chainloader maps the dynamic linker, using the file descriptor number it was passed. It closes the FD and passes control to the dynamic linker.
• As before, the dynamic linker loads and maps the executable and its libraries, doing open() via RPC.
• As before, the dynamic linker passes control to the executable.

Tasks

ELF chainloader:

• Make rtldi work on a recent Ubuntu:

  • Stack protector needs to be disabled (-fno-stack-protector)

  • a_ptr field has been removed from libc's elf.h

• Pare down rtldi:

  • Removed hacks that we don't need

  • Changed to use dietlibc for string functions and system calls

  • Rewrote the assembly code entry point

• Make the chainloader work on amd64

• Change it to take a file descriptor argument instead of using a fixed filename

• Improve error handling

• Make coding style changes
• Check that it works with gdb

• Check that it does not cause the stack to be writable

  • Link with -Wl,-z,noexecstack to fix this. dietlibc is not yet marked as not requiring an executable stack. See also PlashIssues/LintianExecutableStack.

• Consider rewriting the data above auxv on the stack so that ps gives nicer output for the process

• Check whether the segment gap issue causes the chainloader to leave behind undesirable mappings for ld.so in the gap
• Eliminate use of data segment on i386
• Ensure all source files contain copyright messages
• Make the chainloader relocatable, so that we do not need to build in a load address
• Check whether interrupted system calls can break the chainloader. It looks like dietlibc does not automatically reissue a system when it returns EINTR. Try interrupting system calls with SIGSTOP to test this.

Hook up the chainloader:

• Move code out of SVN scratch into main package

• Install chainloader into chroot

• Move ld-linux.so.2 out of chroot into /usr/lib/plash/lib

• Change execve() and FsOp to use chainloader

  • This will require that fsop_exec() return a file descriptor, as it did before. 120 removed the FD argument and return value, which actually became unused after 30 (a big commit from an import from before SVN) when the --fd option to ld.so was removed.

  • In the code before 120, PlashGlibc would allocate a spare FD slot number which it passed into fsop_exec, so that fsop_exec could return an argv that referred to the FD number. libc would use dup2() to move the FD returned by fsop_exec into the slot it has reserved. I have taken a different approach this time: fsop_exec returns a list of (index, FD) pairs, which specify the index of the argv to insert the FD number into, replacing a placeholder argument.

• Change plash.process module to use chainloader

• Change pola-run.c (not supported) to use chainloader

• Change pola-shell (not supported) to use chainloader

• Make sure we have tests that check that execve() works on #! scripts and returns an error when the executable can't be found

How should FsOp and pola-run find ld.so?

• When running under plash-pkg-launch, we can use /lib/ld-linux.so.2 in the sandboxed process's namespace because the contents of /usr/lib/plash/lib are bound over /lib. Eventually Plash's /lib/ld-linux.so.2 will be installed from a Debian package.

• However, pola-run does not use the same trick of binding /usr/lib/plash/lib over /lib. Instead, it sets LD_LIBRARY_PATH to point to /usr/lib/plash/lib.

• We could use LD_LIBRARY_PATH to find ld.so. When LD_LIBRARY_PATH contains multiple entries we might have to do multiple lookups, though that should not be a performance problem because ld.so would have to multiple lookups too. When LD_LIBRARY_PATH is not set or contains no matches, fall back to /lib/ld-linux.so.2. That is not quite ld.so's normal library lookup behaviour: it also searches /usr and directories in /etc/ld.so.conf.

• We could introduce a new search variable, perhaps LDSO_PATH.
• We could change pola-run so that it follows plash-pkg-launch's behaviour by binding /usr/lib/plash/lib into /lib instead of setting LD_LIBRARY_PATH. This would solve other problems, such as programs which unset LD_LIBRARY_PATH. Or it could just bind /usr/lib/plash/lib/ld-linux.so.2 into /lib.

• The problem with using an environment variable is that FsOp does not know about the sandboxed process's environment variables.

• FsOp could ignore the sandboxed process's namespace and always use /usr/lib/plash/lib/ld-linux.so.2 from the parent environment. This is not a long-term solution because it doesn't allow the sandboxed process to supply its own ld.so and libc.so (which need to be in sync with each other). However, it is not a regression from the current behaviour which of course already uses a fixed ld.so.

  • This is what it now does.

Docs:

• Update ChrootSetuidJail page

• Put notes on how to port the chainloader somewhere: entry.S needs porting

Testing

The test suite (all_tests.py) can be run in two ways: with or without wrapping it in run-uninstalled.sh. The latter would test with ChrootSetuidJail. Currently not all tests pass when not using run-uninstalled.sh, because pola-run-c and pola-shell are not installed (but are tested). How should we resolve that? Options:

• Another wrapper script, similar to run-uninstalled.sh, to test in a different way. -- Added run-mostly-installed.sh

• Stop testing pola-run-c and pola-shell. Don't want to do this, because I don't want them to bitrot. I don't want to remove them from the source tree. I would like them to continue to work, even if they are not supported.
• Install pola-run-c and pola-shell in the Debian package.

• Change test framework so that it can skip tests for programs that are not installed. This doesn't help, because it will still not be testing pola-run-c and pola-shell against ChrootSetuidJail.

Notes

See Story16Notes for chronological notes.

Questions

Would there be any advantages to using uclibc instead of dietlibc?