• StoryTest3

Make glibc's test suite work with PlashGlibc

Status: in progress

I tried in the past to make glibc's test suite work outside of the glibc build tree, but that turned out not to be viable. Following Story6, it should be a lot easier to make glibc's test suite work with PlashGlibc.

Tasks

• Run tests on unmodified glibc 2.7 (since that is the current base version of PlashGlibc) and make sure there are no failures

  • There are failures on Ubuntu as a result of /bin/sh being dash

  • Extend BuildTools to run these tests

• Fix simple_wrapper to return exit status codes properly

• Write script to search output of make -k check for failures -- tests/format_glibc_tests.py

  • Extend it to know about the current known failures. Divide output into expected and unexpected failures, and unexpected successes. The list is quite long, containing ~75 tests. The list contains the reasons for the failures, where known. I can use this as a to-do list for fixing tests, for fixing the easiest cases first.

• A large number of tests assume they can invoke ld.so directly. Instead of relying on each test invoking simple_wrapper.py, consider running all of make check with PlashGlibc under simple_wrapper.py or pola-run.

  • This would require being able to handle exec'ing ld.so directly. Could have FsOp or the ELF chainloader check for the INTERP field in executables, or just add a special-case wrapper for FsOp that looks for an "ld.so" filename.

    • Added python/scripts/test_wrapper.py, which provides a quick hack that wraps FsOp to support static executables.

    • Incidentally, why does ld.so segfault when invoked on a statically-linked executable instead of failing gracefully? ld.so expects the executable to have a DYNAMIC section and does not fail gracefully if it lacks one. Also, ld.so can't invoke ld.so because it expects to relocate itself, not to be relocated by the caller.

  • Would probably need to address PlashIssues/MakeRebuildsTooMuch. -- Worked around it for now by changing glibc's Makerules

  • I discovered that make uses vfork(), which was broken. See PlashIssues/VforkBroken.

• Extend BuildTools to run these tests for PlashGlibc

• Bugs found:

  • nptl/tst-fork1: apparently finds a deadlock in fork(). Raised as a bug: PlashIssues/ForkDeadlock.

• Some tests failed on i386 but not x86-64. These failures happened because ld.so and libc.so were mismatched, and this caused a segfault on i386 but not on x86-64.

  • elf/tst-leaks1-mem: this invokes the Perl script glibc-build/malloc/mtrace (a #! script). test_wrapper.py was incorrectly treating #! scripts as statically linked, so they were being exec'd directly. That meant the process was using Plash's libc.so (because LD_LIBRARY_PATH was still set) with the host's ld.so.

  • stdio-common/bug5.out and stdio-common/test-popen.out: These explicitly unset LD_LIBRARY_PATH when starting a subprocess. This means the subprocess uses the host's libc.so with Plash's ld.so and segfaults.

• debug/tst-chk3.out (built from debug/tst-chk1.c): 4 cases in this test fail, all of which are "When the format string is writable and contains %n". The writability check is not working. The check for %n is implemented in stdio-common/vfprintf.c, which calls out to a function called __readonly_area(), which is implemented in sysdeps/unix/sysv/linux/readonly-area.c. This tries to read /proc/self/maps, which is not available under Plash (or if it is available, it will have information for the wrong process).

• rt/tst-cpuclock2.out sometimes fails (on i386) with the output:

• live thread clock fffd9e76 resolution 0.000000001
  live thread before sleep => 0.019379600
  self thread before sleep => 0.000207576
  live thread after sleep => 0.472465764
  self thread after sleep => 0.000229747
  libgcc_s.so.1 must be installed for pthread_cancel to work
  Didn't expect signal from child: got `Aborted'
  

• The last two lines show the problem. The test uses pthread_cancel() and glibc needs libgcc_s.so.1 for the unwind support that libgcc provides. glibc dynamically loads libgcc. I suspect that it is trying to dlopen() the library from a thread but dlopen() is not currently thread-safe in PlashGlibc. There is a race condition between ld.so and libc.so. They use the same file descriptor to talk to the ServerProcess but do not share a lock to co-ordinate use of this FD.

Unmodified glibc test failures

../configure --prefix=/usr CFLAGS="-O2 -g0" CC="gcc -fno-stack-protector"
make -k check >make-log 2>&1
grep '\*\*\*' make-log

glibc 2.7 on Ubuntu gutsy, x86-64:

• localedata/tst-fmon
• nptl/tst-cancel7:
• nptl/tst-cancelx7
• nptl/tst-cleanup0: bashism in nptl/Makefile

glibc 2.8 on Ubuntu feisty, i386:

• iconvdata/tst-iconv7

  • This post has a fix

• nptl/tst-cancel7
  • When run normally, it fails with "child X still running".
  • When I strace this, it hangs, which appears to happen because the thread doing system() is not being cancelled.

  • This post and this post are related.

  • I'm not sure how fcntl() locking is supposed to interact with signals and sigsuspend().
  • It appears that this test checks that cancelling system() is handled properly. From sysdeps/unix/sysv/linux/system.c, "We have to and actually can handle cancelable system(). The big problem: we have to kill the child process if necessary." Having to kill the child process is slightly unusual.

  • Note that sysdeps/unix/sysv/linux/system.c inlines the clone system call, which could be a problem for PlashGlibc. Should add a test for system().

  • This fails on Ubuntu because of a bashism. system() invokes /bin/sh (and this is hard-coded, not changeable via any environment variable). When doing /bin/bash -c "/bin/echo foo", bash is clever enough to invoke /bin/echo with execve() without forking. When system() kills the subprocess, it kills the intended subprocess. But dash forks a further subprocess and waits, so system() just kills dash, not dash's subprocess.

• nptl/tst-cancelx7
• nptl/tst-cleanup0

glibc test framework

The glibc tests are run entirely through makefiles. Problems:

• The test targets are *.out files, which are written even if the test fails. This means that each time you run "make check", it gives you the next test failure; it does not redo the failures unless you delete the *.out files.

  • You can remove the *.out files with make tests-clean (see Makefile -- the one in the top-level source directory, not the generated one in the build directory).

  • A simple fix would be to output to *.out.tmp and rename the file into place after the test has succeeded, as the glibc makefiles do in a number of other rules.

• If you use make -k check, you can get multiple failures in one run, but they are not nicely formatted in comparison with Python's unittest output.

  • You can grep the output for "***", but this includes lines for subdirectories containing failures as well as the failures themselves.
  • The failure lines contain absolute pathnames.
  • The contents of the *.out files are not included, so you can't quickly tell why the test failed.
• Tests are obscurely named, e.g. "tst-cancel25".

The rules for making the *.out targets are in Rules.

Some of the tests are slow because they contain sleeps and timeouts. Would using -j with make help? glibc's main Makefile contains .NOTPARALLEL (which overrides -j), but it appears to be possible to pass a -j flag to the subdir makefiles via PARALLELMFLAGS.

How to run a single test, such as csu/tst-empty:

bash -c 'make -C glibc-source/csu objdir=$(cd glibc-build && pwd) $(pwd)/glibc-build/csu/tst-empty.out'

Dan Kegel seems to have another way of running glibc's tests which involves building a shell script. See crosstest-howto.

The output here on an eglibc mailing list contains collated glibc test results but I don't know what produces it.