Linux ptrace() mechanism
Extensions
Jeff Dike implemented PTRACE_SYSCALL_MASK in order to implement time virtualisation efficiently in User Mode Linux, which relies on ptrace(). This would be useful for implementing PtraceJail efficiently. The patch was posted to linux-kernel as an RFC. Jeff Dike has not submitted it for inclusion in the mainline kernel.
utrace reimplementation
Roland McGrath's utrace is a reimplementation of the ptrace mechanism. As of 2007/12/01, it has not been merged into the mainline kernel, but it is apparently included in Fedora kernels.
Fail-safety
When ptrace() is being used for security, if the tracing process dies we want the traced processes to be killed or at least stopped. If the traced processes were no longer traced, all system call restrictions would be lifted. Based on the current documentation for ptrace(), there does not appear to be a way to use it in a fail-safe way.
Systems using ptrace
Subterfugue, written in Python
Libraries
Gotchas
The meaning of syscall numbers depends on whether the process is in 32-bit mode or 64-bit mode. See CESA-2009-001 and blog post, Bypassing syscall filtering technologies on Linux x86_64.