Plash Object-Capability Protocol

Contents

Plash Object-Capability Protocol

The Plash Object-Capability Protocol (POCP) allows references to an arbitrary number of objects to be exported across a connection. It is layered on top of SimpleMessageProtocol.

Objects can be invoked, passing data, file descriptors and references to other objects as arguments. Object references can be dropped, allowing the storage associated with the reference to be reclaimed; the storage associated with the object itself can also potentially be reclaimed.

Protocol

There are two endpoints to a connection. Each may export object references to the other. The protocol is symmetric -- it doesn't distinguish between client and server. For the sake of explanation, however, let us call the endpoints A and B. Everything below would still hold true if A and B were swapped.

At any given point, A is exporting a set of object references to B. Each reference has an integer ID. These are typically interpreted as indexes into an array (the "export table"), so that when A receives an invocation request from B, it can look up the object in the array and invoke it.

The set of object references that A exports to B may be extended with new references by messages that A sends to B (but not by messages B sends to A). These object references may be removed by messages that B sends to A (but not by messages A sends to B). Messages in the protocol contain object IDs, which contain two parts. The lower 8 bits are a "namespace ID". This indicates whether the reference is to an object exported by A or by B, and whether a newly-exported reference is single-use. The rest of the object ID is the reference ID (an index into an export table).

The possible namespace IDs are:

#define CAPP_NAMESPACE_RECEIVER                 0
#define CAPP_NAMESPACE_SENDER                   1
#define CAPP_NAMESPACE_SENDER_SINGLE_USE        2

The messages that A may send to B are:

"Invoke object" message

"Invk" cap/int no_cap_args/int cap_args data + FDs

Invokes an object X. X is denoted by the object ID cap. X must be an object that B exports to A, so cap may only use the RECEIVER namespace. (Since A is sending, B is the receiver.)

cap_args is an array of object IDs of length no_cap_args. These denote objects to be passed as arguments to X. These object IDs may use any of the three namespace IDs:

For the RECEIVER namespace, this refers to an object that B exports to A.
For the SENDER namespace, this indicates that A has added a new reference to the set of objects it exports to B. From this point on, B may send A messages referring to this ID (except that B will refer to the object with the RECEIVER namespace instead of SENDER).
The SENDER_SINGLE_USE namespace works the same as SENDER, except that it indicates to B that the reference is single use. B may only invoke this object once. Once B invokes the object, the reference becomes invalid. (However, B may pass the object as an argument without this restriction.)

The message may include file descriptors to pass as arguments to X.

When B receives this message, it invokes X with the specified arguments. If cap is a reference that is exported as single-use, B removes the reference from its export table.

"Drop object" message

"Drop" cap/int

Drops a reference. cap is an object ID that B exports to A, so cap may only use the RECEIVER namespace.

When B receives this message, it removes the reference cap from its export table. B may also delete the object X that cap denotes if there are no other references to X.

Closing the connection

Violations: If either end receives a message that is illegal, such as messages that contain illegal object IDs, it may choose to terminate the connection. This would mean closing the file descriptor for the socket. Assuming there are no other copies of this file descriptor in the system (in this or other processes), the other end will get an error when it tries to read from its socket, and also regard the connection as broken. Having closed the connection, an endpoint is free to delete its export table, and possibly free the objects it contained references to.

In general, endpoints are free to close the connection anyway, if they want to.

When no references are exported from B to A or A to B, it is conventional to close the connection, because it is of no use: no messages can legally be sent on it. Conventionally, A will close the connection rather than sending a "Drop" message for the last reference that B exports to A, when A exports nothing to B.

Conventions

Initial state of a newly-created connection

POCP is unusual in having no special set of messages for setting up a connection. All setup -- such as establishing the initial sets of objects exported each way across the connection -- is done out of band. The connection is assumed to be already authenticated. (Contrast with the authentication steps in the X11 and D-Bus protocols.)

A and B start off holding socket file descriptors connected to each other (typically created by socketpair()). A exports M references to B; these are given IDs 0 to M-1. B exports N references to A; these are given IDs 0 to N-1.

The numbers M and N must be made known to both A and B by some means outside of the protocol, just as the file descriptors are obtained by some means outside of the protocol.

If A and B have differing views about what M and N are, one will probably send messages that the other sees as a protocol violation, and the latter may close the connection.

Of course, M and N and the file descriptors can be sent in invocations using the protocol. See the conn_maker object.

See ProtocolEnvVars for the PLASH_COMM_FD and PLASH_CAPS environment variables.

Call-return

With most invocations, you want to receive a result (even if it's just an indication of success or failure). In these cases, an object X is invoked with a message starting with "Call". The first object argument is a "return continuation", C. When it has finished, X invokes C with arguments containing the results.

What happens if C is never invoked? This might happen if a connection is broken. C will get freed in this case, perhaps as a result of a "Drop" message, and this can be used to indicate to the caller that the call failed.

What happens if C is invoked more than once? C should simply ignore any invocations after the first one.

A return continuation is typically exported as a single-use capability. This is not so much to stop it being invoked more than once (because subsequent invocations can easily be ignored), but more to prevent the build-up of exported references:

When A repeatedly calls B, B might fail to drop the references to the return continuations that A passes it after invoking them. This would cause A's export table to fill up with useless references. A could not legally re-use the IDs for these references according to the protocol. However, if A passes the return continuations to B as single-use references, B cannot legally use their IDs after invoking them, so A can re-use the IDs and free up space in its export table. (If B does invoke an already-invoked single-use reference, it is violating the protocol and A might close the connection as a result.)
However, this was not the immediate motivation for adding single-use references to the protocol. More importantly:
libc.so and ld-linux.so (the dynamic linker) both need to make calls to objects in order to open files, etc. So they both need to pass return continuations, and allocate IDs for them. If the return continuations' IDs are invalidated after each call, libc.so and ld-linux.so can allocate the IDs without regard to each other. It is much simpler when they don't need to co-ordinate but can still share the same connection. Each return continuation can be exported with the same ID; these are the only objects exported from this end of the connection.
The same issue arises when passing control to a new process image using "exec".
Without single-use references, ld-linux.so might make a call and receive an "Invk" message as a result (but not wait for any further messages). Then libc.so might make a call, then listen for a result and receive a "Drop" message for a reference it never exported. libc.so would treat this as a protocol violation and shut down the connection. With single-use references, the "Drop" message is unnecessary, because it is implied.

Future extensions

Message pipelining

The protocol does not provide a facility for message pipelining, ie. letting A invoke the result of a call to B before the call returns (saving the time of a round trip).

Such a facility involves letting A's messages add entries to B's export tables. A would be able to choose IDs for references that B exports. It would no longer be the case that B allocates all the IDs that it exports.

Comparison with other protocols

E Pluribus protocol

http://www.erights.org/elib/distrib/index.html

The ELanguage's protocol for remote object invocation is called Pluribus. It is currently implemented in Java. There does not appear to be a specification document for Pluribus, but its properties are covered by the documentation for E.

The ELanguage documentation talks in terms of communication between "vats" rather than processes.

Pluribus is intended for distributed use, while POCP is intended for processes on the same machine.
- Pluribus provides an encryption layer, while this is out of scope for POCP.
- In POCP, object references sent over the connection are small integers. In Pluribus, object references are large, unguessable integers known as "Swiss numbers".
Pluribus provides message pipelining.
Pluribus does "path-shortening": when A sends B a reference to C, this can cause a connection to be set up between B and C. As a temporary measure, the protocol does forwarding, and B will send its messages to C via A. In contrast, POCP always does forwarding.

D-Bus

http://www.freedesktop.org/wiki/Software/dbus

D-Bus provides a more structured data format: it defines an encoding of lists and trees of values into messages. In contrast, POCP messages just consist of an array of bytes, and encoding argument lists etc. is layered on top of POCP.
D-Bus objects are named by an object path string, rather than integers.
D-Bus object references are serialised into the same part of the message as other data. In contrast, POCP segregates object references and data in the message encoding.
D-Bus does not provide any reference counting: it provides no way to free a reference.
Call-return is a primitive in D-Bus but not in POCP.
D-Bus does not allow sending Unix file descriptors in messages.