Advanced Namespace Tools blog

16 January 2018

A brief exploration of the srv function of lib9p

About the one thing almost everyone knows about Plan 9 is that "everything is a file" and they might also have heard that these files are generally accessed using the 9p protocol. The purpose of this post is to dig a bit deeper into how 9p servers and clients often communicate. Note that all code in this post is copy-pasted from the 9front distribution of Plan 9.

You don't need lib9p to have a 9p server

Before getting into the main topic, it should be pointed out that using lib9p for 9p servers is entirely optional. Many - perhaps most - of the 9p servers in Plan 9 do not do so. Rio doesn't use it, neither does upas/fs, nor do cwfs or fossil. Instead, these servers simply include fcall.h, the library routines that handle conversion to and from the machine independent "on-the-wire" format of 9p. They provide all their own logic for managing the data structures that correspond to files in the fs.

For the third edition of Plan 9, released in 2000, the developers saw that this approach, while perfectly workable, led to a fair amouint of parallel "boilerplate" code. Because the whole idea of an fs interface was premised on having the filesystems behave in the way you expect filesystems to behave, it makes sense to have a library which provides standardized routines for common operations. Some developers still prefer to handle everything themselves, but using lib9p can be very helpful for convenience and guaranteeing correct behavior.

The core of lib9p: the srv() function and Srv structure

When you write a 9p server using lib9p, what you are doing is providing portions of the Srv structure with pointers to the specific functions that will be called by the srv() function in response to incoming requests (T-messages at the protocol level.) Understanding this flow of control is crucial. Your server is going to have a main() function which probably calls postmountsrv() to begin the service loop. postmountsrv uses a forker helper function to call postproc() with a pointer to the Srv structure, which invokes srv() with that pointer, and srv() invokes the actual loop in srvwork() (in the original Plan 9 from Bell Labs, there is no srvwork() and the loop happens directly within srv()):

while(r = getreq(srv)){
	incref(&srv->rref);
	if(r->error){
		respond(r, r->error);
		continue;	
	}
	qlock(&srv->slock);
	switch(r->ifcall.type){
	default:
		respond(r, "unknown message");
		break;
	case Tversion:	sversion(srv, r);	break;
	case Tauth:	sauth(srv, r);	break;
	case Tattach:	sattach(srv, r);	break;
	case Tflush:	sflush(srv, r);	break;
	case Twalk:	swalk(srv, r);	break;
	case Topen:	sopen(srv, r);	break;
	case Tcreate:	screate(srv, r);	break;
	case Tread:	sread(srv, r);	break;
	case Twrite:	swrite(srv, r);	break;
	case Tclunk:	sclunk(srv, r);	break;
	case Tremove:	sremove(srv, r);	break;
	case Tstat:	sstat(srv, r);	break;
	case Twstat:	swstat(srv, r);	break;
	}
	if(srv->sref.ref > 8 && srv->spid != getpid()){
		decref(&srv->sref);
		qunlock(&srv->slock);
		return;
	}
	qunlock(&srv->slock);
}

So a request arrives, let's trace the case of Tremove. This calls out to the sremove function, still within srv.c in lib9p:

if((r->fid = removefid(srv->fpool, r->ifcall.fid)) == nil){
	respond(r, Eunknownfid);
	return;
}
if(!dirwritable(r->fid)){
	respond(r, Eperm);
	return;
}
if(srv->remove)
	srv->remove(r);
else
	respond(r, r->fid->file ? nil : Enoremove);

This code checks for a couple possible errors, then looks to see if srv->remove exists. That corresponds to whether or not your fs provides a definition for a remove function. If your code does provide a remove function, that function is called with the request r as the parameter. If it doesn't, the code checks if file trees are in use (more on those in a moment) and if they are not, returns an error to the client that the file can't be removed.

The respond() function

The code you have provided receives the request, performs whatever operations it decides are appropriate based on the content of the request, and then calls the respond function to complete the operation. This is again part of lib9p/srv.c. The core of respond() is again a switch-case statement based on message type:

switch(r->ifcall.type){
/*
 * Flush is special.  If the handler says so, we return
 * without further processing.  Respond will be called
 * again once it is safe.
 */
case Tflush:
	if(rflush(r, error)<0)
		return;
	break;
case Tversion:	rversion(r, error);	break;
case Tauth:	rauth(r, error);	break;
case Tattach:	rattach(r, error);	break;
case Twalk:	rwalk(r, error);	break;
case Topen:	ropen(r, error);	break;
case Tcreate:	rcreate(r, error);	break;
case Tread:	rread(r, error);	break;
case Twrite:	rwrite(r, error);	break;
case Tclunk:	rclunk(r, error);	break;
case Tremove:	rremove(r, error, errbuf);	break;
case Tstat:	rstat(r, error);	break;
case Twstat:	rwstat(r, error);	break;
}

Continuing our trace of the execution of a Tremove message, we show the body of the rremove function:

if(error)
	return;
if(r->fid->file){
	if(removefile(r->fid->file) < 0){
		snprint(errbuf, ERRMAX, "remove %s: %r", 
			r->fid->file->name);
		r->error = errbuf;
	}
	r->fid->file = nil;
}

You see that unless file trees are in use, this function does nothing - it assumes that the remove function called previously has already done everything that is needed. After this returns, the respond function performs quite a bit of bookkeeping on the state of the 9p connection, a sample of which follows:

r->ofcall.tag = r->ifcall.tag;
r->ofcall.type = r->ifcall.type+1;
if(r->error)
	setfcallerror(&r->ofcall, r->error);
qlock(&srv->wlock);
n = convS2M(&r->ofcall, srv->wbuf, srv->msize);
if(n <= 0){
	fprint(2, "msize = %d n = %d %F\n", srv->msize, n, &r->ofcall);
	abort();
}
assert(n > 2);
if(r->pool)	/* not a fake */
	closereq(removereq(r->pool, r->ifcall.tag));
m = write(srv->outfd, srv->wbuf, n);
if(m != n)
	fprint(2, "lib9p srv: write %d returned %d on fd %d: %r", n, m, srv->outfd);
qunlock(&srv->wlock);
qlock(&r->lk);	/* no one will add flushes now */
r->responded = 1;
qunlock(&r->lk);

The "actual sending of the response" is done by first convS2M (one of the fcall.h routines) to convert the message to wire-ready-format, and then write(srv->outfd, srv->wbuf, n) to actually transmit the message on the file descriptor for the client connetion. Shortly after, respond returns control to the function in your 9p server it was called from - which is probably going to return shortly after the respond was called, and the flow of control will be returned to the while loop of srvwork().

A recapitulation of the control flow

Let's review how this worked.

Your 9p server defines a set of functions that are function pointers within a Srv structure
You call postmountsrv() to start the lib9p service loop
The srvwork() function runs a while loop to process incoming requests
A request is received and passed to a specific lib9p function such as sremove()
The sremove() function calls the srv->remove() function provided by your fs code
Your fs code does what it needs, then calls the respond() function in lib9p
The respond() function calls an additional message-specific function such as rremove()
The respond() function finishes 9p housekeeping and then returns control to your fs
Your fs function completes and returns control to the srvwork() loop

Simplified, we have a loop of srvwork(), in which we have repated calls to different functions within your fs, each of which is probably going to call back to lib9p respond() function during the course of their operation. Each incoming request causes the flow of control to "bounce back and forth" between lib9p and your fs functions several times. Reasoning about this can become tricky, especially if your fs needs to do things like defer the handling of some requests - in which case you may not call respond() until a subsequent condition is met. Hubfs does this continuously - it receives read requests, which it must queue and not send a respond() for until new data has been written, at which point the pending requests can be fulfilled.

9pfile: file tree handling for additional convenience

The Plan 9 authors provided an additional optional mechanism to make handling arbitrary file trees even easier: the 9pfile functions. We saw above several places where the behavior of the lib9p srv-provided functions branched depending on whether fid->file existed. This is determined by whether or not your fs invoked the alloctree() function to provide a pointer to a file tree prior to calling postmountsrv(). If it did, the following logic in srv.c/sattach() will trigger:

if(srv->tree){
	r->fid->file = srv->tree->root;
	incref(r->fid->file);
	r->ofcall.qid = r->fid->file->qid;
	r->fid->qid = r->ofcall.qid;
}

The file tree functions are in lib9p/file.c and are well suited to fileservers which may create arbitrarily structured trees. When using file trees, operations like walking directories are handled automatically by the library with no need to provide service functions. An absolutely minimal and simplistic 9p server might provide no service functions save "read".

Plan 9 Advanced Namespace Tools A distributed systems fantasia