Advanced Namespace Tools blog

06 March 2018

History of ANTS, part 2

After describing how I arrived in the promised land of namespaces, I will move on to the process of writing the software that became ANTS. The first piece is hubfs, which is still the most generally-useful thing I have done in Plan 9, and the program in which I have invested the most time in continuing improvements.

Pre-hubfs: scripting pipe devices, and iosrv

During my years of using linux, I always found "screen" to be a useful part of my workflow. I got in the habit of pretty much always working within a screen session, not only when SSH'd to a remote system, but locally as well. Decoupling my work from the outer layer of the user interface seemed to improve just about everything. Given Plan 9's nature as a distributed OS, I was rather surprised that there didn't seem to a screen-equivalent program - how was I supposed to avoid losing state on remote systems if I needed to disconnect? There was a weird old program called "consolefs" which was focused on serial lines, but didn't seem like an optimal general solution.

Things originated from me playing around with having two machines mounting the same window system, and then using the pipe device to create share input/output to a shell. This old blog post and one from the next day show me playing around with sharing a single rio between multiple machines and using the pipe device and tee and shell directions to create a rough system for sharing a single shell.

I had been learning C from k&r along with Nemo's "Intro to OS Abstractions" Plan 9 book, and the fact that I really wanted a usable "screen" type program prompted me to begin development in earnest. During the next couple weeks I hacked for about 16 hours a day on a program that was the direct forerunner to hubfs: iosrv. It is almost exactly like hubfs, except with a hackier implementation and a much more awkward interface. It creates pipes with the pipe device, wraps them into a /srv with exportfs, and then creates an internal data buffers for each pipe/file descriptor. The abstraction called a "hub" which extends the idea of a pipe originated here. I recall spending seemingly infinite hours trying to get the locking correct.

Looking back at the iosrv code it is clear that it really is the same program as hubfs, just implemented more awkwardly. The same basic idea of providing a mux-buffer for each shell fd and having an outer shell-client program that sits between the user and the shell 'inside' the io ubs, and a lot of the control interface and internal data structures are the same.

Hubfs

I proceeded immediately from iosrv into rewriting it as hubfs. I was mostly displeased with the hacky mechanism of creating pipes with the pipe devices and then running an exportfs into /srv. I knew things should be done as a real 9p fs. Since I hadn't written a 9p fs before, there was a lot of learning involved in making the translation. It took me awhile to grok the basic concept of how lib9p worked, that the flow of control was set by the lib9p service loop and my fs functions were being called from within it. I think this "design pattern" is very beautiful and powerful and I feel like it could probably be used more often than it seems to be.

From checking sourcesdump, I see that I uploaded hubfs.tgz on August 23, 2009. That makes just over a month of development from "rc script experiments" through iosrv to hubfs. I remember I was doing absolutely nothing except coding during this entire time, working with absolute and complete obsessive focus. Most of my previous experience of creative flow had been in the context of piano playing and musical performance and the feeling of programming when "in the zone" had certain similarities, but also strong contrasts. In addition to the emotional/logical divide between music and coding, programming seems to have the capacity to take over my brain for a longer span of time, up to several weeks for a single problem.

Because ANTS is "namespace tools" it seems relevant to explain just how hubfs fits in. In a trivial sense, any 9p fs is in a certain sense a 'namespace tool' because it works by creating an fs interface within your namespace, but this is vacuous. More relevantly, persisting a shell provides a means of saving and moving between namespaces. The main example in ANTS is that a hubfs is started within the independent "rootless" namespace created during boot, allowing that namespace to be accessed via the hubfs from the main namespace. This is a specific example of a general principle - one purpose of hubfs is to allow data to flow easily between divergent namespaces.