Advanced Namespace Tools blog
10 January 2020
Tribes, Fossil, 9front, Bell Labs
This post is a survey of technical and community issues connected to Plan 9 root file servers. The author is not a member of any Plan 9 tribe save Grid. Opinions are entirely my own and a lot of unsourced claims will be made based on my years of participation and observation of the Plan 9 community.
ANTS installer: fossil root for 9front code
The ANTS iso installer scripts include fossil+venti root install option. This makes it easier to use the 9front codebase with fossil+venti root. The 9front developers have made a conscious and deliberate choice to remove fossil support from their distribution, so undoubtedly they don't see this as an improvement. I have been using a primarily 9front codebase on top of fossil+venti root fileservers for my home grid of systems and for most of the 9gridchan.info servers for the past several years with good results. My datasets are small and my results may not translate well to users with much larger quantities of data. The latest ANTS release contains support tooling for venti replication as well as installer support.
This distribution of Plan 9 has admittedly flaky maintainership, but competent users should be able to track 9front changes even if mycroftiv has wandered off to some distant hypercube.
A highly subjective historical view
When 9front forked from the Bell Labs distribution of Plan 9, one of the major changes was removing the option to use the fossil file server as a root file system and making cwfs64x the primary fs in the installer. In the late 2000s era of Plan 9, many users reported unreliable behavior of the fossil+venti combination. The resources being given to Plan 9 by the corporate heirs to Bell Labs were clearly insufficient to properly develop and debug the operating system. As I understand it, there were two engineers working part-time on the public Plan 9 infrastructure. Without sufficient resource investment, even the best efforts of skilled developers may not scale to the needs of a full operating system project.
At the same time, the late and sorely missed Uriel was attempting to grow an independent community of people dedicated to the software principles espoused by the unix tradition and embodied in Plan 9. Unfortunately, in my view, Uriel's health problems influenced his communication, and his passionate desire to see Plan 9 succeed and thrive led him to sometimes rancorous debates that left both sides with emotional consequences. I struggle with mental health and communication issues myself and I write these comments from empathy, not blame. The way it all played out was to create divisions and bad feelings between the corporate and university Plan 9 developers and researchers and much of the grass-roots Plan 9 community.
Fossil-venti unreliability was a focal point of dissatisfaction. When 9front forked, removing fossil support from the installer was part of a clear message that the standard Bell Labs setup was not reliable. Discussions of the motivations and necessity for the 9front fork have often referenced poor user experiences with fossil+venti in the time prior to the fork. My own experience of trying to use fossil+venti in the 2000s era supports this perspective. In fact I found things so unreliable that a major factor in the software that became ANTS was "how to deal with the failure of the root filesystem" because that was a regular and expected occurence in my use of Plan 9.
The bug and fix
In 2012, after 9front had already forked, Richard Miller found and fixed a bug in fossil that I believe was the primary source of many fossil-venti reliability issues. He wrote a very nice post describing the bug and its solution.
I believe the precise nature of this bug explains a lot of things. As I understand it, the condition for triggering the bug is making changes to filesystem data during the archival snapshot process. For experienced Plan 9 users with always-on fileservers set to do a daily snapshot in the middle of the night, things were reliable. For a new user who installed Plan 9 and started immediately messing around and installing software while the initial archival snapshot is happening, there was a good chance that corruption would sneak into the fs from the very beginning. This matches my memory of frequent troubles with installs that seemed to break right away.
In 2015, I migrated ANTS to 9front, and because I have always used and valued the powerful operations that venti-backed fossil rootscores support, I decided to figure out how to use 9front/ants with a fossil root. I was expecting trouble because of the prevailing sentiment in the 9front community was that fossil was a no-good very-bad filesystem, and my own previous struggles. In practice however, what I discovered was that fossil seemed to be working just fine. I worked out a system for getting 9front+ants+fossil onto vultr vps nodes and started hosting 9gridchan.info websites on this platform as well as using it for my home systems.
Between 2015 and now, I believe I have accumulated 25+ machine-years of trouble free 9front/ants/fossil service on my home systems and public servers. Most of these systems only store a few gigabytes of data total so the workload is light however. Other users with large data sets have expressed some disappointment with performance.
The Tribes
Most people understand that the Plan 9 user community is both small, and strongly fractured into tribes who seem to harbor personal grudges toward each other. It's easy to understand why the sense of humor and strongly critical perspective of many "9front tribe" people rub some the wrong way - especially the people who are the actual target of satire. People just aren't really into being criticized and satirized as a general rule.
It's also easy to understand why what can only be referred to as the "Old Boys Club" of Plan 9 has critics. Plan 9 has always had overtones of elitism - from my reading, Rob Pike's writings and commentary set a lot of the tone which Uriel amplified. The "haughty style" of commentary on computing, in which the Sad and Fallen state of computing into complexity and muddled design is Lamented by the Sage goes back a long way - Dijkstra in particular perpetrated many famous examples. Humans love having social status, and nothing signals high social status more than condescending to the un-enlightened.
New users arriving in the Plan 9 community in the 2000s discovered a mismatch between the rhetoric of how much superior everything was in the Plan 9 world, and the reality of unreliable root fileservers, challenging and complicated administration with documentation focused on Murray Hill's system, and lots of rumors of intriguing code and research projects that didn't seem to feed into the publicly available Plan 9 experience. The operating system seemed rather stagnant and buggy, but the veterans of the Plan 9 community seemed to talk in a parallel universe in which Plan 9 was perfectly meeting all of their computing needs and any issues were the result of user error and inexperience.
The Present Day and SHA1
So, 9front tribe doesn't want to use fossil, and labs/legacy tribe doesn't want to use 9front code. I believe that both of these perspectives at this point are based more on human social conflicts than solid technical motivations. I am trying to offer a solution to users who are interested in exploring the most practical ways to use Plan 9. The major fly in the ointment is the outdated sha1 hash function used by venti. As a result of this yet-to-be-fixed shortcoming, the world of venti-backed public plan 9 services is not yet ready for large scale production use. For personal and hobby use howeveer, I believe the sha1 issues are manageable.
I hope that this post can serve a constructive purpose in building mutual understanding between humans and increasing code-sharing and collaboration. Given the way things often go, I'm probably screwing up and just pissing everyone on both sides off more by presenting my own warped and subjective view of things. I recall being told to "stop alternating history" for sharing my perspective on the history of Plan 9 community and software - so please offer your own corrections and perspective, especially if you think I have made any factual mistakes or you have first hand knowledge and experience to share.
I've always had pretty good interactions with other humans who use Plan 9 and I'm really into the whole "tolerant and diverse community" thing. I'd like a lot more people from different places and with different backgrounds and cultures and life experiences to get into Plan 9 and find it to be a fun and welcoming experience. I've been trying since 2008 to help people learn to use Plan 9 to communicate and connect and cooperate and have better and more direct control over their software and system, outside of systems of hierarchical authority and control and exploitation and coercion.
I've made a lot of mistakes in my own communication and community building efforts. 2020 seems like a Year of Change and I'm hoping for positive change in the Plan 9 community. I'm particularly excited for the reincarnation of a real-life Plan 9 conference/hackathon event. Humans do better in real life social situations than internet diatribes and perhaps the time is ripe for the ever-blooming Tree of Files to send forth new fruits.
Peace, love, hypercubes, and ultrafilters to all
Mycroftiv