The future of Offpunk: UNIX command-line heaven and packaging hell
by Ploum on 2023-10-01
A story about how the UNIX philosophy made me develop tools I’m actually proud of and why packaging is holding me back.
Two years ago, I decided that I wanted to be able to browse Gemini while offline. I started to add a permanent cache to Solderpunk’s AV-98, the simplest and first Gemini browser ever. It went surprisingly well. Then, as the excellent forlater.email service went down for a week, I thought that I would add a quick and hackish HTTP support to it. Just a temporary experiment.
The same week, I serendipitously stumbled upon chafa, an image rendering tool which was on my computer because of neofetch. I thought it would be funny to have pictures rendered in webpages in my terminal. Just an experiment to take some funny screenshots, nothing more.
But something really surprising happened: it was working. It was really useful. I was really using it and, after adding support for RSS, I realised that this experiment was actually working better for me than forlater.email and newsboat. Offpunk was born without really thinking about it and became a real project with its own philosophy.
Born on Gemini, I wanted Offpunk to keep its minimalistic roots: keeping dependencies under control (making them optional and implementing the underlying feature myself as soon as it makes sense), keeping it simple (one single runnable python script), caring as much as possible about older versions of python, listening to people using it on very minimal systems. I also consciously choose to use only solutions that have been time-trial-tested. I’ve spent too many years of my life falling for the "new-trendy-technology" and learned from those mistakes. The one-file aspect assured that it was really easy to use and to hack: open the file, modify something, run it.
I’m not a good developer. Anything more complex than that is too much for my taste. Unless forced, I’ve never used an IDE, never understood complex toolchains nor packaging. I modify files with (neo)vim (without any plugin), compile from the command line and run the resulting binary (not even needing that step with python). Life is too short for making it more complex. I like to play with the code, not to learn tools that would do it for me.
But offpunk.py was becoming fat. 4500 lines of organic python which have grown over an AV-98 structured to be a test bed for an experimental protocol. The number of people able to understand its code entanglement varied between 0 and 1, depending on the quality of my morning Earl Grey.
I wanted to make life easier for contributors. I also realised that some features I developed might be useful without offpunk. So I stepped into a huge refactoring and managed to split offpunk into several components. My goal was to separate the code into multiple individual components doing one thing and doing it well. And, to my own surprise, I succeeded.
Netcache.py
I called the first component "netcache". Think of netcache as a cached version of wget. If possible, netcache will give you a cached version of the URL you are asking. If no cache or too old and if allowed to go online, netcache will download it.
It means that if you like Offpunk’s core concept but don’t like the interface and want, for example, a GUI, you could write your own browser that would, using netcache, share the cache with Offpunk.
Netcache is currently working just well enough for my needs but could do a lot better. I should, for example, investigate replacing the network code by libcurl and implementing support for multithreaded concurrent downloads.
Ansicat.py
Coloured output in your terminal is done through a standard called ANSI. As I wrote the first HTML to ANSI renderer for offpunk, I started to understand how awful the HTML standard was. Armed with that experience, I started a second renderer and, to be honest, it is actually not that bad. I’m even proud of it.
Ansicat is really useful when in a terminal because it will render HTML and gemtext in a good, readable way. If the optional library python-readability is present, ansicat will try to extract the main content from a web page (and, yes, python-readability is one dependency I would like to reimplement someday).
With netcache and ansicat, you can already do something like:
netcache https://ploum.net | ansicat --format=html
Yes, it works. And yes, as a UNIX junkie, I was completely excited the first time it worked. Look mum, I’m Ken Thompson! Making ansicat a separate tool made me think about adding support for other formats. Like PDF or office documents. How cool would it be to have a single cat command for so many different formats?
Opnk.py
While netcache and ansicat were clear components I wanted to split from Offpunk’s core since the start of the refactoring, another tool appeared spontaneously: opnk.
Opnk (Open-like-a-punk) is basically a wrapper that will run ansicat on any file given. If given a URL, it will ask netcache for the file. Result will be displayed in less (after passing through ansicat, of course).
If ansicat cannot open the file, opnk fallbacks on xdg-open.
That looks like nothing but it proved to be massively useful in my workflow. I already use opnk every day. Each time I want to open a file, I don’t think about the command, I type "opnk". It even replaced cat for many use cases. I’m considering renaming it "opn" to save one character. Using opnk also explains why I want to work on supporting PDF/office documents with ansicat. That would be one less opportunity to leave the terminal.
Offpunk.py
Through this architecture, Offpunk became basically an interface above opnk. And this proved to work well. Many longstanding bugs were fixed, performance and usability were vastly improved.
Everything went so well that I dreamed releasing offpunk 2.0, netcache, ansicat and opnk while running naked with talking animals in field of flowers under a rainbow. Was it really Earl Grey in the cup that day?
Packaging Offpunk.py
Now for the bad news.
As expected, the refactoring forced me to break my "one-single-python-file" rule.
I felt guilty for those people who told me about using offpunk on very minimal systems, sometimes from a USB key. But I thought that this was not a real problem. Instead of one python script, I had four of them (and a fifth file containing some shared code). That should not be that much of a problem, isn’t it?
Well, python packaging systems would like to disagree. Flowers fade, the rainbow disappears behind black and heavy clouds while animals start to look at me with a devilish look and surprisingly sharp teeth.
I’ve spent many hours, asked several people on the best way to package multiple python files without making the whole thing a module. Without success. Hopefully, the community is really helpful. David Zaslavsky stepped on the mailing list to give lots of advice and, as I was discouraged, Austreelis started to work really hard to make offpunk both usable directly and packagable. I’m really grateful for their help and their work. But, so far, without clear success. I feel sad about the amount of energy required to address something as simple as "I’ve 5 python files which depend on each other and I want to be able to launch them separately".
The software is working really well. The refactoring allowed me to fix longstanding bugs and to improve a lot of areas while adding new features (colour themes anyone?) On my computer, I added four aliases in my zsh config: offpunk, opnk, ansicat and netcache. Each alias runs the corresponding python file. Nothing fancy and I want to keep it that way. I know for a fact that several users are doing something similar: git clone then run it from an arbitrary location.
Keeping things as simple as that is the main philosophical goal behind offpunk. It’s an essential part of the project. If people want to use pip or any other tool to mess up their computer configuration, that’s their choice. But it should never be required.
Which means that I’m now in a very frustrating position: Offpunk 2.0 is more than ready from a code point of view. But it cannot be shipped because there’s currently no easy way to package it. The pyproject.toml file had become an obstacle to the whole development process.
I’m contemplating putting everything back in one big file. Or removing the pyprojects.toml file from the repository and releasing offpunk "as it is".
Some will call me an old conservative fart for refusing to use one of those gazillion shiny packaging system. Others will judge me as a pretty poor programmer if I managed to do 20 years of Python without ever understanding pip nor using an IDE.
They are probably right. What would you seriously expect from someone doing a command-line tool to browse Gemini and Gopher?
But there’s maybe an easier solution than to change my mind and offpunk’s core philosophy. A simple solution that I missed. If that’s the case, don’t hesitate to drop a word on the devel mailing-list, Austreelis and I will be happy to hear about your opinion and your experience.
While you are at it, bug reports and feedback are also welcome. I’ve this odd custom of finding embarrassing bugs only hours after a release. I really hope to do better with offpunk 2.0.
And after we’ve solved that little packaging anecdote together, I will happily return to my bare neovim to code all the ideas I want to implement for 2.1, 2.2 and many more releases to come.
As a writer and an engineer, I like to explore how technology impacts society. You can subscribe by email or by rss. I value privacy and never share your adress.
I write science-fiction novels in French. For Bikepunk, my new post-apocalyptic-cyclist book, my publisher is looking for contacts in other countries to distribute it in languages other than French. If you can help, contact me!