2. 2018-01-05

    I am a fairly sanguine UK rail commuter. I do not understand why they schedule the annual price hike for exactly the same week the holiday maintenance work is likely to have overrun, affecting all the services. Surely it would make more sense to raise the price at the start of the financial year, in April?

    posted by cms on
    tagged as

  3. 2018-01-04

    Old Street station has a popup "Black Mirror" shop. I am not sure how I feel about this. I am fairly sure I know how Dan Ashcroft would feel about this.

    posted by cms on
    tagged as

  6. 2025-11-20

    According to the Discourse, somebody killed perl

    There's been a flurry of discussion on Hacker News and other tech forums about what killed Perl. I wrote a lot of Perl in the mid 90s and subsequently worked on some of the most trafficked sites on the web in mod_perl in the early 2000s, so I have some thoughts. My take: it was mostly baked into the culture. Perl grew amongst a reactionary community with conservative values, which prevented it from evolving into a mature general purpose language ecosystem. Everything else filled the gap.

    I remember Perl

    Something to keep in mind, is that although this is my personal take, and therefore entirely an opinion piece, I was there at the time. I stopped doing Perl properly when I left Amazon, I think this would have been around 2005. It's based on the first hand impressions of somebody who was very deeply involved in Perl in its heyday, and moved on. I have a lot of experience, from both inside and outside the tent.

    Perl's roots are sysadmin

    What culture? Perl always had a significant amount of what you might call "BOFH" culture, which came from its old UNIX sysadmin roots. All of those passive aggressive idioms and in jokes like "RTFM", "lusers", "wizards", "asking for help the wrong way" etc. None of this is literally serious, but it does encode and inform social norms that are essentially tribal and introverted. There implicitly is a privileged population, with a cost of entry to join. Dues must be paid. Cultural conservatism as a first principle.

    This stems from the old locked-down data centre command culture. When computer resource was expensive, centralised, fragile, and manually operated, it was rigidly maintained by gatekeepers, defending against inappropriate use. I started my career as an apprentice programmer at the very end of this era, (late 80s) pre-web, and before microcomputers had made much inroads, and this really was the prevailing view from inside the fort. (This is a drawback about fort-building. Once you live in a fort, it's slightly too easy to develop a siege mentality). Computers are special, users are inconvenient, disruption is the main enemy.

    An unfortunate feedback loop in this kind of "perilous" environment is that it easily turns prideful. It's difficult to thrive here, if you survive and do well you are skilled; you've performed feats; you should mark your rites of passage. This can become a dangerous culture trap. If you're not careful about it, you may start to think of the hazards and difficulties, the "foot guns", as necessary features - they teach you those essential survival skills that mark you out. More unkindly, they keep the stupid folk out, and help preserve the high status of those who survived long enough to be assimilated. Uh-oh, now you've invented class politics.

    The problem with this thinking is that it's self-reinforcing. Working hard to master system complexities was genuinely rewarding - you really were doing difficult things and doing them well. This is actually the same mechanism behind what eventually became known as 'meritocracy'1, but the core point is simpler - if difficulty itself becomes a badge of honour, you've created a trap: anything that makes the system more approachable starts to feel like it's cheapening what you achieved. You become invested in preserving the barriers you overcame.

    (This is the same mentality that built leetcode interview pipelines BTW, but let's leave that sidebar alone for now)

    So the UNIX operator culture tended to operate as a tribal meritocracy (as opposed to the UNIX implementer culture, which fell out of a different set of cultural norms, quite an interesting side bar itself2), a cultural priesthood, somewhat self-regarding, rewarding of cleverness and knowledge hoarding, prone to feats of bravado, full of lore, with a defensive mentality of keeping the flame aloft, keeping the plebs happy and fed, and warding off the barbarians. As we entered the 90s it was already gently in decline, because centralised computing was giving way to the rise of the microcomputer, but the sudden explosive growth of the WWW pulled internet / Unix culture suddenly back into the mainstream with an enormous and public opportunity vacuum. Everyone suddenly has an urgent need to write programs that push text off UNIX file-systems (and databases) and into web pages, and Perl is uniquely positioned to have a strong first-mover advantage in this suddenly vital, novel ecosystem. But its culture and values are very much pulled across from this previous era.

    (Springing out of this, Perl had an, at best grudging, tolerance for 'difficult genius' types, alongside this baseline culture. Unfortunately, this kind of toxic personality tends to thrive in the type of culture I've described, and they do set to help the tone. I'm not here to call out people specifically, because I'm trying to make a point rather than feed a culture war, or dig up gossip, but there were several significant examples, you can probably find lore if you like. I think the kindest way I can describe the compounding effect of this is that there was a strong cultural norm along the lines of "It's OK to be rude, as long as it's for a good cause".)

    A fort within a fort

    I remember this tension as always being tangibly there. Perl IRC and mailing lists were quite cliquey and full of venerated experts and in-jokes, rough on naivety, keen on robust, verbose debate, and a little suspicious of newcomers. And very cult-like. The "TIMTOWTDI" rule, although ostensibly liberal, literally means 'there is more than one way to do it in Perl' - and you can perhaps infer from that that there's little to no reason to do it using anything else. Elevating extreme flexibility like this is paradoxically also an engine of conservatism. If Perl can already do anything, flexibly, in multiple ways, then the language itself doesn't need to change - 'we already have one of those here, we don't need new things'. This attitude determined how Perl intended to handle evolution: the core language would remain stable (a fort inside a fort, only accessible to high level wizards), while innovation was pushed outward to CPAN. You could add features outside of core by writing and consuming third party libraries, you could bend language behaviour with pragmas without modifying Perl itself. The very best CPAN modules could theoretically be promoted into core, allowing the language to evolve conservatively from proven, widely-used features.

    On paper, this sounds reasonable. In practice, I think it encoded a fundamental conflict of interest into the community early on, and set the stage for many of the later growth problems. I'm not going to pretend that Perl invented dependency hell, but I think it turned out to be another one of those profound misfeatures that their cultural philosophy lead them to mistake for virtue, and embrace.

    An interesting thing I think has been missed discussing the context of the original blog piece, about whether Perl 6 significantly impacted Perl growth, is the fact that Perl 6 itself manifested out of ongoing arguments. Perl 6 is a schism. Here's a oft-cited note from Larry Wall himself about the incident that sparked Perl 6, at YAPC OSCON 2000

    We spent the first hour gabbing about all sorts of political and organizational issues of a fairly boring and mundane nature. Partway through, Jon Orwant comes in, and stands there for a few minutes listening, and then he very calmly walks over to the coffee service table in the corner, and there were about 20 of us in the room, and he picks up a coffee mug and throws it against the other wall and he keeps throwing coffee mugs against the other wall, and he says "we are f-ed unless we can come up with something that will excite the community, because everyone's getting bored and going off and doing other things".

    (Pause a second and ask yourself about the sort of social culture that both allows this kind of behaviour at public events, and then chooses to embrace it as a key piece of cultural lore)

    The impact of Perl 6

    Perl 6 was really a schism. Perl was already under a great amount of strain trying to accommodate the modernising influx of post dot-com mainstream web application building, alongside the entrenched conservatism of the core maintainers, and the maintenance burden of a few years exponential growth of third-party libraries, starting to build a fractal mess of slightly differentiating, incompatible approaches of those multiple ways to do things that were effectively now table-stakes language features, as the deployment landscape started to tiptoe towards a more modern, ubiquitous WWW3.

    So, while I agree that it's wrong to generalise that 'Perl 6 killed Perl', I would say that Perl 6 was a symptom of the irreconcilable internal forces that killed Perl. Although, I also intend to go on to point out that Perl isn't dead, nothing has actually killed Perl. Killed Perl is a very stupid way to frame the discussion, but here we are.

    So... Perl 6 is created as a valve to offset that pressure, and it kind of works. Up to a point. Unfortunately I think the side effect really is that the two branches of the culture, in the process of forking, double down on their encoded norms. Perl 5.x beds down as the practical, already solved way to do all the same things, with little need to change. Any requirements for more modern application patterns that are emerging in the broader web development environment, like idk, Unicode, REST clients, strict data structures, asynchronous I/O, whatever? That can either wait for Perl6 or you can pull things together using the CPAN if you want to move right now. Perl 6 leans the other way - they don't need to ship immediately, we have Perl 5 already here for doing things, Perl 6 is going to innovate on everything, and spend its time getting there, designing up-front.4 They spend at least two years writing high level requirement specs. They even spin out a side-project trying to build a universal virtual machine to run all dynamic programming languages that never delivers5

    This is the landscape where Perl's central dominance of 'back end' web programming continues to slip. Unfortunately, alongside the now principled bias toward cultural conservatism, Perl 5 has an explicit excuse for it. The future is over there, and exciting, and meanwhile we're working usefully, and getting paid, and getting stuff done. Kind of OK from inside the fort. Some day we'll move to the newer fort, but right now this is fine. Not very attractive to newcomers though, really. And this is also sort of OK, because Perl doesn't really want those sort of newcomers, does it? The kind that turns up on IRC or forums and asks basic questions about Perl 6 and sadly often gets treated with open contempt.

    Meanwhile, over there

    Ruby has sprouted "Ruby on Rails", and it's taken the dynamic web building world by storm. Rails is a second generation web framework, that's proudly an 'opinionated web framework'. Given that the web application architecture is starting to stabilise into a kind of three-tier system , with a client as a web browser, a middle tier as a monolithic application server, and a persistence layer as a relational database , and a split server architecture serving static and dynamic content from different routes, here is just one way to do that, with hugely developer friendly tooling turning this into a cookie-cutter solution for the 80% core, and a plugin and client-side decoration approach that allows for the necessary per-site customisation.

    Ruby is interesting as well. Ruby is kind of a Perl6 really. More accurately it's a parallel universe Perl5. Ruby comes from Japan, and has developed as an attempt to build something similar to Perl, but it's developed much later, by programming language enthusiasts, and for the first ten years or so, it's mostly only used in Japan. To my line of thinking this is probably important. Ruby does not spring from decades of sysadmin or sysop culture. Ruby is a language for programmers, and is at this point an sensible candidate for building something like Rails with - a relatively blank canvas for dynamic programming, with many of the same qualities as Perl, with less legacy cruft, and more modern niceties, like an integrated object system, exceptions, straightforward data structures. Ruby also has adopted 'friendliness' as a core value, and the culture over there adopts a principled approach to aggressively welcoming newcomers, promoting easiness, and programmer happiness and convenience as strong first class principles.

    Rails is a huge hit. At this point, which is around about the time I stopped significantly using Perl (2004-2005) (because I quit my job, not out of any core animosity toward it, in fact, in my day, I was really quite a Perl fan), Rails is the most appealing place to start as a new web programmer. Adoption rate is high, community is great, velocity of development is well-paced, and there's a lovely , well-lit, onboarding pipeline for how to start. You don't even really need to know ruby. It has a one-shot install tool, and generates working websites from templates, almost out of the box. It's an obvious starting point.

    Perl being Perl, develops several analogue frameworks to Rails, all of them interdependently compatible and incompatible with each other and each other's dependencies, all of them designed to be as customisable and as user configurable as they possibly can be6

    PHP

    There are also the other obvious contenders. PHP has been there all along, and it's almost coming up from entirely the opposite cultural background of Perl. PHP is a users language. It's built to be deployed by copying script files to your home directory, with minimal server side impact or privileges. It's barely designed at all, but it encounters explosive growth all the way through the first (and through into the second) web era, almost entirely because it makes the barrier to onboarding so low as to be non-existent. PHP gets a couple of extra free shots in the arm

    1. Because its architecture is so amenable to shared-server hosting, it is adopted as the primary implementation language of the blogging boom. An entire generation of web developers is born of installing and customising WordPress and text-pattern et. al by installing it directly into your home directory on a rented CPanel host account. It's the go-to answer for 'I'm not a programmer really but how do I get a personal web site'7 This zero gate-keeping approach keeps the PHP stack firmly on the table of 'basic' web programmers all through the history of the web up to the current day.
    2. Because of these initially lightweight deployment targets, PHP scales like little else, mostly because it's execution model leans strongly towards idempotent execution, with each web request tearing up and tearing down the whole environment. In a sense, this is slower than keeping hot state around, but it does lend itself extremely well to shared-nothing horizontal scaling, which as the web user base increases gigantically throughout the 2000s era, is the simplest route to scaling out. Facebook famously, is built in PHP at this point in time.

    Python

    There is of course one other big horse in the race in this era, and it's a particularly interesting one in many ways, certainly when contrasted with Perl. This is of course, Python. Python is a close contemporary of Perl's but once again, its roots are somewhere very different. Python doesn't come from UNIX culture either. Python comes from academia, and programming language culture. It's kind of a forgotten footnote, but Python was originally built for the Amoeba operating system, and its intention was to be a straightforward programming language for scripting this8. The idea was to build a language that could be the 'second programming language' for programmers. Given that this is the 1980s, early 1990s, the programmers would be expected to be mostly using C / C++ ,perhaps Pascal. Python was intended to allow faster development for lighter weight programs or scripting tasks. I suppose the idea was to take something that you might want to build in a shell script, but provide enough high level structured support that you could cleanly build the kind of things that quickly become a problem in shell scripts. So, it emphasises data structures, and scoped variables, and modules, and prioritises making it possible to extend the language with modules. Typical things that experienced programmers would want to use. The language was also designed to be portable between the different platforms programmers would use, running on the desktops of the day, but also on the server. As a consequence, it had a broad standard library of common portable abstractions around standard system features - file-systems, concurrency, time, FFI. For quite a long time, one of python's standard mottoes was 'batteries included'.

    Python never set the world on fire at any particular moment, but it remained committed to a clear evolutionary incremental development, and clean engineering principles. Again, I think the key element here is cultural tone. Python is kind of boring, not trying to be anyone's best language, or even a universal language. Python was always a little fussy, maybe snobby, slightly abstracted away from the real world. It's almost as old as Perl and it just kept incrementally evolving, picking up users, picking up features, slowly broadening the standard library. The first time I saw Python pick up an undeniable mainstream advantage would also have been around the early 2000s, when Google publicly adopted it as one of their house standard languages. Never radical, just calmly evolving in its environs.

    Nature abhors a vacuum

    When I sketch out this landscape, I remain firmly convinced that most of Perl's impedance to continued growth were cultural. Perl's huge moment of relevance in the 90s was because it cross-pollinated two diverging user cultures. Traditional UNIX / database / data-centre maintenance and admin users, and enthusiastic early web builders and scalers. It had a cultural shock phase from extremely rapid growth, the centre couldn't hold, and things slowly fell apart.

    Circling back though, it's time to address the real elephant in the room. Perl manifestly did not die. It's here right now. It's installed I think by default, on almost every single computer I own and operate, without me doing a single thing to make that happen. It's still used every day by millions of people on millions of systems (even if that isn't deliberate). It's still used by many people entirely deliberately for building software, whether that's because they know it and like it and it works, or because they're interfacing with or working on legacy Perl systems (of which there are still many), or maybe they're using it still in its original intentional role - A capable POSIX-native scripting language, with much better performance and a broader feature-set than any shell or awk. I still occasionally break it out myself, for small scripts I would like to use more than once, or as parts of CLI pipelines.

    What I don't do any more is reach for Perl first to make anything new. In my case, it's just because I typically am spoilt for options that are a better fit for most tasks, depending on whatever it is I'm trying to achieve. By the time I came to Perl, (1998-ish), I was already on my third career phase, I had a strong UNIX background, and had already built real things in lisp, java, pascal, visual basic and C++. My attitude to languages was already informed by picking a tool to fit the task at hand. Boy did I love Perl for a few years. The product/market-fit for those early web days was just beautiful. The culture did have too much of the negative tropes I've been pointing at, but that wasn't really a problem personally for me, I'd grown up amongst the BOFHs inside the data centres already, it wasn't too hard for me to assimilate, nor pick up the core principles. I did occasionally bounce off a couple of abrasive characters in the community, but mostly this just kept me loosely coupled, I enjoyed how the language solved the problems I needed solving quickly, I enjoyed the flexibility, and I also enjoyed the way that it made me feel smart, and en-route to my wizard's robes and hat, when i used it to solve harder problems in creative ways, or designed ways around bugs and gremlins. For a good 3-4 years I would have immediately picked it as my favourite language.

    So as I say, I didn't fall out of it with any sense of pique, I just naturally moved to different domains, and picked up tools that best fit. After Amazon, I spent a lot of time concentrating on OS X and audio programming, and that involved a lot of objective C and C++. The scripting tools in that domain were often in ruby, sometimes python. For personal hacking, I picked up lisp again9 (which I'd always enjoyed in school). I dipped in and out of Perl here and there for occasional contract work, but I tended to gravitate more towards larger database stuff, where I typically found C, java and python. The next time I was building web things, it was all Rails and Ruby, and then moving towards the web services / REST / cloud era, the natural fits were go, and of course node and JavaScript or TypeScript. I've always been a polyglot, and I've always been pretty comfortable moving between programming languages. The truth of the matter is, that the majority of programming work is broadly similar, and the specific implementation details of the language you use don't matter all that much, if it's a good fit for the circumstances.

    I can't imagine Perl disappearing entirely in my lifetime. I can remember entire programming environments and languages that are much, much deader than I can ever see Perl becoming.

    • Pascal used to be huge for teaching and also for desktop development in the 8/16 bit era
    • Objective C - only really useful inside the Apple ecosystem, and they're hell bent on phasing it out.
    • Before I got into the Internet, I used to build application software for 16 bit Windows (3.11) which was a vast market, in a mixture of database 4GLs (like PowerBuilder, Gupta/Centura SQLWindows) and Win16 C APIs. This entire universe basically no longer exists, and is fully obsolete. There must be many similar cases.
    • I mean who the hell realistically uses common lisp any more outside of legacy or enthusiast markets? Less people than Perl I'm sure.

    Perl also got to be if not first, then certainly early to dominate a new market paradigm. Plenty of things never manage that. It's hard to see Perl as anything other than an enormous success on these terms. Perl innovated and influenced languages that came after in some truly significant ways.

    • Tightly embedding regular expressions and extending regular expressions (the most commonly used regular expression dialect in other tools is Perl)
    • CPAN, for package/library distribution via the internet, with dependency resolution - and including important concepts like supply chain verification with strong package signatures
    • A huge emphasis on testing, automated test harnesses, and CI. Perl test format (TAP) is also widely found in other CI/harness systems
    • Blending the gap between shell / scripting / and system programming in a single tool. I suppose this is debatable, but the way Perl basically integrated all the fundamental POSIX/libc as native built-ins with broadly the same semantics, but with managed memory and shell conventions was really revolutionary. Before this, most languages I had ever seen broadly tended to sit in one box, afterwards, most languages tended to span across several.
    • Amazing integrated documentation, online, in-tool and also man pages. POD is maybe the most successful ever implementation of literate programming ideas (although most of the real docs don't intertwingle the documentation very much iirc)

    Just these points, and I'm sure there are many others that could be made, are enough of a legacy to be proud of.

    Counterfactuals are stupid (but also fun). If I squint, I can imagine that a Perl with a less reactionary culture, and a healthier acceptance of other ideas and environmental change might have been able to evolve alongside the other tools in the web paradigm shift, and still occupy a more central position in today's development landscape. That's not the Perl we have, though, and that didn't happen. And I'm very confident that without the Perl we did have, the whole of modern software practice would be differently shaped. I do think Perl now lives in a legacy role, with a declining influence, but that's really nothing to feel shame or regret for. Nobody is going to forcibly take Perl away as long as POSIX exists, and so far as I can see, that means forever. In 2025 too, I can see the invisible hand creeping up on some of these other systems I've mentioned. Rust is slowly absorbing C and C++. Ruby (and of course Rails) is clearly in decline, in a way that probably consigns it to become a similar legacy state. From a certain angle, it looks a lot like TypeScript is slowly supplanting Python. I won't be entirely surprised if that happens, although at my age I kind of doubt I'll live to see the day.

    Footnotes

    1 : Meritocracy is a fun word. It was originally coined as a pejorative term to describe a dystopian mechanism by which modern i.e. Western / British society entrenches and justifies an unfair and unequal distribution of privilege

    2 : The UNIX implementer culture, is scientific/academic and fell out of Bell Labs. I guess you could extend this school of thought as a cultural sweep towards building abstracted cloud operations, toward plan 9/ Inferno / go

    3 : Web 2.0 was first defined in 1999 by Darcy DiNucci in a print article , the term didn't become mainstream until it was picked up and promoted by Tim O'Reilly (then owner/operator of perl.com, trivia fans), an astute inside observer of the forces driving web development

    4: Another unfortunate bit of luck here. Right at the point of time that 'agile' starts getting some traction as a more natural way to embrace software development - i.e. iterating in small increments against a changing environment and requirements, Perl 6 decides to do perhaps the most waterfall open source development process ever attempted. . It is fifteen years before Perl 6 ships something resembling a usable programming language.

    5 : The Parrot VM, a lovely quixotic idea, which sadly fizzled out, after even Perl 6 stopped trying to target it. Interestingly enough, both python and ruby both made relatively high profile ports to the JVM that were useful enough to be used for production deploys in certain niches.

    6 : A side effect of this degree of abstraction, is that as well as being very hard to get started, it's easy to fall foul of performance overhead.

    7 : This ubituitious ecosystem of small footprint wordpress custom installs gives birth to the web agency model of commercial website building / small ecommerce sites, which thrives and is suprisingly healthy today. Recent, and slighly optimistic surveys have pitched WordPress as powering over 40% of all websites today. Now this is certainly inflated, but even if the realistic number is half of that, that's still pretty damn healthy.

    8 : It's often repeated that Python was designed as a teaching language, but as far as I know, that's not actually the case. The designer of Python, Guido Van Rossum was previously working on a project that was a intended as training language, called ABC, and many of ABC's syntax and structural features influenced or made their way into Python.

    9 : Common lisp is a better answer to an infinitely flexible 'everything' chainsaw language than perl, IMHO

    posted by cms on
    tagged as

  7. 2018-01-02

    Bought some shoes, in the rain

    posted by cms on
    tagged as

  9. 2025-06-24

    Calm down, not gigs I'm playing, gigs I'm going to.

    Festival season is upon us, I realised, as I found myself watching the Amazon Prime streaming footage of this years Primavera Sound. Primavera seems huge now, and I'm surprised/pleased to notice I'm not really getting any pangs of FOMO. It's fun to dip in and out of the streams though, and the Amazon presenting team / live mixing are endearingly sloppy and untogether. I guess PS has outgrown me, or I've aged out? I think it's them not me, it's only just about recognisable as the buzzy hip live music mash-up I first started going to almost twenty years ago. Maybe I'll go for 2027 (which by my count will be the 25th anniversary year) - or maybe I'll find something a bit smaller and leaner and odder. I've been eyeing OFF in Poland for a while...

    Live music has changed so much recently. I was gently flabbergasted when the Beta Band, a band I used to gently follow around the country, announced a reunion cash-in tour, and sold out the early access, and the general access tickets in about five minutes flat, while I sat there bewildered trying to figure out which Ticketmaster entry-point seemed the least likely to meltdown while I was using it. This being a band I regularly used to pay a fiver on the door to see play in a half-empty student union bar, back when they were hot stuff 🤣.

    That appears to be the way things are nowadays. You have to pre-register for the pre-register, and get a code from pre-ordering some merch, and know how to game the electronic queues. And then you have to remember which one of sixteen different apps (all owned by Ticketmaster) your electronic tickets are squirrelled away in them, when you finally arrive at the show dates, up to a year later. That's if you remembered to put in your diary, and didn't just forget about it entirely. Anyway, I'm not too sad about missing the Betas, might as well let some other people have a chance - although I expect I'd have had all the nostalgia, I'm quite sure they'll do the festivals in another year or two after this level of interest. While I was in Ticketmaster, failing to get Beta-d, I saw some St. Vincent tickets for Somerset house in July, on sale before any announcement, so I picked up a couple of those instead.

    I have had a bit more success with some of my other pre sales. Upcoming I have

    I think that's it so far. I might have forgotten some, as I said. At my age I need to write things down. Like this.

    Two things all of these have in common - I had to buy all of them months and months in advance using a 'pre-sale code', and, somehow they're all female artists. I don't think those two facts are related. It's fairly noticeable that women artists really dominate popular music these days. I cynically suspect this might be related to the fact that all the money fell out of it a couple of decades back. The boys have all moved on to being startup founders.

    I'm not going to see Oasis, but I will be tracking the integrity of their reunion over at my hand made personal oasis bust-up tracker https://haveoasissplitupyet.lol

    My children are going to Glastonbury without me 😬. Most likely, I will watch some of it on the TV.

    Oh, I am going to Latitude again, although I'm so spectacularly uninterested in almost all of the musical acts playing, I hardly count it as a gig. I expect I'll be down the front for Alison Moyet though.

    posted by cms on
    tagged as

  10. 2017-12-31

    I've been writing a couple of things in hy again this week. What's Hy? It's a cute idea. It's a lisp that compiles? (transpiles? I never get the difference) to the Python AST. I guess the elevator pitch might be something like clojure but for python. So yeah, a rich, super stable class-tree sort of OO language, with enormous portablility and twenty-odd years of library support for everything you might want to do, but with a nice, dynamic, lispy language and a repl.

    I've played with hy a little bit on and off over the years. Actually, when I was working at SMR, I actually deployed some in production. (Somehow, I doubt that's still a thing). Python is my go-to scripting language, because it's very plain, very portable, batteries included, somewhat modern, probably already installed everywhere I work. I try to use it for scripty things, rather than shell or perl or something. Lisps are my favourite programming language. I just like how it fits together. I know lots of people don't, and I'm fine with that, but I always enjoy it.

    So over the holiday weekend I found myself wanting a couple of almost throwaway scripts, and I decided to reach back into the hy bucket, and give that another try. I wrote a script to grab my selfie tweets from a twitter archive, and a rough script to publish formatted micro-blog entries directly from the shell.

    It was a fun exercise. Hy has moved on a bit since I last tried. (They seem to have removed let, and car, and cdr, and lambda which I feel funny about), but by and large it works really well.

    Things I like

    • I like mapping over lists of things, and in straight Python this is often clumsy and leads to densely nested comprehensions
    • actual lambdas!
    • Python3 support ( hy3 )
    • easy importing of python modules
    • mostly seamless python interop
    • repl works great
    • the repl shows you the pythonic syntax of the forms it evaluates, which is helpful if you know Python
    • emacs mode (obv)
    • it has lazy sequences
    • and multimethods!
    • it is fun to work in

    Things I like less

    • Missing some olde lisp things like car/cdr/lambda
    • Things often expect you to be using methods on stateful objects, which gets you an OO impedance mismatch (I have the same problems in scala and clojure)
    • Slightly more typed than you expect, whilst not really offering you a type system. (Particularly with distinctions between lists, sequences, iterators.)
    • it often seems easier to imperative loop with for than map / reduce / filters, and this seems weird.
    • i don't feel I have any understanding about setv variable scoping.
    • no STM, which I think is one of the most interesting things about clojure
    • I don't think the error handling does restarts and conditions and things

    Summary

    I don't think I would choose to use it to build any complicated systems. (Typically this is true of Python as well to be fair). I'd love to see something like an idomatic web framework in it. I could imagine using it to build serverless workers over something like apex up or chalice perhaps. I should totally try that!

    I am not really very good at it yet, so I doubt I'm writing optimal programs. My scripts often look like Dr. Moreau designs halfway between a python script and something more lispy. This could well improve as I understand the underlying sequence / itertools glue a bit more, I'm often routing around confusing sequenced things. I absolutely enjoy writing little scripts like this in it, and I think I maybe enjoy it more than I would if I was writing plain python. I gave some thought about why this might be and I think I figured it out.

    It could just be as simple as being all about the code editing. Python, and it's whitespace delimited blocks, is fine, and super readable, but it's always slightly fiddly to edit. Some of this is my toolchain, I'm sure. There's a lot of bells and whistles you can glue over emacs for Python work, and they're pretty good, but I do always find it a slightly fiddly experience. Balanced expressions and sexprs though are obviously an absolute joy to edit in emacs, alongside an embedded inferior lisp repl, and although it's nowhere near as integrated an experience as using slime with a "real" lisp, it's closer to that than editing Python ever feels, and for me that's a significant productivity win. So I think it will stay in the toolbox.

    I recommend Hy to anyone who is interested in interesting lightweight languages, especially scripting languages. Obviously it's particularly relevant to anyone who likes python or lisps, even if just as a curiosity. If you work with Python and like using emacs though, and like the sound of 'Python but with structured editing' I would strongly recommend you look at how it might integrate into your workflow.

    posted by cms on
    tagged as

  12. 2017-12-30

    I'm sick of Twitter, folks. I've decided to do something both mild and drastic about it. For 2018, I have resolved to stop using it.

    I am not sure what it is for anymore, it certainly doesn't feel like it is for me. I think I've been disengaging slowly for the last couple of years, and in 2017 I repeatedly found it too aggravating, and depressing to engage with. I think I would have already ragequit, had one of last year's resolutions not been that silly selfie thing. Thus a seed was planted about resolutions and exits. Brains often work that way. (Referendums are silly though)

    I was late to twitter. I downloaded my twitter archive, whilst I was scraping out all of the 2017 selfies, and apparently my first tweet is from Dec 2007.

    I was late to Battlestar Galactica as well.

    I probably spent a little while reading twitter before registering, although I don't remember anything specific. I can't remember why I signed up in the first place. Looking at that first month of odd, stilted entirely quotidian status posts, I can tell I'm working on Logical Bee, mostly alone, babysitting that dog. It's winter. Maybe I'm lonely? I have a dim memory of thinking it was pretty dumb for a long while before getting involved at all. I remember fiddling about connecting it to things, and experimenting with SMS tweets and emails. I don't think it really clicked for the longest while. I remember a sense of a clique I wasn't ever going to be able to get into. That first wave of web-natives, younger than my generation. More entuned to a web of application services and APIs than hypertexts and data servers. I remember tweetups being a thing, and a Bristol one being announced, and spending an hour or two before deciding firmly I wasn't the kind of person that went to that kind of thing. I quite wish I had gone now. I didn't used to be a very good joiner-in of things. I'm not much better at that now. A little bit, perhaps. Now I know to try.

    It took the longest while, but eventually it clicked. I liked the lightness of it. It was sort-of social networking, but social networking at arms length. Lots of irony, lots of whimsy. I just remembered the earliest phase of my binning Facebook was to convert my facebook to just echo my tweets back into it, for the muggles to read. I remember being very snobby and standoffish about things like hashtags and @replies. My first reply wasn't until August 2008.

    To Daveh! Either I don't know how to reply yet, or the Twitter archive has incorrectly threaded that reply back together. Either seems plausible.

    I didn't use a hashtag until May 2009. Even then I was repurposing "get off my lawn" meta-commentary. Amused to see that my next half dozen hashtags are complaining about moonfruit's use of them for viral marketing. Many years later I ended up working there for a season. Again we see the seeds are sown, and the fruit is reaped.

    Not too ashamed of that one. It's interesting looking back at tweets like that, I have a sense that the prevailing vibe of Twitter at the time was that the cool kids were beating out the idiots. I don't get that vibe off Twitter now.

    By this point it was clearly very firmly entrenched in my daily desktop routine. Once I got hold of smartphones that could run twitter, I think my usage ramped up. I remember by the time I got to last.fm, I was tweeting all the things, curating a couple of hashtags (#fantasypeelsessions for serendipitous word groups that sounded like band names, #fisharecool for cool fish facts), running multiple joke twitter accounts, writing bots, and generally really enjoying it. I remember when I got to Makeshift, and twitter seemed to be used as the wiring behind at least half of everything there, it then seemed like a necessary internet plumbing for web apps. With hindsight I think that was the peak. It was downhill from there. I don't like it any more, I have detected an opportune moment, and I have decided to leave. At least for one year.

    I'm not going to use this post for arguing about why I think it's broken. One of the largest problems I have with it is the sheer concentration of negativity. And one of the reasons I want to move away from it is to focus on building things that are more positive. It's not just Twitter. I'm pretty broken-hearted with the state of the web in 2017 - it's very far from what I signed on to help build as one of those idealistic Gen X web 1.0 types. And again, rather than just bemoan that, I'd rather start focusing on ways to think about fixing that. And for me, in 2018, this means I'm going to go small, and focus on building things and content I can own, in the sidelines. I expect I will be updating here more. I plan to double-down a bit harder on indieweb things, and federated stuff. POSSE all the things. Death to silos. I've been experimenting with micro.blogs and mastodon.social, and I want to play more with beaker and dat, and blockstack and IPFS and other idealistic p2p proto-webs. Maybe even frogans?. The real web looks more like that. Maybe I can help figure out how to make it a bit easier for everyone to clamber onboard.

    "But CMS, I think we're Twitter-friends, what does this mean for US?"

    First off, that's flattering, almost-certainly-entirely-imaginary-cms-fan, thanks! I like you too! Occasionally some of my tweets get as many as five or six engagements, and I do enjoy keeping up with some lovely people. Some of whom I met or perhaps only know through twitter. I'm sorry if this feels like a breakup; It's not you, it's me, as they say in the rom-coms. (Actually, I'm not dumping anyone.)

    Something else I want to push for in 2018 is better quality, stronger, social engagement. I want to cultivate more real contact, more high bandwidth engagement and connection with all the good people. This can work two ways of course. If you only really interact with me on a tweet by tweet basis, and you think you're going to miss that, then do please reach out. We can have coffee, or get beers, or just go fish in a lake or something else entirely. And I'm going to be pushing myself to reach out to more people in turn myself, something I'm astronomically poor at. Please help me with this if you can!

    IRL networking I plan to ramp up a bit. More meetups, tech and maybe otherwise. Maybe I'll rescind my conference ban. Maybe I'll start some of these things, or start helping to organise them more.

    I'm not doing an *infocide*. As well as publishing things hanging from here, which has plenty of RSS feeds, if you can still figure out how to integrate those into your workflows then I'll probably never be very far away. Also, if you look at the home page, there's a list of dozens of other not-Twitter platforms you can stalk me on or connect to me via (maybe we are already!) - If my plan comes together, I hope to be syndicating and updating the useful ones of these more actively.

    I don't intend to delete or remove my twitter account, and I will set things up so I still get notifications, so nobody gets ignored. I might even automate some notifications to my twitter feed about updates to things elsewhere. I'm just not going to be participating as a human. I expect I will remove all the apps, so my turnaround on mentions might slow right down.

    If you're in the select category of people who only know how to contact me with twitter, there are many options. I haven't changed my phone number, should you know me well enough to have one of those. If you're looking for a way to DM to me, I cannot endorse keybase strongly enough. I think they're trying to do something really interesting, and could do with some more network effect. Sign up to keybase, and keybase message me, I love getting keybase messages, and I always respond. Invite me to your keybase groups! Also, please share your slacks and your newsletters and your mailing lists with me, if you think I'd like them, or they'd like me.

    Email still works, and I still read it. My address is even on my website.

    Finally, if you're reading this, and we've Twitter interacted in some way, let me say a goodbye for now. If I was annoying, or argumentative, I'm sorry, I can be hard work soemtimes. Maybe some of that might have been caused by the platform? If I was fun or charming or interesting, then let's work to stay in touch! If you don't really care, you're not even sure how you got here from off of twitter, that's cool too, maybe I'll see you again in a year from now.

    posted by cms on
    tagged as

  13. 2017-12-30

    With all this focus on RSS generation for micro blog, I've been optimising my engine. I've learned how to use SBCL's profiler, and I have shaved a third off the cost of generating indexes

    posted by cms on
    tagged as

  14. 2017-12-29

    To think I used to worry about Disney Princesses

    posted by cms on
    tagged as

  15. 2017-12-28

    Let sleeping dogs lie

    It's been a month now, and I ought to be used to it, and in many ways I am, but in surprisingly many ways I'm still not; I don't have a dog anymore. He got too old, and he got too sick, and tired, and uncomfortable, and he had to be put to sleep, back on the 28th of November. How does it feel? Terrible.

    It was an enlarged heart that did for him. Poetically enough, his heart was just too large for him to carry on. The photo above is taken on the last morning, before I headed out to work. I knew there was very little chance he'd be coming back from the vet's appointment later that day. We had a little conversation and I carefully explained to him that he was a very good dog.

    Of course he was actually a terrible dog. A brilliantly terrible one, as most dalmatians are born to be. He'd not really been himself for a couple of years, stumbling about and complaining about most things, but right up until the last couple of weeks he was coping mostly, and remained good company. In his prime though, that dog was an athlete, who used to literally fly, and if I open my mind's eye a little, that's what I can see, streaking around the Bristol countryside, barely controllable, raiding bins, and laughing at you, over his shoulder.

    I don't really know what to write. I have to write something though. This website, which has been knocking around for fifteen years or more, only really took initial form as a rudimentary 'blog' so I could share dog photos with his burgeoning fanbase. Most of that has bitrotted now, but when I feel better I would like to clean it up some. So I can't really even let go of him without marking some notice here. I don't need to trot out all of the anecdotes, they're probably dull and too personal. After all, outside of my immediate circles, he's just some bloke on the internet's dog. To me, and to some of his internet fans though, he's the best dog in the world. Every single word of that is true.

    posted by cms on
    tagged as

  17. 2017-12-27

    When you have, as I have, a race condition in posting that exists somewhere between systemd, rsync, bash, perl, and that's before you even get to the CMS, it is probably time for some refactoring

    posted by cms on
    tagged as

  18. 2017-12-27

    If I included an image maybe it would look like An image

    posted by cms on
    tagged as

  20. 2017-12-27

    Shopkins movie, and perhaps a curry? I feel like death...

    posted by cms on
    tagged as

  21. 2025-06-04

    I Asked AI to Rebase My Git Branch and Accidentally Discovered the Future (and Uncovered the Past)

    One of the things I initially missed most, when shifting from emacs to zed was magit, which has been my main git interface for IDK, let's say about 20 years 🤷.

    I'm a bit of a sloppy worker, but a meticulous git user, lots of atomic annotated commits, many linear branches, I'm cherry picking and rebasing, and stashing, and magit integrates all of this naturally into the normal flow.

    Zed has git integration, obviously, but it's entirely basic - pull, push, commit, branch, diff, that's about it. It does let you stage incrementally, which is most of the problem solved, and of course, there is a terminal integration, so you can pop a shell and use git CLI (like a farmer 🤪).

    It's OK, but clunky, obviously. A little bit of friction and dissonance in this slick world of modern native editors that were actually made this century. I think the kids are a lot looser with git than I am, and perhaps they have a point, but I like the discipline.

    Here comes the machine mind

    You probably figured out what's coming next, if you bothered to read this far.

    I've been using the zed LLM integration to write commit messages for a while - it's pretty well suited to that, given a bit of context, you can generate a draft commit message that summarises the changes, and it's you tweak and approve them before you apply. It's a pretty good example of the kind of low-hanging, small improvement, you can achieve with even simple model, precisely applied to a narrow context that involves generating prose. Smoothing out busywork.

    Obviously, zed is pretty agentic 🤢, because that stupid word is all the rage these days. I guess you can open a chat box and ask your editor to vibe code your whole application. (Good luck with that, if you do, I think that's liable to create more work in integration than it saves in writing, maybe that's just me)

    I use the agents for a bit of boilerplate here and there - refactor this, replace these magic numbers with proper constants, redo this part to use an iterator, what is the type checker complaining about here, how do I configure the language server to disable a misfeature, again, it's not too bad at doing think kind of drone effort, and there is a small but appreciable productivity gain to be had.

    I cross the streams, and surprise myself

    This last week, I was suddenly inspired to cross the streams, and something interesting happened, surprising me enough to bother drafting a post on the topic. I don't really like to thought-leader, but occasionally something will delight me enough to want to share.

    I wanted to tidy up a messy WIP branch that collected a couple of different ideas in progress, (and had also coincided with me correctly figuring out how to enable auto-linting in zed, so I suddenly had a lot of aesthetic formatting corrections dropped suddenly into an already untidy sandbox)

    A minute or two fiddling with rebase in magit, but in zed...? Time to roll up my sleeves, and flex, and take out the ol' git pitchfork, or hitch up the rebase propagator to the reflog tractor (I do not really know what farmers do). But, wait. I wonder if...

    So I pull up the agent and ask it "can you run an interactive rebase on this branch please, and group all the white-space only changes into one commit, the other formatting changes into another, and then separate the removal of the obsolete class from the other feature work?"

    And, it kind of worked! It got stuck a couple of times, and I had to pop in and edit a couple of things, and I restarted it over with the order of the commits I wanted a little more explicitly instructed once I recognised where the conflicts were going to land, but I got the result I intended, certainly with no more fiddling about than I would have had to do if I'd been performing the task manually, maybe less? It's hard to measure, but I enjoyed the experience.

    One thing I definitely realised. It was less irritating. I was able to complete a disruptive, yet necessary chore, while in the middle of doing something more interesting, with much less context switching than performing it manually. And that got me thinking about interfaces a bit more. Chat bots are unquestionably a very ergonomic interface.

    Increasingly, I am starting to think that a key part of unlocking the value of LLMs, may be through thinking about them more as solutions for interface design problems.

    Considering git interfaces

    Git is a classic case. Git's interface sucks space balls, everyone knows it. I mean I know a few people who like it, and I'm happy for them (but I think they are weirdos). It does allow for a bunch of studly machismo and nerd flexing, for anyone who has invested enough time in learning its arcana to impress people with stunt-git trick shots, and that can be fun. I have definitely enjoyed being the knuckle-cracking "stand back everyone and stop panicking, I know how to fix this" guy on a number of occasions, but that is a sideshow. You can do circus tricks with power tools, and some people do, but it's not the reason the tools were made.

    Git has a compelling storage model, and a commit graph workflow that solves a bunch of annoying challenges with incremental and concurrent code editing and integration, efficiently and better than previous source code version systems. That's why it became a huge success.

    Git's horrible ergonomics and implicit barriers to entry, but compelling powers of sharing and integration, allowed GitHub to spring into existence, as a multi trillion zillion company from out of nowhere, just by slapping a nicer set of user abstractions on top of git's ugly robotic core. (in the process accidentally inventing some other significant ergonomic problems, like pull request based workflows, and IDK, tag driven releases, but I guess that's a different blog post)

    Git's foul ergonomics are what pulled me into learning magit, which has a reasonably steep learning curve of its own, but also follows the emacs way of having a lovely manual. It's a better thought out UI. It also leverages many common emacs behaviours, so when you're working in emacs already, which I typically was, again, you get this reduction of context switching.

    The mythical 'Flow State'

    Why is this so important? At this point, it's tempting to dive off into a side bar about "programmer flow state", a long held shibboleth of the developer community about which I don't have much truck, but many thousands of words can be found about it on the web already. - I don't like anything that reinforces programmer identity as a higher state of being, and I dislike how easily this concept is weaponised towards shunning collaboration and social work (e.g. "coders must not be interrupted during holy flow state") , again this is clearly a different adjacent blog post - but the notion is not fundamentally baseless.

    Programming tasks often require holding a lot of accrued context about a chain of thought, and carefully expressing those in a narrow, precise domain, incrementally progressing towards a well defined future state. It's not easy for a human mind to do that, it takes a bit of effort. Effortlessly breaking out of one domain into another mode of expression isn't really possible. To do that even passably well, perhaps you would need a different kind of "mind", even?

    I think version control is interesting here, precisely because it's liminal stuff. It's programming-adjacent work in a certain sense, it's a chore - any time you have to break thread to address some version control nonsense or busy work, you are in essence interrupting yourself. It makes a lot of sense to try and find a low friction simplified user interface to mediate these kind of tasks. Like GitHub, or magit. The ideal is to minimise the amount of disruption you face while working on these background or side channel threads of work. You can mitigate against this in two main ways I think. You can look for ways to simplify the UI to better suit a particular working context, as we have been discussing; alternatively you can divide the work and make the secondary context a first class task that's managed separately.

    One way to do this is to structure the way you work so you can plan your version control stories a little ahead of time, and use discipline in your task management to make it better fit onto the VC, with strategies like formal branching and ticketing protocols, rigorous task mapping that accounts for tech debt, probably integrated into a project management system.

    Another way you can do it is to make it literally someone's job and push the load out sideways - examples of this might be code review protocols, gatekeepers for merges (in the olden times, with less sophisticated version control software, I've often worked on teams where there was a nominated 'merge master', whose entire job, or a large portion of it would be to basically do the integration and harder version control work on behalf of the feature developers), also other adjacent roles like scrum masters, or DBAs.

    How about SQL ?

    That train of thought got me thinking about SQL. I think SQL is another interesting example of 'programming-adjacent work', although it might be a bit more subtle of an example than it first appears. Let's have another digression then. I really like SQL, although it's obviously covered in warts and sharp edges, I've always appreciated it's utility, and to a certain extent it's ergonomics. It does share some of the properties I've discussed with git - it's very often a boundary, and context shift away from the main thread of programming, and programmers tend to hate it, and like to avoid it, and make up nasty memes about it for slack, and that kind of thing - just like version control (or meetings 😘), it's essential and necessary work in a lot of software development, but it's another liminal place, where you have to get pulled out of the context of thinking about your feature work, and software architecture, and land in another place for a while with an annoying external syntax, and a lot of aggravating round trips into different tooling.

    Indeed, over the years, a very common programming pattern is to try and slap a more program-ergonomic abstraction layer in front of the SQL, once again to try and minimise the friction and narrow the interface - I'm thinking of things like various ORMs, 'noSQL' database engines that bring the data modelling and querying closer to the application layer, all of them moderately successful, and yet SQL still hangs around everywhere, slightly annoying everyone, like a remote senior cousin inevitably invited to every wide social gathering, tolerated, rather than enthusiastically invited.

    That's because SQL is already an ergonomic abstraction. It's kind of the ur-DSL. SQL is there because databases have a lot of inertia associated with them. All the data is often where the money is. Data is the raw stockpile of materials, the raw ingredients of the information that's necessary to run large information technology applications. Data tends to accrue value cumulatively, and you want to keep it all in a big lump in one place (this is why we have terms like 'data-mining' and 'data-warehousing'), so you can correlate it, and leverage sexy network effects from having it all integrated into one humongous data domain. Once you pass a certain critical data mass, you need to access it multi-modally , i.e. there will be many different use cases for the same information sets, different users and different applications will emerge from, or require access to, various intersectional pieces of that data blob. Now, reading and updating that data blob by hand, in your preferred programming stack would really sting. You'd still have to leave your application software context, but you'd now need to delve into a world of low level file systems, and data packing, and indexed data access, and write locking, and concurrent editing, and wire protocols, and cache invalidation, and the whole nine yards of that side of computer science.

    I pause a little here, because I realise I'm probably making it sound kind of fun to a particular audience segment (amongst which I include myself, periodically), but let's not lose track of the core point. If your intended task is to make a cool dating app that helps your users get laid, low level storage systems coding is a horrible, high effort context switch away from the feature track you're working on. And of course, all this data access code you're having to do while you go will also need tracking in your version control system, more context switches. Instead, we SQL.

    LLMs suck at SQL though

    As an aside, I have found LLM coding assistants to be generally pretty bad at SQL writing. I have two hand-wavy personal theories about why this is the case

    1. Programmers, on average, in my experience are really pretty bad at SQL, probably because of the reasons we delved into above. So the training set of the models is full of low quality material. (cheap shot, maybe? 😅 Cut me some slack, I've spent many hours of my life fixing other people's bad SQL)
    2. Effective SQL generation requires a lot of external context the model doesn't have - not only the entire database schema, but also ideas about the data contents, and distributions that a pre-trained model doesn't necessarily have any access to. I think getting good SQL results from LLMs would need a lot of context prompting, or specific training. Still doesn't entirely explain why they're so bad at basic join syntax.

    I'm tempted to infer something from those two sub-points about SQL requiring fundamentally different kinds of reasoning to other forms of writing, but I'm probably just seeing the face of Poseidon appearing in the patterns of my mental sea foam... I'll leave it there for now. (a third parallel blog post? This stuff is getting fractal). BTW - The name for that fascinating phenomenon is *pareidolia* , and it's something worth keeping in mind when discussing "AI" concepts...

    The surprising invention and nature of SQL

    Ahem... Another really interesting aside is to have a look back at the history of SQL. SQL is rather old. It's basically my age, and I've already pointed out I've been using emacs professionally for at least a few decades. SQL emerges from IBM in the 1970s, as a research project, greatly influenced by E.F. Codd's classic article 'A relational model of data for large shared data banks'

    The primary designers of SQL were Don Chamberlin and Ray Boyce, part of IBM's system R research project, who were tasked with looking into ways to apply Codd's relational / mathematical principles of database modelling to IBM's database businesses. IBM's database business at this point was most of IBM's business, and IBM's database business was pretty huge. Prior to relational databases, the existing big iron database management systems were awkward weird transactional / hierarchical databases, like IMS/360 where you pretty much had to write a specific computer program to be batch executed from a queued transaction management system. Each 'query' was more akin to an independent program. In order to change the report, you'd develop a new program, and you would need appropriate programmer time and skill to do it, and your best turnaround for results would be several hours, probably more like days.

    So the system R researchers wanted to make this more flexible, but their ambitions didn't end there. Chamberlin and Boyce wanted to make information retrieval accessible to non-programmers. Here's Chamberlin

    "Ray and I hoped to design a relational language based on concepts that would be familiar to a wider population of users. We also hoped to extend the language to encompass database updates and administrative tasks such as the creation of new tables and views, which had traditionally been outside the scope of a query language.[...] What we thought we were doing was making it possible for non-programmers to interact with databases. We thought that this was going to open up access to data to a whole new class of people who could do things that were never possible before because they didn’t know how to program."

    Can you see where I'm heading? SQL is a frantically successful example of what we used to call '4GLs', (fourth generation languages), when I was a school kid, (although by that point, the text books - and what we didn't yet call the hype cycle - were already breathlessly excited by the imminent arrival of the Fifth Generation Languages, and systems...). The terminology is dated, and stretchy, and marketing-fed, but the central gist is - 4GLs are languages that were operating at a higher abstraction level. Your inputs and controls would describe a program at an abstraction level much higher than the operation of the system, and the 4GL would write a program for you that ran at the lower level. All a bit hand-wavy, but SQL querying has some really interesting properties related to this delegation.

    • it's declarative. Your query describes the data structures you want to retrieve, and the actual details of how the data is retrieved from disk is decided for you by a query planner.
    • it's live and interactive - you can interrogate the system interactively at a REPL
    • it's got a weird-as-hell syntax full of special cases that tries valiantly to use ENGLISH LIKE words, all IN UPPER CASE SHOUTING that deal in terms of database structures and relational operators, not computer terms like bytes and loops and sorts.

    The query planner is worth a little thought - The query plan uses a bit of maths, and some heuristics and a bunch of information and sampled data about your system, and works out a series of reads and sorts and filters that produce the data structures your query is requesting. Crucially you don't tell it HOW to do it, just WHAT you want it to do. (sorry, the upper case is a bit addicting, I'll stop). Most of the time you don't think about it too much more than that. However, most SQL database systems will show you their plans if you ask them, typically by using the EXPLAIN keyword, which will show you what the planner thinks it should do to build the result set you wanted. If you don't like what it's decided you can't tell it to do things differently, but you may teach it to do things differently, typically by updating the available indexes and constraints, or maybe by re-balancing the statistics it uses to decide about cardinality and seek times, and that kind of thing.

    Here again we have a division of labour - the idea is that the structural and statistical and optimising and runtime bits of maintaining the SQL system can be delegated to the programmer and technician classes, who can be more concerned with the implementation and operational parts, and the query writer (a non programmer, if you remember) can just get on with expressing their tasks in a lower friction, narrowed domain, where they don't have to context switch out as hard from their task at hand, writing lovely business reports for the sales and finance teams to make slide decks from.

    Now I'm not saying that SQL writing is 1970s prompt engineering, but I'm also not not saying that, right?

    Here's Don again, after the fact

    "Ray and I were wrong about the predominant usage of SQL. Typically, SQL is embedded in a host programming language and used by professional programmers"

    Is SQL any good though?

    SQL was, as I have said, a spectacular success. It's still bloody everywhere, fifty years on. It was also an abject failure. The syntax is a mess of unaligned clauses with special cases everywhere (INSERT and UPDATE have such radically different approaches to clauses but sort of do the same thing. So does DELETE really). You need to understand maths to do it properly. You also need to understand the precise details of the database schema. The database schema is maintained in a separate dialect that's somehow intertwined with the querying DSL and uses the same interfaces, but again obeys a different syntax and semantics. Record locking is both implicit and explicit. The concurrency model is insane and nobody actually understands it properly. NULL breaks everything including query logic. And most damningly, it's never used as an interactive REPL by non-programmers. It's folded into programs mostly. In fact these days, an eye-wateringly huge number of applications bundle an entire SQLite embedded RDBMS inside their application deployment.

    Repeated interface patterns, and why I care

    There's a pattern emerging here I think - querying and reporting wasn't quite entirely programming adjacent busywork, but it was business-adjacent drone work, and SQL was an attempt to narrow the interface with an ergonomic DSL that got out of the user's way and reduced the scope of the context switch needed to engage with the reporting system. It kind of failed at the interface, if you ask me (and most folks who have to use it), but it did succeed amazingly at reducing the complexity of the context switch. Using SQL to mediate data persistence inside your application is kind of like using magit to fold your git operations right inside your emacs workflow, they both serve to shrink down the cost of flipping out of your primary task domain, into some other essential, dependent domain.

    Either reduce it to a tight DSL, or extract the work and organise it so it can be delegated to another worker. Sometimes, you put a DSL on top of a DSL, to tune it down even further. Sometimes you build a team to own the work in this domain. What if you build a DSL over a DSL but the DSL could sort of work like a team you can delegate tasks to? That's a compelling interface, that is.

    When I look at it like this, using an agent REPL in zed to run git operations strongly reminds me of that (failed) SQL/4GL promise. I tell the thing the result I want from it, and it builds a plan of execution, and I can iterate on that, interactively. Only this time, I'm using literal English sentences to describe the narrow domain the system has trained itself on, not some bastard awful pidgin version of it, that somehow ends up with the worst features of both natural languages, and programming languages. Sorry SQL, it has to be said. And you know what, I kind of love you anyway.

    I think this pattern can be found in several places of software development, if you squint right. Programming-adjacent work gets reduced with DSLs or narrow abstractions, which bring the benefits of reducing that tedious, expensive context switch. I guess we have lots of it in CI automation, pipeline building, that kind of thing. DevOps is built out of this stuff, infrastructure as code, deployment charts, meta deployment charts, run-books, playbooks. These domains are also places I've found LLM-based assistants super helpful - help me grind out some yaml please so I can add a pull request pipeline that does this thing I just thought of, without me having to spend quite so much time reading up the stupid YAML syntax for this weeks CI system, and spending hours sitting in a push/fail/edit/push loop on some git forge. GraphQL over REST apis - maybe? Unit test generation and test harness design is mostly busywork in a DSL following some declarable constraints. I think I already mentioned figuring out the precise type annotations for things. It makes me think that coding-assistants are perhaps more of a user interface paradigm than they are a coding one. More like a 4GL than a semantic IDE.

    In conclusion

    What's my point? I'm not completely sure (he says after several thousand words)

    • I like zed, I'm finding it more useful than I thought I would, and I've stuck using it on and off for a few months now.
    • I keep finding small useful things for LLMs to do, and I enjoy that process.
    • The most value might be in 'programming-adjacent' things - e.g. as I mentioned, I'm now personally only writing a small proportion of my commit messages by hand, and I think my commit messages are considerably better off for it.
    • Tools that reduce the context switches for 'programming-adjacent' work will win
    • Conversational interfaces are very low friction for human minds.
    • LLMs are much better than humans at context switching to a different domain, while keeping track of and applying accrued context across different tasks
    • Is there something you need to do that applies a tight but boring DSL across a defined data set? A model might actually be quite good at that. Another weird thing I've found them almost spectacularly good at is networking and debugging - describe a topology, give it a tcpdump, and watch it spit out a bunch of diagnostic suggestions and potential remedies for you to use.
    • These tools are one of the biggest tech shifts I've seen in my rather long, slightly storied, career that spans a whole bunch of tech paradigm shifts.

    Summary footnotes

    1. This is just some opinions, inspired by how astonished I was to successfully use an "agentic" tool to run some gnarly git stuff I would have had to pull out the manual to do. There's all sorts of important discourse about LLMs and the out of control tech hype cycle around them that I don't go near, and I'm not writing a manifesto
    2. I'm not a historian, or a first hand witness of the pre-RDBMS data scene, although I did work alongside people who came from that background, so I do have a lot of secondary source exposure. I haven't done much research other than light web searching, so please take my historical characterisations with the appropriate amount of salt. I know pretty much zilch about how IMS/360 actually worked, I just remember the name, I'm generalising.
    3. I wrote this blog post by hand, it's not the sort of thing I think LLMs are much use at. I'm trying to express my own opinions here, in my own voice, because I was excited about a couple of ideas and wanted to try sharing them
    4. I wrote this blog post in emacs though.
    5. I probably won't write any of those other parallel blog posts, who has time to write blog posts?
    6. I did use Claude to fact check the post. Don't laugh, they can actually do that sort of thing now. This stuff moves fast.
    7. Finally, I feel I ought to note that Ray Boyce died tragically young at 27 in 1974, shortly after the presentation of the SEQUEL design paper. He left an astonishingly outsize impact after such a short career. Put a dent in the world, as they say. Don Chamberlin, happily, is still with us so far as I know.
    posted by cms on
    tagged as

  22. 2017-12-27

    Crash early, crash often

    posted by cms on
    tagged as

  23. 2017-12-27

    Does anyone know if eating your own bodyweight in Stilton is a good way to clear up Xmas flu?

    posted by cms on
    tagged as