3. 2010-12-06

    One year ago




    posted by cms on
    tagged as

  4. 2010-12-05

    I've been having persistent niggles with my home router / 802.11x base station / DSL modem. It's a D-Link DSL-2740B , itself bought as a replacement for my ISP-provided machine, an O2 wireless III (a re-badged Thomson SpeedTouch) which proved itself a low performer at both wireless and routing, and particularly dismal at doing both simultaneously.

    I picked up the D-link cheaply, in a clearance bin in John Lewis. In most respects it has been a splendid replacement for the O2. WiFi is fast, routing is consistent, ADSL sync is better. However, it does have one stupid bug. It can't do DHCP reliably. After a certain period, it starts sending out broken leases to clients; either issuing them with IP addresses that are already in use, or more commonly issuing a working address, but nullifying the nameserver settings. A reboot will restore sanity, but involves an irksome couple of minutes of network outage. Afterwards it is only a matter of time before the problem re-emerges, noticeably quicker if there's an increased rate of new leases issued, such as a group of visitors armed with smartphones popping in.

    I'm consistently amazed at how flawed home router appliances are. How anyone 'normal' is supposed to cope with these things, I have no idea. I've updated the firmware to the last available revision, fiddled with the limited options in the admin interface, to little avail. Web searches turn up a few people commenting on the same problem, but no solutions offered. This leaves me with three straightforward, yet unappealing options.

    1. Buy another router. Either another toy one, which seems likely to smuggle in some fresh nugget of buried failure, or buy something more professional, and hence eye-wateringly expensive
    2. Set up static configuration for every client. Seems a stupid solution in 2010 for a primarily wireless network
    3. Disable DHCP on the router, and add another, more reliable DHCP server to the network


    Option 3 initially seems least aggravating. In the past, my strategy for service infrastructure has always been using home servers, with some form of UNIX. These days though, I'm trying to minimise the number of computer-type devices I have to keep running 24/7. I no longer find any joy in being a home UNIX administrator, and it's nice to correspondingly reduce power consumption, fan noise, and cabling. So the idea of setting up a computer just to act as a DHCP controller is slightly repellant.



    The only machine tethered to the network is a modest, first-generation, G4 mac mini . It's chief use in the past was as a basic freeview PVR, using Elgato eyeTV , but the London flat's TV reception is too poor for this, so it mostly acts as an AFP -capable network interface to my firewire Drobo . It's a very old, low power machine, but would certainly be capable of acting as a DHCP server.

    It wasn't immediately obvious how best to do this. Obviously I could install any of the common free UNIX DHCP software, using MacPorts , or homebrew , or fink , or even just hand rolling something from tarballs, but all of these come with overheads, adding dependencies, requiring build tools, and subsequent package management, and all the little bits of service glue needed to make it run neatly as a daemon. Experience has shown me that integrating third-party UNIX services into a vanilla Macintosh can get fiddly, fast.

    There's no obvious DHCP server component on desktop OS X, but there's a latent capacity somewhere, demonstrated by 'Internet sharing', which lets you easily set up a Macintosh with a network connection as a basic router. After a little bit of poking around with this, and some internet searching, I discovered that this facility is part of the bootpd service. It's documented, and after a little trial-and-error, I figured out a way to run a DHCP server facility only, using just the built-in Apple utilities.

    Here's an overview of my network configuration
    • The D-link router provides NAT routed internet via O2 ( actually Be ) ADSL 2+ with a static IP.


    • The private subnet is 192.168.1.0/24


    • The router's internal address is set as 192.168.1.1


    • The mac mini is connected to the router via wired ethernet with an address of 192.168.1.4, and runs headlessly.


    • Everything else connects to the D-Link router wirelessly, using a mix of 802.11n and 802.11g




    Here's how to set up bootpd to act as a DHCP server for this network.



    First, configure the mac mini to have a static IP. Using screen-sharing from another Mac ( Cmd-K, vnc://192.168.1.4 ) to configure the network interface in system preferences.








    Next, configure your computer to also have a static address on the same subnet. If you get something wrong, and need to troubleshoot settings, you'll still need to be able to connect between the router, the mini and your workstation. I picked 192.168.1.111, as being well outside the range of anything I'd expect to be routinely allocated.

    Now you need to produce your bootpd config file ( /etc/bootpd.plist ). Unfortunately this means an XML property list. Every time I feel smug about how the Macintosh is re-invigorating UNIX with the old, crufty bad bits removed, I ought to remind myself about the maniacally stupid idea that is XML plists. Instead I thank my stars that I have a capable text editor. It's not that fearsome a property set, and is well explained in the man page , so you could build one by hand. An alternative approach, the one I used, would be to set up internet sharing temporarily on the mini for an interface you're not using; I chose firewire. Take a copy of the /etc/bootpd.plist file this will create, e.g. / etc/bootpd.plist.template , and then disable internet sharing again, which will remove the /etc/bootpd.plist file if it still exists. Now rename your template back to /etc/bootpd.plist and edit it.

    The options are all well documented, and it turns out that you need hardly any of them to get up and running.

    The key options are


    • dhcp_enabled: an array of network interface device names to answer dhcp requests on - I just have en0, which is the built-in ethernet
    • Subnets: an array of property dictionaries, that represent networks we're interested in serving. We only want a single dictionary for 192.168.1.0/24.
      • net_address: , is the network address - 192.168.1.0,
      • net_mask: the netmask for our subnet range - '255.255.255.0',
      • dhcp_router: default gateway address - 192.168.1.1
      • net_range: an array of strings representing the bounds of a pool of addresses to allocate from - 192.168.1.12 to 192.168.1.254
      • allocate: a boolean that is set to indicate that we're interested in issuing addresses for this subnet
    <ul>

    Most of the other defaults are sensible. I've kept all the other values that were generated for my template. Here's what I have in my file.


    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
    <plist version="1.0">
    <dict>
    <key>Subnets</key>
    <array>
    <dict>
    <key>_creator</key>
    <string>cms</string>
    <key>allocate</key>
    <true/>
    <key>dhcpdomainname_server</key>
    <string>208.67.222.222,208.67.220.220</string>
    <key>dhcp_router</key>
    <string>192.168.1.1</string>
    <key>lease_max</key>
    <integer>3600</integer>
    <key>lease_min</key>
    <integer>3600</integer>
    <key>name</key>
    <string>192.168.1</string>
    <key>net_address</key>
    <string>192.168.1.0</string>
    <key>net_mask</key>
    <string>255.255.255.0</string>
    <key>net_range</key>
    <array>
    <string>192.168.1.12</string>
    <string>192.168.1.254</string>
    </array>
    </dict>
    </array>
    <key>bootp_enabled</key>
    <false/>
    <key>detectotherdhcp_server</key>
    <integer>0</integer>
    <key>dhcp_enabled</key>
    <array>
    <string>en0</string>
    </array>
    <key>replythresholdseconds</key>
    <integer>4</integer>
    </dict>
    </plist>

    Next, create two empty files that bootpd expects to use. ' /etc/bootptab ', for any static address maps, and /var/db/dhcpd_leases , which will be a persistent database for issued leases. Now connect to the router, and disable it's DHCP server.

    The bootpd binary lives at /usr/libexec/bootpd . If you run it from a terminal with a -d flag, it will stay in the foreground and emit debugging info to stdout. You'll need root privileges for it to run, I just used sudo /usr/libexec/bootpd . Now request a dhcp address from a different network client. I used an iPad. It's a good idea to make a note of the network MAC address. If everything is working, you should see some output acknowledging the request, and then some more as a lease is issued. The client should then configure it's network interface with all the settings from your Subnet definition above. If it doesn't, and the output isn't helpful enough, there's also a further -v switch for more verbose logging.

    Initially I had trouble getting any leases issued although all requests were logged fine. It turned out I'd misconfigured the netmask when I set up the static address for the mini. If the network details don't match the defined subnet exactly, then bootpd will just fall back to default behaviour for the subnet, which is to just observe. Once I fixed that, things started working as they should. By default, a line is written to logs in /var/system.log for every request recieved, and one for every lease issued.

    The remaining task is to configure the service to run as a daemon from launchd. Luckily, there is a launchd profile for bootpd present, /System/Library/LaunchDaemons/bootps.plist .

    You can install this persistently into launchd like so

    sudo launchctl load -w /System/Library/LaunchDaemons/bootps.plist

    Running sudo launchctl list should then show a com.apple.bootpd service enabled. If for some reason you need to disable it once again, you can uninstall the service using

    sudo launchctl unload -w /System/Library/LaunchDaemons/bootps.plist



    posted by cms on
    tagged as

  5. 2010-11-28

    How about some photos of squid flying through the air? I've heard anecdotal reports of this sort of thing happening, which on the face of it sound reasonable, if not a little far fetched. They do possess all the right sort of equipment, and controlled jet propulsion through the air isn't really that far from their usual method of locomotion at speed, which is controlled jet propulsion under the water, after all.

    The full writeup in the parent post contains plenty of detail about a recent observation of groups of squid exhibiting fairly controlled, short flight. Not only does the article contain lots of interesting links to scientific write-ups of arial squid observation , but it also contains several high-resolution photo images of the buggers captured in the act.

    It would make a lot of sense for them to use as an evasive action. Squid can manage impressive accelerations in their submarine environment, but through the air, they would perform even more rapidly, over short distances. "Short" is of course, relative. One of the write-ups based on observations estimates 20cm squid reaching 10m in a controlled flight. They seem to form their bodies into lifting, braking and stabilising shapes as they go. Squid are ace.

    posted by cms on
    tagged as

  7. 2010-11-22

    I'm amused that on porting his blog to wordpress jwz has seemingly reached the same level of disgruntlement with wordpress in about a day that it's taken me twelve months or so to reach.

    Don't get me wrong, wordpress certainly gives you all sorts of awesome features OOTB, but at a certain cost of complexity, which makes things tricky to customise. Themes are hard to tweak, and the cost of entry to plugin-writing is large enough to put-off simple customisation in favour of out-sourcing to the lazyweb directory of plugins, which correspondingly increases the complexity of your install.

    Most pertinently, there's the security record, a cynic might suggest it's a lack of security record. I'm gradually coming around to the line of thought that the frequency of updates actively contributes to the problem. The continual treadmill of manually updating drives people to investigate the auto-upgrade procedures, which are all built around interfaces that sound to me like designed-in exploit vectors, like having all the .php files in the software tree writeable by the httpd user, or running an FTPd service on the webhost that can chdir to the http script directories. Furthermore, the autoupgrade process is prone to terrifyingly unfriendly fail-states .

    I'm not sure if there are any significantly appealing alternatives out there. I think there's probably a circular life cycle to the blog software used by any mildly technical person , that moves serially from 'simplest possible lazyweb solution', through 'this simple thing has been customised past the point of sanity, I'll write my own' all the way through to 'writing blogging software is hard, I'll just use wordpress' and subsequently right back to square one.

    The elephant in the room is the simplest option. Just host your data in an fully managed service like wordpress.com , or tumblr or posterous . Or if you really don't care about handing every last bit of data you can generate about yourself into the possibly malevolent skynet-cum-panopticon Google-monster, you could get all oldskool with blogger As ever, I just can't get with the idea of giving all my content to an at-best disinterested third party. After all, that's where jwz started out , and look where that's got him. Manually migrating to wordpress, and grumbling.

    posted by cms on
    tagged as

  8. 2010-11-21

    If you can read this message, it means I have properly configured blog posting by email. Is this useful? Is this a good idea? Time will tell.






    posted by cms on
    tagged as

  10. 2010-10-14

    In my piece yesterday, about my blog on last.fm I didn't link correctly to the article . Fixed now.

    posted by cms on
    tagged as

  11. 2010-10-13

    The other day at work , prompted by a shoutbox conversation with one of our users , I did a little bit of exploring some of the artist catalogue data. The idea was to find band names that were repeating words, such as ' Talk Talk ' and ' The The '. Coincidentally, I had a freshly installed database server with just this sort of information on it, and needed a good excuse to stress test it a little. PostgreSQL's regular expression support is brilliant , and it was a very trivial exercise to quickly knock up a query that returned promising data. In the process of refining it, I got a chance to play around with the Hadoop cluster. I wrote the whole thing up over on the company blog, if you'd like further details. Fame fame fatal fame, it can play hideous tricks on the brain, as the song goes .

    posted by cms on
    tagged as

  12. 2010-08-26

    Yesterday at work , I had to clean after up a particularly freaky Slony-I replication fault. I still haven't managed to understand quite what went wrong there. So this morning, I arrived at work in full diagnostic mode, jokingly grumbling about 'howfuckedismydatabase.com'. Laurie was particulary amused by this curmudgeonly joke, and we bantered about it. I pitched a few ideas about how such a joke site might operate, and we left it there and moved on.

    Except Laurie didn't. Despite my attempts to dissuade him, he registered the domain, and started knocking together some pages based on the earlier jokes. I chipped in a couple more suggestions, and suggested some error messages, and within twenty minutes or so he had an operational site . Then we shared it with a couple of like-minded people, and left it be. A few of the other people at work passed it around, and a couple of people submitted it to reddit.

    Within an hour or so things had started to really snowball. One of the reddit submissions gathered hundreds of upvotes, and for a period of time we were the number one story on hacker news . Laurie added a twitter button and a comment form to the site, and retweets and emails started accumulating fast. By mid-afternoon the site was approaching 200 hits a second, which it handled with aplomb, because he had coded it efficiently, and
    configured the server sensibly.

    It felt great to watch so many people comment positively about some of my dumb jokes, pretty much in real time. It gave me a really direct experience of something I'd always innately understood about the internet, but had not yet witnessed close to home; the ability to quickly reach an appropriate audience for almost any content, regardless of how specialised. Our little shared joke quickly reached out to thousands of people, who found something within it they also related to. This really amazes me.

    It also showed me something about my own character. While I was perfectly happy to joke about the idea, it needed somebody like Laurie, with the skill and enthusiasm to pick up on it and make it into something tangible and exciting. I'd instinctively shied away from broadcasting it further than my desk, and my initial reaction was that developing it any further would be a waste of time and money. I was very wrong about that, it turned out to be an interesting experience, and enormous fun. I think this means I should endeavour to be a little less cynical.

    posted by cms on
    tagged as

  14. 2010-08-06

    I ran into some problems while I was trying to install python bindings for the Growl notification framework on my MacBook Pro. My Mac is running the current release of Snow Leopard ( 10.6.4 ) and I'm using a python.org installed binary package of python, under /usr/local/python. Building using distutils and the supplied setup.py failed, seemingly because the compiler was unable to find quite routine include files, such as stdarg.h and float.h .

    /Developer/SDKs/MacOSX10.4u.sdk/usr/include/stdarg.h:4:25: error: stdarg.h: No such file or directory

    This error message both confused and perturbed me, because stdarg is a fairly fundamental component of a working C library, and I am pretty certain that my compiler isn't that fundamentally broken.

    Picking apart the build output from the generated Makefile, I see that it is setting the -isysroot gcc flag, to /Developer/SDKs/MacOSX10.4u.sdk/ . I presume this is because the python installation is built to use the OS X 10.4 compatability SDK. This is why it's pulling in /Developer/SDKs/MacOSX10.4u.sdk/usr/include/stdarg.h . That header is a stub, and included the following stanza



    /* GCC uses its own copy of this header */
    #if defined(GNUC)
    #include_next


    #include_next is a gcc extension to cpp, and instructs the preprocessor to start searching for the include file again starting with the next directory on the include path after this one. Standard libraries like stdarg and float can be quite compiler specific, and as the comment indicates, GCC is expected to have it's own copy of this header file, which would be put away somewhere under /usr/lib/gcc .

    At this point, a nagging memory of building cocoa apps with XCode resurfaced, suggesting that the 10.4 SDK isn't compatible with gcc-4.2 ( the system default gcc under snow leopard ). GCC 4.0 is supplied though, for use with building against legacy SDKs. On this whim, I tried exporting CC=/usr/bin/gcc-4.0 and rebuilding, and everything worked as it should.

    From inspection, it seems like the snow supplied leopard python is built to use 10.6 SDKs and gcc-4.2 and may well be a more sensible python to use. Further googling ducking , turned up this bug report .

    posted by cms on
    tagged as

  19. 2010-05-19 

      It's been a good long while since I've been able to blog anything of substance. Some of that is down to lack of time, I've been frantically busy the last few months, much as you'd expect. There also were some dull technological barriers that were making it awkward to update and maintain this site. <br><br> I've moved the hosting to a new location, it's currently residing on a xen virtual server instance provided by <a href="http://linode.com">linode</a>. While I was migrating things around, I've tried to package it up a little more portably, and in future I ought to be able to move it easily to anywhere I can run a linux host. I also took the opportunity to tidy up the page templates, and cobble together a new theme. I'm still poking that around a little bit, let me know if you find any rough edges.<br><br> <h4>New job!</h4><br><br> Aside from adjusting myself to my <a href="http://beatworm.co.uk/blog/ada/im-a-rapper-with-a-baby/">wonderful new daughter</a>, I've gone and got myself a new job. I wasn't really aware that I was looking for one, but life can surprise you like that sometimes. I'm now working as a Database Architect at <a href="http://last.fm/user/colins/">Last.fm</a> and I couldn't be happier about that. Not only is last.fm an awesome site, which has long been one of my favourite things on the web, but the intersection between high volume web services, big databases, and music nerdery is very definitely my kind of niche.<br><br><br><br> <h4>Move to London</h4><br><br> One small drawback with this full-time role was that it was based in London. We did weigh up the various commuting options, but after some deliberation, decided to take the plunge, and relocate, at least temporarily to London.<br><br> This meant finding somewhere to rent. Somewhere to rent that would take a baby and a dog. A location in the city with suitable dog exercise routes close at hand. Ideally a place from which I could daily commute to <a href="http://maps.google.co.uk/maps?f=q&source=s_q&hl=en&geocode=&q=last.fm&sll=53.800651,-4.064941&sspn=16.450021,44.428711&ie=UTF8&hq=last.fm&hnear=&ll=51.528457,-0.086882&spn=0.004052,0.010847&z=17&iwloc=A">Shoreditch</a> without too much trouble. Obviously we'd have to be able to afford it on one salary, whilst still maintaining a mortgage on the house in Bristol in the interim. Rather a tall order.<br><br> After balancing up the variables we settled on the Balham/Clapham area. After a few complicated expeditions up to view properties with little success, we managed to secure something with only days to spare, just down the road in Tooting Bec. A rather roomy <a href="http://maps.google.co.uk/maps?f=q&source=s_q&hl=en&geocode=&q=Elmbourne+Road,+Wandsworth&sll=51.434735,-0.150461&sspn=0.008628,0.015535&ie=UTF8&hq=&hnear=Elmbourne+Rd,+Wandsworth,+Greater+London,+United+Kingdom&ll=51.434655,-0.150611&spn=0.008575,0.015535&z=16&layer=c&cbll=51.434733,-0.15047&panoid=hH03gMlSJRXaMcWDe_cQ_w&cbp=12,273.93,,0,6.37">ground floor flat facing right onto Tooting Bec common</a>. It's just a short walk to the Northern line, which leaves me with a manageable half-hour or so trip to work, door to door.<br><br> So the last few months have seen quite a furious pace of changes. Mostly I've been finding it all invigorating, and exciting, rather than incapacitating, but things can seem to be wooshing by, and there <em>definitely</em> aren't as many hours in the day as there used to be. Five years ago I'd have been amazed at what the me of 2010 would be getting up to. Interesting times.
    posted by cms on
    tagged as