Thursday, July 26, 2012

Jenkins forces reinstall of server images on Rackspace

This is much more DevOps than Dev, but it is important as well ! One difficulty we have doing remote development with a team spread over at least a million cities (I haven't actually counted) is that the local development LAN for Bob is frankly inaccessible for Alice and Charlie. Maintaining n-1 VPN links is just going to drive us all mad.
So the simple solution is to rent a couple of cloud servers and host the development builds on there. The cost is pretty minimal, and it seemed good - until I realised that Jenkins, our friendly Continuous Integration Butler, was unable to re-install a cloud server from an image. Or rather it *could* - but Jenkins did not know when the server was rebuilt, and the server itself, which could signal jenkins, would have no state, so could not signal jenkins with *why* it had been rebuilt. So, Pantry Bell ( We simply run a webserver, that when called (perhaps from one Jenkins job) will force a re-imaging of a Rackspace server. This has a simple callback in rc.local (wget`hostname`) So - a server forces a rebuild, on successful rebuild, the server calls home, and then the "real" jenkins job can be triggered using the Jenkins API calls. Amazingly enough it works, has trivial security, no REST compliance but it works