Dustin Sallings

Hello World in Go -- A Memcached Server

2009-11-12T00:00:00-08:00

Hello World in Go – A Memcached Server

I sat down last night to learn go. I’ve been a fan of concurrency-oriented languages since I wrote my first erlang program in early 2004 (which I still use). I’ve been itching to give go a go since the announcement.

The first thing I thought of that could naturally be modeled as a concurrent program was a memcached server. That has sort of become my “hello, world” as I have implemented the first python server for the memcached binary protocol (using asyncore), the main server (which Trond made beautiful), an erlang server, and a twisted server.

My go server has a lot in common with ememcached since both languages are concurrency oriented. There is one process/goroutine for managing the actual data storage, one for accepting TCP connections, and one for each connected user.

The TCP listener just accepts and spins up new processes or goroutines to handle the IO on the connections and then they wait for data. Once they’ve read a single request, they dispatch to the storage mechanism which will sequentually process each operation, regardless of input concurrency. Very simple, very easy to understand.

I mostly liked what I saw. The code feels a little C-like, but in practice is fairly intuitive and pleasant to work with. There are many reviews of the language and I’m not attempting to write one, but I will say that I got hung up on the lack of exceptions which required me to do weird error handling in sequential code.

The concurrency primitives remind me a bit of alef which I remember from early plan9 distributions, though I never actually got a chance to work with it before it got killed off. The one part of alef I really liked lived on in go, which further motivated me.

ZFS for MacOS X

2009-10-23T00:00:00-07:00

ZFS for MacOS X

So I’m a pretty heavy ZFS user. I’ve got a FreeBSD box with a ZFS root that all my important stuff is on, and a mac mini with ZFS that all my movies are on.

Around Snow Leopard, Apple seemed to have not only dropped support for it, but the download from macosforge seemed to not work, either.

I’ve been hoping someone would come around and fix this, but nothing’s happened.

Then today, I read this:

The ZFS project has been discontinued. The mailing list and
repository will also be removed shortly.

That made me very sad, so I decided to do something about it.

Github Project

I set up a mac-zfs project on github and started hacking. It didn’t take long and I was able to get to my zpool that has all of my music. (for the record, Bad Religion was my ZFS test case)

Downloads

I wanted to make it a bit easier to get going, so I created an installer.

You still need to kind of know what you’re doing to make use of it. But I’m hoping we can work together to make it easy for everybody.

Update:

We’ve got a google code page up for bug tracking and what-not and a mailing list for general discussion.

Come join us.

Memcached Report Card

2009-10-22T00:00:00-07:00

Memcached - Report Card for an Open Source Project

While approaching our 1.4 release, the memcached project slipped into a state where we weren’t pushing out new code very frequently.

Some members of the community demanded releases occur more frequently, so dormando published a proposal for a release cycle for memcached that would help drive us into the direction of universal happiness.

I had a feeling that his plan and efforts to keep us on it were quite effective, but the purpose of this post is to sort of take an objective retrospective look and see how the reality of the progress of the project holds up to my perception.

Release Scheduling

This chart pretty much speaks for itself. It’s clear to see that we very quickly approached the stated goal of 30 days between releases.

I should note that 1.4.1 was released later than expected because a user hit a real live failure scenario in a production system that took a while to isolate. Once we did, we wrote some tests to ensure that the entire class of bug described can’t happen again.

These releases are bigger than they appear, but very well tested on our build farm and in production environments. To see what goes into each release, check the release notes for 1.4.0, 1.4.1, and 1.4.2.

Bug Status

We’ve been taking bugs very seriously.

Crash bugs are resolved very quickly, but bugs in general are resolved within our release cycle.

Please let us know via our mailing list or our bug tracker if you’re having issues or can’t figure something out. We fix things very quickly, but we can’t fix what we don’t hear about. Remember that goes for the wiki docs, too.

The following chart shows the average age (time between when they were filed and when they were closed) of bugs by month in memcached. The error bars are showing the minimum and maximum age for any given month. Issues marked as invalid, won’tfix, and had other indicators of not being actual defects or missing features in the software were excluded.

Four anomalies were identified and described briefly in the chart. These are all considered trivial issues that were not detrimental to the state of the server, but they do show that we’re sometimes not too quick in cases like these. (more details on A, B, C, and D).

I also found it useful to consider what I called the “bug load.” That is, the number of bugs going in and out of the project by month.

The following chart is showing bugs activity in the memcached project.

Positive numbers are bugs being reported to the project. Negative numbers are bugs that we closed (bugs removed from the project). The line indicates the net number of bugs at the end of the given month.

So What’s the Grade?

I’d say we’ve got a solid B. We have room for improvement and I’m confident we’re making it.

If you disagree (and can articulate why specifically) or otherwise have feedback, a warm, active community awaits.

EMemcached

2009-10-11T00:00:00-07:00

Why an Erlang Memcached Implementation?

I wrote an erlang implementation of a memcached server speaking the binary protocol over the weekend. You’re all probably wondering why.

It was half a learning exercise, and half a way to plug a memcapable interface into more applications. In its current form, I consider it a framework to take existing erlang backend stores (e.g dict, ets, dets, mnesia, riak, etc…) and put a memcached interface on them. On its own, it’s a little boring.

The first implementation of the binary protocol was in my memcached-test project (which I still actively use as a reference implementation when playing around with features and testing clients and things). It’s a simple asyncore based server in python (with a sample synchronous client).

A bit later, came the actual production C server version we all know and love (which also had my java client to talk to).

I wrote twisted memcached and built an S3-backed server one weekend.

That leads us back to the present erlang-based server.

But Wait, There’s More!

If you have a basic understanding of erlang, you may find this implementation to be the best documentation of the protocol in existence.

In the main loop of a connection, all I’m doing is calling process_message in a loop with the connected socket, a reference to the storage server (an erlang gen_server implementation), and the result of a call to gen_tcp:recv(Socket, 24). That last call will either return {ok, SomeData} or {error, SomeReason}.

The Fun Part

The only definition of process_message/3 I have is shown below. The only valid way to call this is when the third argument is the {ok, Data} tuple where Data in this case is a binary pattern. Some of the values are filled in (which means they must match for this function body to be invoked), and some are bindings which will receive the value.

In the code below (extracted from mc_connection.erl), you’ll see that the first 8 bits must exactly be the defined REQ_MAGIC constant, and then the next 8 bits are stored in OpCode, and so on.

Any attempt to process a message not in this form will result in the connection processing crashing (the effect of which is your client being disconnected from it).

So you can see how erlang easily allows us to rip the bits we need out of the header for dispatch, so the next thing is to ask for the remaining data (extra headers, key, and body) before dispatching to the storage server process.

process_message(Socket, StorageServer,
                {ok, <<?REQ_MAGIC:8, OpCode:8, KeyLen:16,
                      ExtraLen:8, 0:8, 0:16,
                      BodyLen:32,
                      Opaque:32,
                      CAS:64>>}) ->

    % After the header is extras, key, and then body
    Extra = read_data(Socket, ExtraLen),
    Key = read_data(Socket, KeyLen),
    % Note that the length of the body from the header includes
    % the lengths of the key and extras.
    Body = read_data(Socket, BodyLen - (KeyLen + ExtraLen)),

    % Dispatch the read data to a gen_server process.
    Res = gen_server:call(StorageServer,
                          {OpCode, Extra, Key, Body, CAS}),

    respond(Socket, OpCode, Opaque, Res).

The Storage Server

A storage server process is a gen_server implementation whose handle_call implementations look will take a tuple of {OpCode, ExtraHeader, Key, Value, CAS} and return a mc_response record.

For an example storage server, consider my two flush implementations in my hashtable store (noting that flash has one 32-bit integer in extra header, no key, and no value):

% Immediate flush. Ignore the current state and make a new one.
handle_call({?FLUSH, <<0:32>>, <<>>, <<>>, _CAS},
            _From, _State) ->
    {reply, #mc_response{}, #mc_state{}};
% Delayed flush. Keep the current state and schedule a
% flush to occur (via handle_info) in the future.
handle_call({?FLUSH, <<Delay:32>>, <<>>, <<>>, _CAS},
            _From, State) ->
    erlang:send_after(Delay * 1000, self(), flush),
    {reply, #mc_response{}, State};
% More stuff Follows

That’s pretty much it. Even if nobody uses this code, it’s useful to me as a protocol reference since it’s easier to read than even the binary specification.

spymemcached Optimizations

2009-09-23T00:00:00-07:00

I got around to pushing out a new RC of spymemcached today. It’s been a while, but I’m glad I got around to it.

The announcement has the release notes (also in the tag), but there is a particular optimization I’ve been thinking about for a while, and would like to go over here somewhere below.

But first, I’ll frame it with a bit of memcached protocol fundamentals.

Introduction to Quiet Operations

Back when we were initially designing the binary protocol, we were considering how we’d handle the multi-gets. We went through several proposals until we realized that the actual essense of multi-get model was really just a class of operation that allowed us to infer some of the responses.

The above diagram shows a simple case of a multi-get. We ask for the values behind four keys. The server sends us responses for two of those keys and then says it’s done. In a client, we can safely assume that it just didn’t have the other two. It doesn’t need to actually tell us that.

So in the binary protocol model, we just made a type of command that didn’t respond with “uninteresting responses.” It’s easy to see how in a get operation, the not found response is uninteresting as we can infer it.

Other Quiet Operations

Shortly before the actual 1.4.0 release of memcached, we defined semantics for all “quiet” operations in such a way that allowed us to maximize efficiency without compromising correctness.

The binary protocol definition goes through these in tremendous detail, but those familiar with the Unix philosophy will probably find such things intuitive.

For example, in Unix, the rm command does not print out any output on success. If it completes and didn’t say otherwise, you can assume it was successful.

Similarly, a quiet delete operation doesn’t need to tell us that it successfully deleted something. That’s its job. We want to know when it fails to do it.

Optimizing with a Quiet Set

The optimization I was interested in was making a multi-set type operation that worked similarly to the multi-get functionality. After struggling with what such an API might look like, I finally decided that the right thing to do is not change the API at all.

Instead, I do something similar to multiget escalation – an optimization that’s been part of spymemcached for a long time now. If many threads are pushing sets in (or even a single-thread since the typical use-case of set is async), the packetization of these commands escalates a sequence of similar commands into a single sparse operation working on all of the items together.

While YMMV, my cache loader test ran consistently twice as fast.

Previously, one million requests would require the client to process one million responses. Now, one million requests (assume none fail) will require the client to process one tiny response.

If any do fail, the respective callers will, of course, be notified. The ones that don’t fail receive synthetic callbacks as the client infers their success.

What You Need to Do

If you’re using spymemcached, upgrade and you get the optimizations.

If you’re a client author, see how much better things are for your users as you make broader use of quiet operations of the binary protocol.

Tornado on Twisted

2009-09-12T00:00:00-07:00

So what’s this about tornado?

The friendfeed guys created an awesome web site with what was obviously (from the outside) quite awesome technology. A couple days ago, they released the the technology behind the site.

The Problems

Tornado really is two different things:

A great framework for building web sites.
A low-level networking toolkit.

Most of us who use twisted were quite surprised to find out that rather than using twisted’s awesome networking core, they reimplemented a bunch of it. Moreover, they kind of had really vague negative things to say about twisted. It’s not clear what problem they had with it, but as stated the logic kind of fails:

Twisted doesn’t have a good web framework.
No major web sites run on twisted.
So… we’re going to build something completely from scratch.

Now, I certainly build stuff from scratch unnecessarily all the time, and I also don’t want to seem unthankful for the free gift brought to us from our friendfeed friends, but in its current state, it’s a technological island.

On one hand, you have low-level missing pieces. From the bottom, you’ve got multiplexing implementations. twisted has been around a while, so it supports native multiplexors including those for select, poll, epoll, kqueue, CoreFoundation, wxPython, win32, gtk, glib, qt, and probably more. While some of them are less interesting than others, that’s a lot of catching up to do. Some user already began adding kqueue support, but it’s more of an example of the kinds of things that you don’t get for free.

At the high end, there is an incredible selection of protocol implementations from twisted you just can’t use. If you’re building a web site on an asynchronous framework, you don’t ever want to block to get further data. So whatever your networking framework, you need it to have a sane way of asychronously communicating to all of your dependencies.

They wrote an http client, though there are things I do with the twisted http client that I don’t see a way to do with the tornado client such as parse infinite xml streams from http responses.

The Solution

So instead of just generally being frustrated, I thought I’d see what it’d take to rip out the stuff that was reimplemented and use just core twisted.

The original tornado code doesn’t have much in the way of tests, so it’s currently in the “appears to work” state, but this is what got me there:

That is, with an offset of -1,297 lines of code, it can be observed to work for the cases I tried, although I’m sure there are still lines that need to be deleted before everything works.

The good news is that this means that you get all the richness of twisted and the good parts of tornado combined.

Everybody wins.

So go grab the code and see if it works in your app, or send me some patches for parts you were able to explore a bit more deeply.

Buildbot and Git Repositories

2009-09-06T00:00:00-07:00

Buildbot and Git Repositories

In a recent conversation with the GitHub guys, I was talking about how my buildbot setup was hitting GitHub and how a recent filesystem glitch of theirs caused my screen to turn red with growl alerts from buildwatch.

The response was a tongue-in-cheek “I refuse to apologize.”

The thing is, that response is absolutely the right one. This is distributed revision control. Why did I have a screen full of growl alerts because of a failure of a filesystem completely unrelated to what I was doing?

I was relying on GitHub to be highly and quickly available to my seventeen (and growing in number) buildbot slaves for this project.

Most of the time, there’s nothing for them to actually get from a centralized revision control – quite simply, they were asking for information that they had cryptographically verifiable assurance that they already had.

Today I made a small change to buildbot that prevents the slaves from ever talking to any network service to pick out a reference version in our most common use cases, thus realigning myself with the thing that initially sold me on GitHub’s service: It enhances collaboration without causing me to be dependent on the service.

For this failure, I am thankful. This new code will always be faster and more reliable for the common case even when GitHub works absolutely flawlessly.

Memcached 1.4.0

2009-07-16T00:00:00-07:00

Memcached 1.4.0

I’m a bit late to the blogging party here, but we finally released memcached 1.4.0. Check out the release notes for more details.

The release notes cover quite a lot of the interesting stuff, but they don’t properly reflect the time and effort that went into making this all happen.

There are a lot of bug fixes, as one might expect after some time. A lot of testing has shown that performance is better pretty much all around, but very few people have ever seen memcached be a performance bottleneck in their applications, so that’s not too exciting.

The biggest part of this release, however, is something I’ve been working on for about two years: the binary protocol.

The Binary Protocol

So what’s the big deal about the binary protocol?

The most obvious thing for protocol implementors is that it’s now really easy to parse the protocol. After reading a fixed-size header, a low-level packet processor can figure out where to dispatch the input and split it into all of its major components (key, value, opaque, cas, extras, etc…).

That’s great for the (small) number of developers who write servers and clients, but what about random people out there who just want to use memcached? Semantic enhancements in the new protocol allow us to build some really cool stuff.

The first example of such a thing is Trond’s replication functionality for libmemcached. We now have a clean fire-and-mostly-forget protocol semantics that allows for improvements like efficient client-side replication. It also makes it safe to make bulk-loaders (even with CAS).

Go Try It

We’ve run tons of tests, others have run tests, there’ve been various deployments large and small, but if you’re running something older, it’s your turn.

We work hard to make sure that the development versions work on all platforms we can find anyone to care about. Each change is built and tested on all supported platforms before the change is accepted into our master branch.

Do note that 1.4.0 has some build issues on OpenBSD, but someone graciously donated a builder to our buildbot farm so they’re all cleared up for 1.4.1 (which is planned for later this month).

In the meantime, there are several ways to pick it up:

Packages

Package systems are slow to pick up… anything it seems. If your system’s package manager is shipping memcached 1.4.0, please let me know.

In the meantime…

Use the Source

You can download the source distribution from the google code download site.

Building and installing is quite simple:

./configure
sudo make install

Then just run /usr/local/bin/memcached whichever way suits your fancy. Personally, I like upstart on Linux, launchd on OS X, smf on Solaris, etc…

Deploying on Amazon EC2/AWS?

I’ve put together some Amazon AMIs that are production ready and ridiculously simple to get going.

Each AMI allocates all but 512MB of RAM on the system to memcached and just starts up happy and running. These images are based on Ubuntu 9.04 and have an upstart config for the actual daemon execution so if we somehow have some kind of crashing bug, they’ll automatically and instantly restart.

Depending on your needs, you can select one of the following:

US

I’ve assembled a 32-bit AMI (ami-39c52450) for small instances, and a 64-bit AMI (ami-1fc52476) for large instances. They show up as the following:

ami-39c52450 - northscale/community-memcached-1.4.0-i386.manifest.xml
ami-1fc52476 - northscale/community-memcached-1.4.0-x86_64.manifest.xml

For example, using the ec2 command-line tools can start an extra large 64-bit instance with the following invocation:

ec2-run-instances ami-1fc52476 --instance-type m1.xlarge

After this instance comes up, you’ll find memcached 1.4.0 listening on port 11211 with about 15GB of RAM at your disposal.

No maintenance should be necessary, but ec2-run-instance’s -k parameter for supplying a root ssh key is still honored in case you want to still look around.

EU

There are European versions of the same images as ami-818ba3f5 for 32-bit and ami-838ba3f7 for 64-bit.

ami-818ba3f5 - northscale-eu/community-memcached-1.4.0-i386.manifest.xml
ami-838ba3f7 - northscale-eu/community-memcached-1.4.0-x86_64.manifest.xml

Project Skyscraper

2009-07-11T00:00:00-07:00

Project Skyscraper

It occurred to me that there’s a lot of value in building xmpp services – much like web services, but using existing connections and xmpp instead of http.

In collaboration with ga2arch, I launched an xmpp service called skyscraper.im. This has actually been running for a while now, but I’ve been too caught up in writing code to write anything about code.

translate.skyscraper.im

The first part of this service is an xmpp adhoc interface to google translate. It actually does support IM, but that’s incidental, the real value is in the adhoc interface.

If you’re unfamiliar with xmpp adhoc, you can think of it much like CGI, but using xmpp as a transport. You take a bunch of simple key/multi-value pairs and send them to a resource somewhere, and it sends you something back. The nice thing about xmpp, though, is that the mechanism for determining what things exist and what parameters they take are very programmatically accessible.

You can discover available commands through translate.skyscraper.im as shown in this video, but I’ll just tell you what it’ll tell you:

The Input

There is one field called in which is the input language in the form of a two-character language code. You may have only one of these (it is of type list-single).

There is one field called out which is the output language and is also in the form of a two-character language code. You can have as many of these as you like (it is of type list-multi).

Finally there’s a field called text which is the stuff you want to translate. You may only have one of these.

The Output

The response is a form much like the one you sent it, the keys are language codes and the values are the text translated in that language.

Note that you will not receive more language translations than you asked for, but you may receive fewer in the case where the upstream translation service can’t perform such a translation.

The obvious benefit here over doing it yourself is that you get full translations all at the same time without having to do any kind of coordination as things are completing (i.e., I do that for you).

conference.skyscraper.im

A fun thing built atop the translate component is the skyscraper muc – an xmpp multi-user chat with automatic translation.

What this means is that you can have several people enter a room with no room in common, all speaking and reading their native language.

Of course, the dream is limited by the translation service, but it does work within the reasonable limits.

If you’d like to try it out, find a friend who speaks another language and both join a chat room at conference.skyscraper.im. Start by each of you telling it your respective languages (e.g. /lang en) and then talk.

I’ve spent very little time on this, so I imagine it falls apart in all kinds of places, but it was really easy to get going with the translate service from above, and as it’s an xmpp server component, it does all this with just one file descriptor and the necessary state to keep up with who’s in what room and what translations are outstanding.

My Github Anniversary (Sort Of)

2009-03-01T00:00:00-08:00

My Github Anniversary (Sort Of)

I joined github about a year ago today. Kind of. It was actually February 29th, but there isn’t one of those this year, so I’m going to have to wait a few more years before I can properly have an anniversary.

Between the time I joined and the time I typed this line, I’ve generated 206 pages of activity (174 pages public).

By the end of my first day, I had migrated two java projects over from mercurial, converted a ruby project from subversion, had that repo forked, watched another ruby project, started a new objective c project, wrote some new code for some of my projects and pushed it and invited a couple of friends (both of whom now share my leap-year-only start date) and added them to a couple of my projects.

Chris became somewhat a github evangelist and made some really cool stuff stuff there (some if it’s cooler than most people can comprehend). Ian throws awesome parties (I’ll eventually make him give me code).

At the point where I started using github, I’d probably been a (somewhat casual) git user for about two weeks. git is great, but the documentation and tutorials were more about laying out an infinitely complex decision tree – that is, git itself is easy to do anything with, but you can do a lot with it, so it comes across as unnecessarily complex.

Github has been really good about making really common paths really easy so that you naturally fall into workflows that minimize the work required to contribute to open source projects down to the point where you can clone a repo, branch, edit some stuff, and notify the maintainer of a project in just a few clicks on the web site.

Overall, it’s been a pretty good year. Just three more until my real anniversary.

Making Use of Caps Lock

2009-02-09T00:00:00-08:00

Making Use of Caps Lock

If you’re like me (and who isn’t), the caps lock key is an annoying waste of plastic. Its only value seems to be to type things to offend people. Luckily, most operating systems allow you to map it to control, or another useful key.

As a fairly new emacs user (and a long-term shell user), having a control key near where my fingers already makes many things far more accessible to me. Highly recommended.

But there’s another thing that the caps lock provides that quickly moves from annoyance to useful feature:

Just about every keyboard ever made has a caps lock indicator. Such a wonderful thing when used correctly.

Amit Singh over at google had a blog post about manipulating keyboard LEDs which inspired me to add this feature to my buildwatch app pretty much immediately.

Due to a fairly dumb bug I fixed today, it hasn’t been working (and I wasn’t paying attention to it anyway), but now, when anyone does a build against my build farm and the build breaks, my keyboard light will come on.

Sort of makes me want to write some bad code.

Buildbot

2009-01-30T00:00:00-08:00

Buildbot

I’ve used a few different continous integration systems, but buildbot has been my favorite for quite a while. It’s got a really nice architecture, a great codebase, and all the tools I need.

Most of these things seem to assume that if it builds anywhere, you’ve done your job. buildbot assumes there may be a relatively large number of build workers and an even larger number of configurations.

For example, I’m building out a buildbot configuration for a project I’m working on now for some portable software that seems to be most popular on Linux, possibly followed by OS X. However, depending on the distribution, toolchain, architecture, and compile-time options, some things just don’t work correctly. I also use it on FreeBSD, but having added a slave for FreeBSD, I found a small compiler error.

TBYB

Most CI systems are all about telling you when you’ve committed code that breaks the build for other people. Isn’t it rather late by that point?

buildbot has a try command that allows you run a complete build across whichever nodes you want (or all) before making your code available to anyone else.

One of the guys I’m working with on this project does most of his work in Solaris. He wrote some code, tested his code, and sent me a patch. I committed his patch to my local git repo, but before pushing it, ran buildbot try to make sure nothing weird happened. There were two different problems that caused build and/or test failures on every OS that wasn’t Solaris.

I was able to fix up his changes so that they worked everywhere, and they never actually made it into a public tree in their broken form.

The Code

buildbot’s codebase has some very robust plumbing, and it seems to support just about anything you might want to do (which other systems allow you subscribe to the tail of the current step’s log in realtime without having access to the slave?).

I’ve had to make some changes to get some features working as I expect, or fixing bug in edge cases, though.

I’ve been doing some work lately with the git support in try. Rather than repeating myself, you can see what I’ve done in my portion of the changelog and imagine how much better your project would be if every potential change could be tested across all your supported platforms before you publish.

commit dfb18e6c177d490da9dcab29e431eff22cfedfec
Author: Dustin Sallings <dustin@spy.net>
Date:   Wed Jan 28 20:26:00 2009 -0800

    Allow users to specify the remote git branch.

    This allows for a case where someone has a repository that tracks
    someone else's repository, has arbitrary local branches, but wants to
    run tries with the delta from the reference repository (i.e. the one
    the master knows about) to the local changes.

    Without this, it's likely the reference repository will not have the
    necessary objects to pull down a base revision to be able to apply
    patches for the try to succeed.

    This also ensures that the current client's view of the reference
    repository is honored.  That is, if the reference repository has moved
    forward, the trier's current tip of the remote is used to compute the
    delta, and that's sent along as the baserev.

commit 38a9c7fc719b44e2cdfa47884182da7128b369d2
Author: Dustin Sallings <dustin@spy.net>
Date:   Wed Jan 28 16:30:08 2009 -0800

    Added --dry-run (-n) support to buildbot try.

    Need to be able to try try when I just want to know what it's even
    going to consider doing.

commit f43143835cba3ca5963e07874da17c1416a031c2
Author: Dustin Sallings <dustin@spy.net>
Date:   Wed Jan 28 08:34:04 2009 -0800

    Refactored try buildName validation for reuse.

commit a88238cae5000c3481877aa354e3c76fc45770b8
Author: Dustin Sallings <dustin@spy.net>
Date:   Wed Jan 28 08:25:52 2009 -0800

    Don't require a list of builders for buildbot try.

    This maintains the current restrictions around builder lists that
    prevent one from trying a build that isn't in the list, but allows the
    user to delegate the selection to the server by not listing the
    builders at all.

    I want my users to always try their builds on every build
    configuration, but I don't want to be sending out buildbot options all
    the time.
commit 99240ada38677a143971fe390beb714c3017c20b
Author: Dustin Sallings <dustin@spy.net>
Date:   Sun Jan 25 10:47:57 2009 -0800

    git_buildbot should show the author, not the committer

    When I'm looking at my waterfall, I'd like to see the names of the
    people who wrote code, not just mine because I happened to have
    cherry-picked or am'd a bunch of changes.

commit 2c6865d83e967ca135acd3810e08af2dfab727b3
Author: Dustin Sallings <dustin@spy.net>
Date:   Wed Jan 21 20:23:35 2009 -0800

    Look at the remote tracking branch in git for buildbot try.

    This allows us to try committed, but not pushed code.

commit a079d84d4056dbf5ab3489cb7f2f8f0e20d91b87
Author: Dustin Sallings <dustin@spy.net>
Date:   Thu Jan 22 09:59:59 2009 -0800

    Try to reclobber on retry.

    On a failed git update in clobber mode, I was getting the following
    error on the second try:

    exceptions.OSError: [Errno 17] File exists: '/path/to/build'

    It seems that the clobber only occurs once, and any error that happens
    during the checkout should redo the clobber.

commit 6c36579a63b58bc986ec56e0272362038be08112
Author: Dustin Sallings <dustin@spy.net>
Date:   Wed Nov 19 04:56:38 2008 +0800

    Get rid of git- commands in git_buildbot.

    Signed-off-by: Dustin J. Mitchell <dustin@zmanda.com>

commit ac70a83fa05c2b1b31dd9411ffc28876fb9e9f20
Author: Dustin Sallings <dustin@spy.net>
Date:   Sat Apr 19 08:40:20 2008 +0800

    Send merge changes from git.

    Signed-off-by: Dustin J. Mitchell <dustin@zmanda.com>

Publishing Changelogs

2009-01-17T00:00:00-08:00

Publishing Changelogs

A user filed a bug against my memcached client because he couldn’t find the changelog and wanted to know what went into the new version.

I have a decent structure around releases, especially with this project. I tag it and write a good summary of changes in the tag including an abbreviated shortlog output, then I send the same out to the mailing list.

Somehow, I expected anyone not on the mailing list to just dig through my tags to find out what’s changed. I suppose that’s asking quite a bit.

Since I’ve been keeping good information in my tags since moving over to git (which actually has proper tag objects), I’ve found it quite easy to automate this process. My new git htmlchangelog takes a list of tags and generates a reasonable changelog automatically from this.

For example, the following command:

git htmlchangelog `git tag | egrep -v pre\|rc` > changelog.html

created the changelog for my memcached client.

Visualizing Git Contributors

2009-01-16T00:00:00-08:00

I was looking for something quick to do today, so I started drawing pie charts showing who commits to various projects.

Pie charts are particularly terrible for communicating something useful to people, but they kind of look nice, so whatever.

Here are some examples:

Linux

Git

Memcached

Emacs

Rails

Git Timecard

2009-01-11T00:00:00-08:00

Git Timecard

I really like github’s punch card feature. It’s a nice way to quickly see when a project is worked on.

However, it’s very limited. You can only see everyone’s work on the master branch. I have lots of other ways I want to look at my repos (including some that aren’t on github).

So I wrote my own.

Here are the neat kinds of things I can do now:

Punch Card of my Work Repo

~/work-project/ % git timecard

Alternate Branch

~/memcached % git timecard rewritten-bin

One User’s Work

~/memcached % git timecard --author=dustin

Last Week’s Work

~/twitterspy % git timecard '@{1 week ago}'..

Git Reroot - When Rebase is Too Gentle

2009-01-06T00:00:00-08:00

Git Reroot - When Rebase is Too Gentle

The fun thing about git is that it’ll do whatever you tell it.

Many newcomers look at is as this really complicated beast that is impossible to understand, but the less resistant users find that it’s very happy to just sit back and happily do whatever you ask of it (even if you ask it to do something stupid).

Recently, I was working on a project, and wanted to rebase a branch that had drifted quite a bit away from the master branch. rebase itself wasn’t getting me anywhere due to various conflicts from some partial merges and manual merges.

As an attempt towards a solution, I created git reroot.

What Does it Do?

git reroot is very similar to rebase conceptually, with one subtle detail – rebase works by rewinding to a merge point and replaying deltas (while dropping duplicates). reroot works by taking a range of commits and placing the commits at the end of the current HEAD by exact tree state.

The distinction is subtle, but important. git does not record changes, it snapshots tree states with some additional metadata. Commit deltas may be computed between any arbitrary trees, so the representations you often see are these deltas.

When Should I Use It?

Quite likely never. It was not appropriate for the project for which I created it.

However, if

you find yourself with a branch that has diverged too far,
you consider the result of this branch to be the desired state, and
it’s OK to think of the commits as snapshots of work instead of changes to previous state,

then you may find reroot helpful.

How Do I Use It?

The invocation recipes are different from that of rebase because it’s more of a “do what I want” kind of tool.

In a really simple case, let’s say you have a branch new-development that diverged from master a while back. Some work has been done on master, but you really just want new-development to be master. For whatever reason, you don’t want to do a merge to get it there, and rebase fails you due to conflicts you really don’t care about.

You would invoke reroot as follows:

git reroot master..new-development

You should see some output that’s showing you progress, and then a line that looks like this:

The newly created history is available as 2015200[...]

This command is completely non-destrutive, and will not affect any ref, so it’s safe to do whenever and wherever you like.

This output is telling you that the new tree is available, but not linked. You may use log (git log 2015200) to examine it, and when you’re ready to overwrite the current ref:

git reset --hard 2015200

If you look through the deltas (git log -p), you may see some changes that are much larger than you’d expect (especially towards the beginning, or any merge points), but at any given commit, the source tree is guaranteed to be in the exact state it was in when the author committed it.

Git Archaeology

2008-12-31T00:00:00-08:00

Git Archaeology

I just spent a while reconstructing the history of my code junk drawer. It’s on its fourth revision control system now (cvs → tla → mercurial → git) and has been through a lot of different tree states.

CVS really only versions files, but allows you to arrange things into a hierarchy, so I had a natural hierarchy and reflected it in a similar way in CVS.

Gnu arch favored smaller repositories, so when I did the conversion from CVS, I broke the snippets down into several different “branches” and versioned each language independently. I had one container branch that had a build config that would recreate the tree. This codebase lived through three different archives (repositories) and some of the individual snippets had a couple versions within that.

Once I started using mercurial more, I needed my snippets with me, but mercurial didn’t have a similar mechanism for managing a collection of repositories (even today, the forest extension is not distributed with mercurial). I had attempted to use darcs to reconstruct a single tree with full history but the trees renamed, but darcs wouldn’t ever complete with a subset of what needed to be converted. I ended up just snapshotting what was in gnu arch and dropping it into a single mercurial repository.

Having moved into git, I finally have the tools to actually put the history back together correctly. By “correctly”, I mean I wanted a single repository with all of the changes in it ordered chronologically (the order in which junk was placed in the drawer) without lots of weird merges that didn’t actually happen. I also needed to dig up all of the history prior to the snapshot I took for mercurial and get it all in place.

Bringing up Snippets

Just to add to the complexity story, keep the following in mind:

After tla, code was committed into mercurial from a snapshot.
That snapshot was (cleanly) converted to git, and more code was committed there.
One failed archaeological excursion had a few commits as well.

I started by going to the latest gnu arch versions of each snippet set and converting them to git repositories (by way of mercurial – but that’s a different story).

Setting up the Repo

I created a repo with a single empty commit in it as the eventual root of all of the other repos.

Once each repository was converted to individual git repositories, I brought added them as remotes to the conversion repository. Each branch needs to be considered related in order to facilitate the eventual merge, so I created grafts that placed the root of each branch atop my empty commit using the following script:

#!/bin/sh

empty=6c417dd379ccdb46de57e7a3860379633c270c9e

for b in "$@"
do
	oldest=`git rev-list --reverse $b | head -1`
	echo "Grafting $b"
	echo "$oldest $empty" >> .git/info/grafts
done

This was run for every remote repo and then each branch was run through git filter-branch to place the changes atop the empty branch in a real history.

Rewriting Tree Structures

These weren’t quite ready to merge just yet. Before I could even consider an actual merge, I needed to modify the tree structures (e.g. take all of the stuff at the toplevel of the eiffel directory and move it under an eiffel/ directory). The previous excursion had done this using a recipe I’d found on the internet somewhere which worked, but did the wrong thing with my version of sed. Using gsed cleaned this up.

For each remote branch, I’d run the following filter:

#!/bin/sh

git filter-branch -f --index-filter \
        'git ls-files -s | gsed "s-\t-&eiffel/-" |
                GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
                        git update-index --index-info &&
         mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' $1/master

Note: Out of pure laziness, I would edit this script for every invocation and then run it for a single remote.

Doing the Merge

The merge was really rather exciting. The image to the left shows a 24-way octopus merge.

That is, after grafting the empty changeset to the bottom of every branch, they now had common ancestry, making a merge possible. Since each branch got its paths rewritten all throughout history, there was no chance of conflict.

So enter the octopus.

Performing a Linear Rewrite

As cool as it was to do a massive octopus merge, I wanted linear history.

It would be possible to produce a graft file to place each change atop a single parent, but that seemed quite hard.

The strategy I employed was to dump the entire history using git format-patch and then write a script to rename all of the patches to be in chronological order so I could use git am to reconstruct the tree.

So I created a new branch from “empty”, and ran git am for a while. A nice bonus is that git apply strips off trailing whitespace for me, so the changes were slightly cleaned on the way in (I could’ve disabled that, but I rather liked it).

Removing Emptiness

I no longer needed the “empty” changeset after git am was complete, so I had to get rid of that. The root node is generally a bit difficult to touch, but I sort of guessed that I could add a graft of a hash without a parent and it’d make that change the new root.

So another trip through git filter-branch and I’ve now got a pretty decent set of history up throgh the snapshot that was taken for the mercurial conversion.

Catching up to the Present

So now that I’ve got everything up to the snapshot, what do I do?

I had a lot of options here – cherry-picking, grafting, format-patch. I think I went with format-patch arbitrarily. Basically, I did a git format-patch of the full history from the latest git repo and applied those changes to the newly created one.

Verification

So now that everything has been all hacked up and history is rewritten and changests grafted, etc… how do I have any idea whether it’s even close to where it was before?

This is where git’s content tracking stuff really saves the day. With the git repo I’ve been using as a remote, I can do a simple diff across the trees from the latest branches (and various other states). The only differences I saw were some new scripts/etc… had been added.

All’s well. I certainly learned a lot.

Using Git Alternates

2008-12-30T00:00:00-08:00

Using Git Alternates

Now that you’re happily using github sync to pull down all your repos into local bare trees, you may want to free up a bit of disk space from duplicate objects (about 120MB for me).

git has a way for multiple repos to share object space by way of alternates. You can read more about alternates in the repository layout documentation, but essentially it’s a text file that contains the location of another objects directory from which objects may be fetched when needed.

Example:

Let’s say you’re me and have checked out my photo album. You’d end up with a .git directory that looks like this:

dhcp-39:/tmp/photo 599% du -sh .git
 18M	.git

By setting up an alternate using my git alternate command:

dhcp-39:/tmp/photo 600% git alternate ~/prog/github/photo.git
.git/objects -> /Users/dustin/prog/github/photo.git/objects

You can then gc and free up gangs of disk:

dhcp-39:/tmp/photo 601% git gc 
Nothing new to pack.
Removing duplicate objects: 100% (256/256), done.
dhcp-39:/tmp/photo 602% du -sh .git
144K	.git

From 18MB to 144KB, and everything pretty much works as it did before.

You don’t need my git alternate command, for that, of course, but it makes it a bit easier when you’ve got a lot of them to do.

Using Github Sync to Track Your Projects

2008-12-29T00:00:00-08:00

Using Github Sync to Track Your Projects

When github announced their API, I very quickly threw together a python implementation.

I didn’t end up doing very much with the project as a whole, but I did write one tool in here that I end up using quite a bit: githubsync.py.

githubsync.py takes a github username and a directory and make sure I’ve got a local copy of every public repo that user has on github.

Grab the repo and try it out:

git clone git://github.com/dustin/py-github.git
cd py-github
./src/githubsync.py dustin /tmp/dustinatgithub

Once that finishes, you will have all of my current public repos in /tmp/dustinatgithub and if you run it periodically, you’ll see new repos I add appear while the existing ones are being updated.

But what about private repos, or even repos that aren’t on github?

The file ~/.github-private is read as a tab-delimited list of repos and their sources and those will also be synchronized. For example:

cool-stuff	  git@github.com:dustin/cool-stuff.git

With that in place, the cool-stuff repo will be created and synchronized along with all of the stuff found through the API.

Wasted Time Developing for iPhone

2008-12-26T00:00:00-08:00

OK, everybody’s written about this, but I just wasted a bunch of time making an iPhone app.

I don’t actually feel too bad about it because it was a pretty stupid iPhone app, anyway, but I’m not going to finish it because my first attempt to run it outside of the simulator was going to cost me a hundred bucks.

The application is an iPhone port of my twister app, but with worse graphics and sound (though the sound is at least potentially better). It’s functional enough to play a few games, but not fully polished.

I was hoping I could stick it on my daughter’s iPhone so she could play, but doesn’t seem to be the case.

If anyone wants to do something with it, it’s over on github.

Of course, I’d love to find out I was wrong and I can actually run my own program on my own phone without paying more…

pfetch

2008-12-26T00:00:00-08:00

About pfetch

For a long time now, I’ve had various cron jobs running to fetch various web resources with which I’d build out parts of my own site, or supply myself with custom RSS feeds after a pass through xsltproc.

This mostly worked OK, but there were a few things wrong with it:

I had to be careful to avoid putting stuff in place upon fetch failure.
Fetch failures would send me email unless I put effort into avoiding that.
Network timeouts would cause cron jobs to start piling up.
I’ve actually had cron get sick of running my jobs and just stop altogether.
Various jobs that ran at various frequencies would be in various scripts and hard to keep up with.
Running through cron means all jobs start at the exact same moment in time, thus are more likely to cause strain on web servers (if everybody does it).
Conditional gets require cross-invocation state to be stored (though I wrote a tool for this).
Sequential processing meant the whole thing took longer.

After a while, the problems added to enough of an annoyance that I decided to do something about it, so a couple months ago I started pfetch.

pfetch is a simple twisted app that does scheduled parallel http requests and optionally runs scripts after successful execution.

Given a list of URLs each with a destination, frequency, and optional (with arguments) to run after each successful (200) response, each URL will begin a fetch cycle starting at a random offset from the start time and loop on the defined interval.

Twister

2008-12-25T00:00:00-08:00

Twister

So, on Christmas, my kids decided they wanted to play twister. They wanted me to spin the thingy and call out moves for them. That got really boring after about five minutes.

I wrote a really simple python script to start calling the moves for me since the spinny thing was getting annoying, and would sometimes end up pointing between two colors or otherwise be too difficult to call.

The first version of the script looked like this:

#!/usr/bin/env python

import random

if __name__ == '__main__':
    colors=('red', 'green', 'yellow', 'blue')
    limbs=('left foot', 'right foot', 'left hand', 'right hand')

    print random.choice(limbs), random.choice(colors)

That was fine, but I still had to run it and call it out. Then I remembered that someone made a talking cat for OS X. All I needed to do was run the output of this thing through that, and there’d be speech and then I could go about my business and let the computer call moves for them.

I thought that was kind of cool, but wanted something a little…more. I ended up writing a full OS X desktop version complete with images, icons, a preference pane, etc…

The kids finished playing (using the prototype) long before I finished writing the app. It was fun for all of us, though. :)

If anyone wants to play a two-player version of twister, though, you can grab a copy.

Download

Version 1.1

Moody Bots

2008-12-24T00:00:00-08:00

Moody Bots

Twitterspy is a rather brute-force way to achieve xmpp functionality for twitter. It makes very heavy use of twitter search to provide track-like functionality to end users.

I’ve noticed in watching the logs that I often will get more errors in attempting searches than successes. This was at least the feel I got from looking at the logs. I wanted a way to communicate this to look at this information via xmpp.

Initially, it seemed like status would be a good way to do this. Currently, the status is used to show stats on how many users and queries the bot knows about. This is already a little weird, and I wouldn’t want to try to shove too much stuff into it.

Next, I thought about using the vcard for it. The bio is a fine way to describe such things. That wasn’t quite right, either. The bio is a better general description of the bot, and not so much status.

Then I discovered XEP-0107 – user moods. User moods in a PEP transport provides exactly the kind of thing I’m looking for.

Twitterspy keeps track of how many of its searches are successful, and how many fail. When many searches are successful, it’s in a good mood, when few are, it’s in a bad mood.

I had my kid look through the XEP to come up with some rules for how to select a mood based on how successful recent searches are. I’ve applied many of her changes, but some still require me to keep a bit more state than I do currently. It’s kind of an exciting thing, though few people will ever actually see it.

The pubsub mechanism will hopefully show itself to be useful, though. I’m hoping to do something cool like have a web status showing moods and all. Ralph Meijer’s moods page is quite inspirational here – as long as I’m capturing the data.

For the rest of you out there: Bring your XMPP services to life. Show their moods.

Building Your Site Connectivity

2008-12-24T00:00:00-08:00

Building Your Site Connectivity

When I started building this jekyll site, I thought it’d be nice to link to all the other places I leave junk around the internet. Rather than manually building a list, I took a bit of time to write something to do it for me using the google social graph API.

I made a simple web form to do this that generates HTML source so it can be further hand-edited if needed, but more importantly, so that I can actually paste the results into my github page and have it actually count for site connectivity.

If you don’t maintain a list of links on your own page, you may find it helpful to link to your friendfeed account.

For example, you can see how friendfeed links me. Change the username from dlsspy to yours for results that make more sense to you.

As it’s just a simple chunk of HTML, I’ve created a gist to house it for now. If you’d like to change this for the better, do it there and let me know about it.

I hope someone (else) finds this useful.

Trying out Jekyll

2008-12-23T00:00:00-08:00

Trying out Jekyll

Since Jekyll seems to be all the craze, I figured I’d give it a shot and see if it solved any problems for me.

So far, I like it.

Dustin Sallings

Hello World in Go -- A Memcached Server

Hello World in Go – A Memcached Server

ZFS for MacOS X

ZFS for MacOS X

Github Project

Downloads

Update:

Memcached Report Card

Memcached - Report Card for an Open Source Project

Release Scheduling

Bug Status

So What’s the Grade?

EMemcached

Why an Erlang Memcached Implementation?

But Wait, There’s More!

The Fun Part

The Storage Server

spymemcached Optimizations

Introduction to Quiet Operations

Other Quiet Operations

Optimizing with a Quiet Set

What You Need to Do

Tornado on Twisted

The Problems

The Solution

Buildbot and Git Repositories

Buildbot and Git Repositories

See Also

Memcached 1.4.0

Memcached 1.4.0

The Binary Protocol

Go Try It

Packages

Use the Source

Deploying on Amazon EC2/AWS?

US

EU

Project Skyscraper

Project Skyscraper

translate.skyscraper.im

The Input

The Output

conference.skyscraper.im

My Github Anniversary (Sort Of)

My Github Anniversary (Sort Of)

Making Use of Caps Lock

Making Use of Caps Lock

Buildbot

Buildbot

TBYB

The Code

Publishing Changelogs

Publishing Changelogs

Visualizing Git Contributors

Linux

Git

Memcached

Emacs

Rails

Git Timecard

Git Timecard

Punch Card of my Work Repo

Alternate Branch

One User’s Work

Last Week’s Work

Git Reroot - When Rebase is Too Gentle

Git Reroot - When Rebase is Too Gentle

What Does it Do?

When Should I Use It?

How Do I Use It?

Git Archaeology

Git Archaeology

Bringing up Snippets

Setting up the Repo

Rewriting Tree Structures

Doing the Merge

Performing a Linear Rewrite

Removing Emptiness

Catching up to the Present