For the last few weeks, I've been coding our live broadcasting platform Mixlr into shape, ready for its big launch next month. But with the amount of distractions available on the internet, it's easy get pulled away. One minute I'm coding, the next I'm reading an article on Hacker News. I don't even know it's happened until it's too late: muscle memory takes over. Somehow, I subconsciously hit CMD-TAB, CMD-L, down, return and there's an interesting article that simply must be read.

It's a killer.

So yesterday I made a solution. Sometimes I edit my /etc/hosts file to make news.ycombinator.com, Twitter, Facebook and so on resolve to 127.0.0.1, and therefore be unreachable. But it's a hassle to manually edit the file again and again.

Enter the noprocrast gem. Noprocrast simply automates adding and removing addictive websites from /etc/hosts. To install it, simply:

Check whether noprocrast is doing anything:

Edit the list of hosts which noprocrast will block:

Activate noprocrast (start blocking sites):

De-activate noprocrast (stop blocking sites):

Blocked hosts are stored in a file called .noprocrast in your home directory. Just in case, noprocrast makes a backup of your hosts file, to /etc/.hosts.noprocastbackup, the first time it runs.

The gem has been built and tested on Mac OSX, but should run fine on Linux. Finally, if you want to improve or expand the code, clone it at Github.

Happy efficient coding.

Follow @rfwatson on Twitter

Mixlr: high-quality audio broadcasting platform launches 2nd December.

Today, I'm inviting readers of Hacker News to sign up for Mixlr. 

I'm a big fan of the concept of Minimum Viable Product. Take an idea, think it through and identify the most fundamental characteristics which can be implemented to get real people using your product. Then, get feedback and move forward iteratively. Keeping the MVP in mind keeps you close to what your real users want to do, and helps to minimize time spent developing features which just aren't used. It's the formula I've followed whilst developing my live music sharing platform Mixlr, and for the most part it's worked great. 

 

Most people are now familiar with the concept the newsfeed. Recently, I decided to implement newsfeed functionality on Mixlr, my embryonic mixset distribution service. This post shows how I have used Redis to create a fast and efficient newsfeed implementation. 

What exactly should a newsfeed do?

First of all, let's define what to expect from a newsfeed implementation.

 

It should log every action, of every user. On Mixlr, users can currently upload, rate and share mixsets, and follow other users. The newsfeed should store a single stream of all of these actions. We shall call this stream the master newsfeed: every action carried out by every user, anywhere on the site.

 

Allow actions to be grouped by perspective. In real-life use, certain subsets of the master newsfeed will be more useful than the entire stream. For example, all of the actions carried out by Fred, which will be displayed on Fred's profile page. Or all of the actions carried out by users who Kevin follows, which will be displayed on Kevin's home page. Or even all of the actions performed by Fred, Kevin and all other users on a particular mixset. It should be possible to efficiently retrieve newsfeeds from all of these perspectives.

Generate arbitrary data formats. It should be possible to generate different formats of data from any given newsfeed perspective. For example, I might want to make the feed of Fred's recent actions available as XHTML in a mobile client or Facebook application, HTML in a normal web page, or JSON as part of an API response. This quickly rules out caching pre-rendered HTML as an implementation.

Be fast. Really fast. Mixlr is built for speed. This means that whatever implementation is chosen, there must be essentially no increase in the time any individual page takes to load. This should hold regardless of the perspective being retrieved, the data format being generated, or the amount of news "items" being requested.

 

A naive newsfeed implementation might try to join whatever database tables it needs to retrieve the newsfeed data, every time the page is requested. The problem is that in a typical database schema, such a request will require a large number of tables to be joined. As the number of users on a website grows, the amount of data being examined increases at a rate far enough exceeding O(n) that performance falls away quickly. In short, if you want your web application to scale, then this is not a good way to implement newsfeeds. 

 

There are a number of strategies which help to ease this problem (all of which are outside the scope of this post). I have approached the problem a different way: remove SQL from the equation entirely.

 

Enter Redis.

Redis is a key-value storage system which is typically configured to be semi-persistent. In Redis' case, this means that data is held in memory and saved at regular intervals to disk. This is perfect for implementing a newsfeed, because in the worst case scenario -- the server crashes without warning -- all that happens is that the last few minutes of newsfeed entries are lost. For Mixlr, and many other websites, this is an acceptable risk. The chances are, no user will even notice. And when everything does work as it should, all newsfeed data will be held in memory and retrieving it is really, really fast.

 

My implementation makes use of all of Redis' basic datatypes. A typical newsitem, representing a single user action, looks something like this:

 

 

First I create an MD5 hash of this newsitem, minus the timestamp. If for any reason the application attempts to add a duplicate newsitem later on, the MD5 for this item will be identical to the first, and can be easily detected and discarded. Using the MD5 as a key, I store the newsitem in Redis as a hash datatype (a JSON-encoded string could also suffice).

I can now retrieve the original data from memory at any time by using the MD5 as a key. (I set this key to expire in 7 days, which means Redis will clear it from memory automatically. Who's interested in old news, anyway?)

 

Now it is time to represent the different newsfeed "perspectives" I want to retrieve. Each perspective relates to a particular view of the newsfeed. These perspectives are simply implemented using ordered Redis lists, one for each perspective. Each list is full of MD5 keys, each one pointing to a newsitem value. 

 

In Mixlr, I maintain the following perspectives:

• Master news feed (every action carried out site-wide)
• User news feed (every action carried out by individual users)
• Instance news feed (every action carried out to an object)
• Friend news feed (every action carried out by a user's friends/followed users)

So in the above case, I insert the MD5 key into four Redis lists:

• the master news feed 
• User #1 (the user doing the rating)
• Mixset #100 (the mixset being rated)
• The friend feed of every user who follows User #1

Retrieving any newsfeed is then a simple matter of querying for the first N entries in the relevant list, retrieving the corresponding JSON-encoded hashes from Redis and rendering some appropriate HTML or other output. 

 

The trade-offs

 

In any real-life architecture, there are trade-offs made between speed, storage type and memory usage. In this case, memory and disk space is being compromised for the sake of speed of access. Retrieving any type of newsfeed is virtually free, which means that page loading speed will not be affected. Writes are still very fast, as there is no disk or SQL access involved. Although memory usage is relatively high, the minimal amount of data which is stored, and the use of MD5 keys to avoid unneccesary data replication, help to keep it within reason. Additionally, appropriate use of Redis' automatic key expiry settings, and a regular cronjob pruning of old lists, will help even further.

 

Conclusion

Using Redis has allowed me to implement an elegant and efficient newsfeed functionality in a way that would have been difficult or impossible until recently. I can be confident that every newsfeed I retrieve will have no impact on performance. If you need to implement similar functionality then I recommend you consider an implementation using Redis.

 

If you liked this post, then follow me on Twitter. You can also sign up as a Mixlr beta-tester here.

 

Further reading: How I implemented Mixlr's backend using NodeJS.

 

This article is a primer on using monit to monitor unix processes. Specifically it shows how monit can be used to improve the stability and availability of a Ruby on Rails web application. All of the techniques discussed would be equally applicable to monitoring almost any other type of Unix process.

Recently I've been working on a new web application. One of the features allows users to upload audio files to the application. This is the story about how tackling an unforeseen bug led to a massive improvement in the application architecture, and an appreciation of a whole new approach to building web-apps.

First of all, let's get clear what's going on in this particular feature. The flow looks something like this:

It's pretty straightforward: the user selects a file, and while they're filling in some additional details and watching a progress meter, the file is being uploaded, processed on the application server and then saved to Amazon S3. Once all the boxes are ticked, everybody can move on.

Now a little bit of background. I've got previous in building websites similar to this one. Last year I launched DonkDJ.com, which revolved around almost the same principle. The user chooses a file to the DonkDJ app servers: then, while they watched a progress meter and (occasionally) clicked on some ads, the song was uploaded, rhythmically analyzed, mutilated usually beyond recognition and returned to the user.

At the time, I used what was probably the latest web technologies to achieve this. I used Starling as a message queue, and a batch of Workling worker processes which handled the processing jobs as they appeared on the queue. Basically, the server-side flow went something like this (I've missed out the UI part this time):

Now there's nothing fundamentally wrong with this approach, and it certainly worked. DonkDJ turned out to be a viral media smash hit, and within 24 hours of launch it was processing up to 5,000 uploads per day, without ever really falling over. Not bad for a 1Gb Slicehost server. 

There are, however, three big disadvantages to this setup:

• It should be clear from the diagram above that architecture is serial. First of all, the complete file has to be uploaded to the server. Then (assuming a worker process is available immediately) the file is processed. Then after it's processed, it's transferred to S3. Finally, the UI can be informed that we're ready to move on. This isn't necessarily the most efficient way to work. What if some of the processing tasks could take place in parallel? That could mean a much shorter wait, a better user experience and quite possibly a resulting higher conversion or repeat-visit rate.
• The communication between different threads (one handling the UI, the other processing the file) was complicated. This wasn't helped by the fact that while making a request from the UI to the server was just a simple AJAX request, it was difficult-to-impossible to send a message in the other direction, from the server thread to the UI. This meant that constant polling, and a lot of SQL attribute-juggling, was necessary to make the different sides communicate properly. Far from elegant.
• The design of Workling meant that the entire Rails environment had to be loaded into memory each time the application needed a new worker. Busy periods required upwards of 8 worker processes to keep up, plus 2-4 extra Rails processes to serve the website itself. Running 10-12 Rails processes on a 1Gb server is no laughing matter, especially when most of them are also processing large binary files.

Now web technology changes a lot in a year, but when I started work on my new project (which requires, as you'll recall, a broadly similar architecture), I was naturally inclined to go with the experience I already had. After all, it had scaled successfully to a relatively high load, and I can see nothing in principle which would stop it scaling further. Just create some EC2 or Slicehost instances with more worker processes and employ nginx or haproxy as a load balancer. And so off I went, building my application, and having quite a lot of fun.

The problem surfaced deep into the development process. As you can see in the first diagram, this application (unlike DonkDJ) asks the user to fill in an additional details form while he or she are waiting for their upload to be processed.

Paperclip (the Rails plugin I used in both applications to handle uploaded files) prides itself on handling file attachments as normal ActiveRecord attributes. In other words, you can do something very roughly like this:

This means that the attachment is processed and saved, all during the ActiveRecord callback cycle. Because Rails opens an SQL transaction every time an instance is saved, this means that I was faced with a flow that looked something like this:

In other words, when the user tries to submit their "additional details" form, then unless file processing is complete then MySQL is unable to save their information -- the database row is locked! Worse, there's no easily catchable exception: MySQL just hangs for 30 seconds and then times out. All this time, the user has a spinny AJAX wheel to stare at. This was a big, big problem.

So I weighed up my options. As I saw it, they were:

• Use an MyISAM table instead of InnoDB. MyISAM tables don't support transactions, and so Rails (presumably) wouldn't lock the row and the problem would be solved. Not really ideal though: I didn't have any need for super-blazing SQL speed, and I didn't feel inclined to sacrifice the benefits of InnoDB for the sake of a Rails plugin. Update: See janl's comment below.
• Hack Rails so that it didn't open a transaction on this particular save. Really, really far from being either elegant or ideal.
• Alter the UI so that the user wasn't able to save their "additional details" form until their upload had been completed. The easiest option but again, it didn't seem right to sacrifice so much flexibility because of the possibly bad unsuitable design of a Rails plugin.
• Create a proxy class that was instantiated in place of ActiveRecord::Base if the database was locked. Upon saving, the data would be saved not to the database, but to Redis, which of course is not affected by SQL locks. Upon being processed, the worker process would fetch the additional information from Redis and save it to the database. Yes, I really did implement this, but please: nobody should have to go to this much trouble to implement concurrency in a web application.

At this point I remembered about nodejs. I wasn't totally sure what it was about, but I needed to solve this concurrency problem, and I was looking for a good solution to implement an upload progress bar (hopefully) without using Flash. Why not take a look, I thought. 

It turns out that nodejs is phenomenally useful. It is, in short, a lightweight implementation of server-side Javascript which is perfectly suited to handling file uploads and background processing. Others have done fine jobs of describing its potential, but I will simply emphasise that its architecture is designed to be event-driven, rather than threaded. Just like programming javascript in the browser, that means that it is perfectly suited to designing a system which reacts to events as they happen, instead of waiting in a queue to fetch a job.

Over this weekend, I've totally re-written the backend of my project using nodejs. This is the new flow:

Here are the advantages:

• Whereas in DonkDJ I was using a Flash plugin to handle the file upload, using NodeJS means that I can get my hands on the data as it arrives, chunk-by-chunk. This means that simply by piping the data I receive into the file-system and back, I can process and transcode the audio as it arrives on the server. This means much quicker processing times and happier users.
• The only reason I can't also simultaneously upload to S3 is that Amazon requires an accurate "Content-Length" header before it starts uploading. Because transcoding changes the size of the file, I have no way of knowing this value until the processing stage is complete. However: what if for some reason the file doesn't need processing? Perhaps it's already compressed enough, or is too small to worry about bandwidth. In this case, I could use the "Content-Length" header of the original request to upload to S3 while simultaneously uploading to the server. (Alternatively, send a message back to the browser and ask it to upload to S3 directly using a Flash object, therefore saving the bandwidth cost, and time, of sending the file twice.)
• NodeJS is handsomely lightweight, handles lots of simultaneous jobs well, and doesn't load the Rails environment at all. Instead, I use Redis to communicate between Rails and the backend processes. This means that I will typically only need a couple of instances of Rails loaded, which is enough to serve a quite busy website if there's no background processing involved.
• I can remove Paperclip, Resque and all other background processing tasks from my Rails application entirely. Rails now handles serving dynamic web pages and nothing else. This is clearly the way it ought to be.

On top of the advantages of NodeJS, it is also perfectly suited to using Websockets, which I've employed to allow bi-directional communication between the worker processes and the browser. Because both server and client are using event-driven Javascript, this makes it wonderfully easy to implement features like upload progress bars and dynamic status messages, which in DonkDJ involved messy hacks, Rails controllers, Flash movies and who knows what else to achieve. (Check out web-socket-js for seamless crap-browser compatibility when using websockets).

So, in conclusion. There is no doubt that some of the advantages of NodeJS could have been achieved in DonkDJ with a better-designed architecture, but it would have been very painful to achieve. If you're looking to build a web application which handles any kind of file-uploading, you should look more into it. It's awesome.

Follow rfwatson on Twitter.

Today, I struggled for an hour to complete a difficult coding task.

The task -- to coerce an existing system to handle certain special cases differently -- was challenging for several reasons.

Firstly, the existing system was already rather complex. I wasn't overly familiar with the code, and so I had little intuitive feel for what was happening on the fringes of my attention. It was as if I was walking in an unfamiliar neighbourhood. Every avenue of code had to be navigated individually, and I could rarely afford my mind to wander from the road and consider the bigger picture.

(When you work with code you haven't written, it takes a lot more energy to navigate. You constantly have to look where you're going, which interrupts thinking about other matters, like how to fix or extend what you're looking at.)

Secondly, the system had been designed, built and deployed without consideration for this newly-required behaviour. In fact, by chance it had been implemented in a fashion which was actively opposed to its integration! Ideally, a large-scale refactoring was called for, but commercial pressure dictated that this was not the time to rip everything apart and start afresh.

(Although a total re-write would have taken longer, and carried a higher risk of introducing new problems, ironically it may well have taken less effort to achieve. It is easier for every developer to write new code from scratch, than it is to make a subtle and effective change to a wall of unfamiliar code.)

Eventually, as I grappled with the problem, I began to understand the best way to solve it. I was pleased because it felt like an acceptable solution. Not by any means a perfect one, but in the circumstances good. (A workaround, which is more than a mere hack.)

I tested it, and it worked -- almost. All the acceptance tests passed, except one.

This edge case (of an edge case) seemed to completely break my solution. I sat for another twenty minutes, my mind still swimming with detail and control-flows, becoming frustrated that my solution didn't actually seem to work at all. When I finally saw the solution however -- unlike the original problem -- it was just a simple, one-line fix.

Finally, everything worked.

Sometimes, dealing with complexity is unavoidable. After thinking in complex terms, making the mental shift back to thinking simply, doesn't happen instantly. It takes a bit of time, and then it happens of its own accord. It doesn't like to be forced.

Next time I will try to recognize when I need to make the shift, and make a cup of tea while I wait.

Why do I never call myself a hacker? It's a matter of communication.

Language usage differs through time and place, but commonly I hear the word hack used in three ways.

hacker n
Refers to an individual who is skilled at avoiding security systems, especially computer systems. The usage I hear most commonly. The connotations are invariably negative, ranging from dishonest to dangerous to plain misguided. Commonly used in the Western media as well as widely in public use.

hacker n
Less widely used. Also refers to an individual who is highly skilled, typically in computer programming or similar, but this time the connotations are positive. Draws on historical associations with activities such as phreaking, academic culture and more recently open source software.

hack v/n
Typically used to describe a quick fix to a problem, especially in programming, engineering and other fields. Although the connotations can be positive, I more often hear it being used -- and more importantly, understood -- negatively (a dirty hack, hacked something together, etc.).

Language usage is always difficult to pin down, but I estimate that negativity is either implied or inferred, at least 80% of the times that the word hacker is used.

If I started a business, I would never choose to name it with a word that is commonly used or understood to have a negative connotation. It would be commercial suicide before I'd even started. I extend the same logic to how I describe myself. I just want to make it as easy as possible to be taken seriously by those around me, whether it's my aunt, my boss or a client of my company.

I like test-driven development.

I like it because writing tests makes me stop and think before I start to code. I save large swathes of time by being able to recognise exactly what an implementation will actually achieve from the top down, before spending time and effort developing it in full from the bottom up.

I like it when, after making an innocuous change to some line of code, a failing test alerts me to a bug that would otherwise have taken me days or weeks to even know about, let alone fix. This happens at least several times per week, if I'm working full-time on a TDD project, which adds up to a lot of bugs avoided, which means much better software.

When I started writing test-first, I noticed the difference in the code and projects I produced immediately, and I've been TDD on my own projects ever since. It works, and I love doing it.

For the past few months I've been working on a large-scale Rails project which for a number of reasons, doesn't lend itself as easily to all-out test-driven development. So recently, I've been writing lots of Ruby code without tests, and that's made me better understand two, seemingly separate, things about TDD.

Firstly, writing test-first permanently changed the way I write code. Even without a looming test failure, the methods I write still expect to have to confront one. They still tend to be short, descriptive and as self-contained as I can make them, and as a result easier to understand and change. This is despite not necessarily writing any tests at all, and presumably just because I'm in the habit.

Secondly, I've realised that when I don't write tests I code too fast. TDD acts as a constraint on myself, something which stops me from running ahead of myself and forces me to stick around a particular problem area for long enough to understand it better, and often learn something that otherwise I'd have missed.

I don't remember ever reading a blog claiming that would be an advantage of TDD before I started. It's just a random side-effect, and this time unexpected. I didn't even notice it until the tests were taken away. Now I understand myself a little better, I can at least try to compensate, slow down and achieve the same result without the tests.

Both of these tell me that shaking things up, doing something a different way for a while and even just for the sake of it, is never a bad idea. You never know what you're going to take away from it.