Continuous integration - some tips

Continuous integration server is a must for every non-basic project.

Here are some of my tips for working with CI servers.

Notifications


IRC, email, campfire - use what fits best for you. Ideally, the notification should come with some info about the code changes that triggered the build. We use IRC - we've got an IRC channel for every project we work on, so the notifications go to the correct channel.

Build time

It's a huge topic. It's so difficult to keep the build time short for every project. I have to admit we've got projects where build time takes 20 minutes. My goal is to get down to below 10 minutes for existing projects and try to keep the new projects below 3 minutes. Slow builds have a very bad impact on so many things. The feedback is late. People tend not to run builds on local machines.

Local builds


It's great that you have a CI server, but the goal is also that all tests can be run on every developer machine in a decent time.

Architecture decisions


That's the most surprising effect of using tests and CI. Once we got to the point where many of our projects have the slow build problem we investigated our technology choices. In many cases the slowness came from using Selenium. Selenium is the only "right" tool when you have an app that is a mix of server-side logic and client-side code (JS). It seems, surprisingly, that many of our projects could be turned into one-page CoffeeScript apps. Obviously it's not an easy goal (and not always possible). It's changing the way we architecture our apps. If the split is achieved you can test the CS code in isolation from the server-side and then test the server-side API, also in separation, but without Selenium.
Selenium is cool, but it has the drawbacks of being slow and unreliable.

Unreliable tests


If you're using Selenium, then you already know what I mean. I hate this fact but sometimes we've got build failures which shouldn't appear, they're most of the time timing issues. It's a hard situation to have. The important thing to remember is that we (the team) are all in it. Every team member should investigate the problem and try to fix them - sometimes adding a sleep (a hack, I know) solves the problem.

Who's responsible for failures?


I like the idea that the last person who triggered a failing build is responsible for fixing it. It's hard to get to this point, especially if your build is unreliable. This leads to:

Never leave a broken build


This is the biggest sin when working with CI servers. If I want to start coding and I see that a build is broken and no one seem to care then it's very depressing to me. In practice, it means that you should commit your changes some time before you finish your work, so that the CI can run it and get you the result back. If it's broken, try to fix that as much as you can. It's less of a sin when the failure comes from unreliability, but leaving a broken build caused by your changes is unforgettable :)

Summary


A green, fast build run frequently on the CI server is a sign of a healthy project. If there's anything broken with the build process, look at it. It's often a smell showing you that something can be wrong in your process. Don't ignore that.

Any tips from your experience?