Tuesday, December 6, 2011

Continuous integration - some tips

Continuous integration server is a must for every non-basic project.

Here are some of my tips for working with CI servers.


IRC, email, campfire - use what fits best for you. Ideally, the notification should come with some info about the code changes that triggered the build. We use IRC - we've got an IRC channel for every project we work on, so the notifications go to the correct channel.

Build time

It's a huge topic. It's so difficult to keep the build time short for every project. I have to admit we've got projects where build time takes 20 minutes. My goal is to get down to below 10 minutes for existing projects and try to keep the new projects below 3 minutes. Slow builds have a very bad impact on so many things. The feedback is late. People tend not to run builds on local machines.

Local builds

It's great that you have a CI server, but the goal is also that all tests can be run on every developer machine in a decent time.

Architecture decisions

That's the most surprising effect of using tests and CI. Once we got to the point where many of our projects have the slow build problem we investigated our technology choices. In many cases the slowness came from using Selenium. Selenium is the only "right" tool when you have an app that is a mix of server-side logic and client-side code (JS). It seems, surprisingly, that many of our projects could be turned into one-page CoffeeScript apps. Obviously it's not an easy goal (and not always possible). It's changing the way we architecture our apps. If the split is achieved you can test the CS code in isolation from the server-side and then test the server-side API, also in separation, but without Selenium.
Selenium is cool, but it has the drawbacks of being slow and unreliable.

Unreliable tests

If you're using Selenium, then you already know what I mean. I hate this fact but sometimes we've got build failures which shouldn't appear, they're most of the time timing issues. It's a hard situation to have. The important thing to remember is that we (the team) are all in it. Every team member should investigate the problem and try to fix them - sometimes adding a sleep (a hack, I know) solves the problem.

Who's responsible for failures?

I like the idea that the last person who triggered a failing build is responsible for fixing it. It's hard to get to this point, especially if your build is unreliable. This leads to:

Never leave a broken build

This is the biggest sin when working with CI servers. If I want to start coding and I see that a build is broken and no one seem to care then it's very depressing to me. In practice, it means that you should commit your changes some time before you finish your work, so that the CI can run it and get you the result back. If it's broken, try to fix that as much as you can. It's less of a sin when the failure comes from unreliability, but leaving a broken build caused by your changes is unforgettable :)


A green, fast build run frequently on the CI server is a sign of a healthy project. If there's anything broken with the build process, look at it. It's often a smell showing you that something can be wrong in your process. Don't ignore that.

Any tips from your experience?


Michael Foord said...

Can you keep your end-to-end functional tests (real database, using selenium) running in under 3 minutes?

Michael Foord said...

Ah, now I see your comment about Selenium being slow and unreliable. Your suggested solution is to separate out tests for the backend and client side. Does that mean you typically *don't* have end-to-end tests on a new project?

We've found the new WebDriver interface to Selenium 2 much more reliable by the way.

Andrzej Krzywda said...

Hey Michael,

The front-end acceptance tests (with a real browser, but the "server" is stubbed out) can take seconds, literally.

The server API tests can also be very fast. That gives us 2-3 minutes for the full-integration tests (with Selenium). It's enough for the most important parts of the code.

michal said...

I agree with everything you said here.

We've been using jasmine together with sinonjs (http://sinonjs.org/) to test our front-end.

We've been also experimenting with capybara-webkit https://github.com/thoughtbot/capybara-webkit to skip selenium all together.

Pedro Mata-Mouros said...

You mention "every non-basic" project. I happen to think there is a huge gap needing to be filled, in bringing CI practices to all those smaller to medium web projects. About 80% of all web projects, basically. I've authored Cintient, an open-source, PHP based, self-hosted CI server. It's made to be dead simple, and hopefully turn around this generalized misconception that CI is only justified on more complex or big projects. Oh, the link: https://github.com/matamouros/cintient.

Andrzej Krzywda said...

michal: We've been trying to convert Selenium tests to Webkit, but we failed. It didn't see some texts that were clearly visible on the screen. Did you have similar problems?

Andrzej Krzywda said...

Pedro: Yeah, I agree, medium projects should also have a CI. I think every project that involves more than 1 person would gain from CI. Single-dev projects also.

Interesting idea with you CI tool. I'm sure it will be useful!

We use Jenkins for our Rails projects.

Pedro Mata-Mouros said...

Thanks Andrzej!

Anonymous said...

About making sure people don't leave builds in a broken state, we stole an idea from something I read about on a NetBeans developpers forum. They use an ugly looking piece of pottery that acts as a trophy for the last person who broke the build. Quite effective! Not having any pottery on hand, we are using a Origami pig for that. The person gets to keep it on its desk until the next person breaks something.

Andrzej Krzywda said...

haha, a trophy, nice!

Wouldn't work well in distributed teams, though. Maybe changing twitter avatar to something embarrassing? :)