I used to be a big fan of Cucumber. We've been using it in every project for our customers.
From my point of view Cucumber delivers 2 important features:
- A special language readable to non-programmers (Gherkin)
- Acceptance tests - embedded browser driver - Webrat/Capybara etc
Cucumber became a whole framework for managing requirements. The implementation is based on step definitions which we could implement in Ruby. Cucumber also gives us tools for running the acceptance tests, tools like tagging, features like documentation etc.
Thanks to tags we could run only the tests that are responsible for a specific part of the system. Another use case is to tag new scenarios with @wip and those are not run.
The biggest Cucumber project we have at the moment contains almost 600 scenarios. The biggest project I'm aware of has 2000 scenarios. It's a lot of "code".
With every new project I hoped that Cucumber can become one of the communication tools between our team and the customer. Only once I managed to get the customer write the scenarios, however the quality wasn't good and they had to be rewritten by a developer.
Every now and then we could send the scenario to the customer to get their approval that they are correct. In this cases the customer was able to read them easily. However, it was difficult to convince the customer to review the scenarios on a daily basis. They preferred writing a document or using a ticketing system like Redmine. The result is that we have requirements in 2 places - tickets and scenarios. It's not perfect, but it works well enough.
My conclusion for now is:
1. It's difficult to get the customer to write/read Cucumber scenarios
We gain a lot from the acceptance tests feature of Cucumber. Every functionality is covered by scenarios and it creates a nice regression weapon in cases where we need to change something in the application. Upgrading from Rails 2 to Rails 3 was one of such occasions. Unit tests would not be enough.
Recently I noticed that it's getting harder to manage the scenarios written the Cucumber way. There are many patterns how to keep the steps definitions, how to split the scenario etc.
I realized that it's the same kind of problem when you write a procedural code with no objects.
2. Cucumber scenarios most of the time are procedural programming.
With procedural programming it's hard to keep data and behaviour together. For simpler apps procedural programming is fine but for a bigger codebase I personally prefer object-oriented programing. I suppose it would be possible to include some object-orientation into Cucumber but I'm not sure if it would feel right and if it's worth the effort.
Summary
I see two problems with Cucumber and its usage in our projects:
It's hard to get customer working with scenarios and it's hard for developers to work with a bigger Cucumber codebase.
If you have customers working with Cucumber - congratulations! If you manage to deal with the procedural style of Cucumber - congratulations!
Is there any solution?
I started experimenting with a new way of acceptance testing based on Capybara and an object-oriented approach. It spread very quickly among the Wrocław Rails developers so it seems to be an idea worth investigating. I will present how it works in my next blog post. Stay tuned.
If you read this far you should
and subscribe to my RSS
10 comments:
We're in the process of looking for a solution other than cucumber - for us - we'd write tests ourselves anyway, so there's little advantage in using a DSL instead of a fully featured programming language.
Another problem was that getting complex cases to work (setting up elaborate test fixture scenarios) was way too time consuming.
Very interesting post. Good to know limitations of DSL for acceptance tests. Did you ever have a need to refactor more than several test cases at once? If yes - what are your experiences? (hard/easy)?
@jasiek:
"setting up elaborate test fixture scenarios"
It's a very important topic, I will try to address it in my later blog posts. In short, I'm not a big fan of fixturies/factories anymore.
@koppernickus
Thanks :)
I don't remember any large functional refactoring. Only the ones where we replaced something under the hood and have acceptance tests as the defense net.
I just commented on my use of Cucumber on another blog (http://www.rubyinside.com/dhh-offended-by-rspec-debate-4610.html#comment-43305).
The gist of it was that I don't use Cucumber for communicating with customers - I use it as the first stage in defining the task's requirements. These requirements end up as executable documentation that developers coming on to the project can learn from.
@Mark
I agree with thinking about acceptance tests as an executable documentation. I'm doing the same.
You also treat them as a documentation for new programmers, which sounds like a good practice.
You say:
'Writing a Cucumber feature is writing prose, not code."
This is where I have a different opinion. As koppernickus commented, it's a DSL for acceptance tests. It is parsed, it is executed, it is being refactored. It's code to me.
It's code that is procedural in its nature. This is what bothers me. I think that a readable Ruby code is better here.
Gherkin is definitely English (at least the way I use it). It's also a DSL, of course. But when I'm writing a Cucumber scenario, I'm definitely thinking in English. I do not refactor it, because it's just the procedure (as you say) that a user follows.
The steps that make the DSL work are a whole different matter, of course.
The only reason I'm still using Cucumber instead of something like Steak is pure laziness. I use the command line methods found in Aruba and have been too lazy to write my own for Steak.
@Mike
I haven't used Aruba. Looks interesting. It should be easy to rewrite it to Steak, under the hood it must be Ruby.
However, if Cucumber works fine in this case then no point in changing that of course.
Post a Comment