I used to be a big fan of Cucumber. We've been using it in every project for our customers.
From my point of view Cucumber delivers 2 important features:
- A special language readable to non-programmers (Gherkin)
- Acceptance tests - embedded browser driver - Webrat/Capybara etc
Cucumber became a whole framework for managing requirements. The implementation is based on step definitions which we could implement in Ruby. Cucumber also gives us tools for running the acceptance tests, tools like tagging, features like documentation etc.
Thanks to tags we could run only the tests that are responsible for a specific part of the system. Another use case is to tag new scenarios with @wip and those are not run.
The biggest Cucumber project we have at the moment contains almost 600 scenarios. The biggest project I'm aware of has 2000 scenarios. It's a lot of "code".
With every new project I hoped that Cucumber can become one of the communication tools between our team and the customer. Only once I managed to get the customer write the scenarios, however the quality wasn't good and they had to be rewritten by a developer.
Every now and then we could send the scenario to the customer to get their approval that they are correct. In this cases the customer was able to read them easily. However, it was difficult to convince the customer to review the scenarios on a daily basis. They preferred writing a document or using a ticketing system like Redmine. The result is that we have requirements in 2 places - tickets and scenarios. It's not perfect, but it works well enough.
My conclusion for now is:
1. It's difficult to get the customer to write/read Cucumber scenarios
We gain a lot from the acceptance tests feature of Cucumber. Every functionality is covered by scenarios and it creates a nice regression weapon in cases where we need to change something in the application. Upgrading from Rails 2 to Rails 3 was one of such occasions. Unit tests would not be enough.
Recently I noticed that it's getting harder to manage the scenarios written the Cucumber way. There are many patterns how to keep the steps definitions, how to split the scenario etc.
I realized that it's the same kind of problem when you write a procedural code with no objects.
2. Cucumber scenarios most of the time are procedural programming.
With procedural programming it's hard to keep data and behaviour together. For simpler apps procedural programming is fine but for a bigger codebase I personally prefer object-oriented programing. I suppose it would be possible to include some object-orientation into Cucumber but I'm not sure if it would feel right and if it's worth the effort.
Summary
I see two problems with Cucumber and its usage in our projects:
It's hard to get customer working with scenarios and it's hard for developers to work with a bigger Cucumber codebase.
If you have customers working with Cucumber - congratulations! If you manage to deal with the procedural style of Cucumber - congratulations!
Is there any solution?
I started experimenting with a new way of acceptance testing based on Capybara and an object-oriented approach. It spread very quickly among the Wrocław Rails developers so it seems to be an idea worth investigating. I will present how it works in my next blog post. Stay tuned.
If you read this far you should
and subscribe to my RSS
