Tuesday, April 26, 2011

Events in acceptance tests

I've been recently experimenting with the idea of events in the acceptance tests.

With the current approach based on Cucumber-like tools I see the following problems:


  • Short scenarios share the same state but they prepare the state every time.
  • Long scenarios are hard to follow
  • Fixtures/Factories are in use, which makes the tests dependent on the implementation
  • Slow builds


The idea is to solve some of the problems by using events.

As "events" I mean the following concept:


  • Split tests into base scenarios and side scenarios
  • Base scenarios can trigger certain events
  • Side scenarios can declare which events they're interested in
  • When the base scenario triggers an event all the side scenarios are executed


In practice it means that some tests depend on other tests. However scary it sounds I believe the reward is worth the risk.

This is going to require more discipline than "normal" tests, you need to find out what are the best base scenarios and which are fine as side scenarios. In my opinion it comes down to better understanding the business domain - which is always critical to the success of the project.

It's important to note that the side scenarios have to leave the application state unchanged, they may create some new content, but it needs to be destroyed at the end of the tests or it must be a content that doesn't influence other tests.

Let's look at an example. In an e-commerce app I think this would be a base scenario.

admin.create_new_product("a book")
user. visit_main_page
user. add_to_cart("a book")
user. confirm_order
admin.should_see_new_order("a book")
admin.confirm_the_last_order
This is the core scenario without looking at any side effects/side scenarios. The shop exists so that admin can sell products.
What are the possible side scenarios here? Several of them come to my mind:


  • user can find the product using the search box
  • google bot can find the product page
  • admin can edit the product data
  • user can recommend the product to his friend
  • the product is in the "recently added" box


As you see, all of the above require that the product is already created.

Here are some tests that require an order to be created by the user:

  • admins receive an email
  • the user data exist in the admin panel
  • the user receives an email


One way to look at it is that the requirements are implemented as events, so that:

admin.create_new_product("a book")
trigger("product:created", "a book")

user. visit_main_page
user. add_to_cart("a book")
user. confirm_order
trigger("order:created")

admin.should_see_new_order("a book")
admin.confirm_the_last_order
How can we implement an example side scenario?

With my current implementation it looks like this.

class SearchProductScenario

  def initialize(test_framework)
    test_framework.subscribe("product:created", self)
  end

  def execute(options)
    product_name = options[:product_name]
    user         = options[:user]

    user.visit_main_page
    user.fills_in "query", :with => product_name
    user.clicks("Search")
    user.should_see(product_name)
  end
end
I've got a simple TestFramework class that is responsible for instantiating all of the scenarios. It's not fully automated yet. It works with RSpec now, but it's framework independent.

We use this approach in one of our projects. It's too early to say if it works well in development. In my opinion it will help with most of the problems stated at the beginning of this post. Scenarios should be shorter and focused on the core of the feature in test. Tests should run quicker as they can share the state with their base scenarios. Fixtures or factories are no longer needed. It's all at the cost of being disciplined in writing scenarios in a certain way.

What do you think about this approach?

Some good news for people interested in this concept! We've been discussing the ideas with other DRUG (Wroclaw Ruby User Group) people and we're going to release a gem which will include this idea. The gem will contain the best things from Cucumber and Steak, so the name is natural: bbq :)

Thanks for reading!

If you read this far you should Follow andrzejkrzywda on Twitter and subscribe to my RSS

5 comments:

Michał said...

It looks interesting. Especially getting rid of fixtures/factories without affecting the performance.

Although one point bothers me a bit:

the side scenarios have to leave the application state unchanged

So either you have to make side scenarios read-only or deal with deep-cloning and nested database transactions.
How do you deal with that problem?

Andrzej Krzywda said...

As I said, it's too early to find answers to all questions. The point that bothers you is the crucial one.

It's hard to discuss it without an example.

So far, where we found the need to change the state we created a new base scenario, but it made sense in our case.

I'm not going to try the deep-cloning/transactions solutions :) Sounds scary to me and I want to avoid it.

Editing a product description sounds like a good side scenario although it temporarily changes the state - at the end of the scenario we just edit it to the original version.

Paweł Pacana said...

Nice article. What I also like in this approach is the ability to control how many times side scenarios should be executed. We can execute them for every event occurence or just once.

I would probably try database savepoints for maintaining state in a single use-case (e.g. http://dev.mysql.com/doc/refman/5.0/en/savepoint.html)

Andrzej Krzywda said...

@Pawel

Other way would be to just trigger each event only once.

Looking forward to your experiment with database savepoints! It may help, but I'm still not sure if that's the right direction. I'd prefer to be database independent in my acceptance tests.

Paweł Pacana said...

@Andrzej You can have many side scenarios attached to one event type. In contrived example after "product:created" you may want to check if a user can recommend a product (but there is no need to check it for every "product:created"). On the other hand you may want to check if product has accurate discount (and you'll probably check this on every "product:created").

For me "practicality beats purity" - advantage of restoring application to previous state means less bugs connected to accidental or implicit state modification in side scenarios.

The workaround you proposed: moving state-changing side scenarios to base scenarios is just about restoring state (or reseting to initial to be precise whereas I propose travelling back to point in time).