Events in acceptance tests

I've been recently experimenting with the idea of events in the acceptance tests.

With the current approach based on Cucumber-like tools I see the following problems:

• Short scenarios share the same state but they prepare the state every time.
• Long scenarios are hard to follow
• Fixtures/Factories are in use, which makes the tests dependent on the implementation
• Slow builds

The idea is to solve some of the problems by using events.

As "events" I mean the following concept:

• Split tests into base scenarios and side scenarios
• Base scenarios can trigger certain events
• Side scenarios can declare which events they're interested in
• When the base scenario triggers an event all the side scenarios are executed

In practice it means that some tests depend on other tests. However scary it sounds I believe the reward is worth the risk.

This is going to require more discipline than "normal" tests, you need to find out what are the best base scenarios and which are fine as side scenarios. In my opinion it comes down to better understanding the business domain - which is always critical to the success of the project.

It's important to note that the side scenarios have to leave the application state unchanged, they may create some new content, but it needs to be destroyed at the end of the tests or it must be a content that doesn't influence other tests.

Let's look at an example. In an e-commerce app I think this would be a base scenario.

admin.create_new_product("a book")
user. visit_main_page
user. add_to_cart("a book")
user. confirm_order
admin.should_see_new_order("a book")
admin.confirm_the_last_order

This is the core scenario without looking at any side effects/side scenarios. The shop exists so that admin can sell products.
What are the possible side scenarios here? Several of them come to my mind:

• user can find the product using the search box
• google bot can find the product page
• admin can edit the product data
• user can recommend the product to his friend
• the product is in the "recently added" box

As you see, all of the above require that the product is already created.

Here are some tests that require an order to be created by the user:

• admins receive an email
• the user data exist in the admin panel
• the user receives an email

One way to look at it is that the requirements are implemented as events, so that:

admin.create_new_product("a book")
trigger("product:created", "a book")

user. visit_main_page
user. add_to_cart("a book")
user. confirm_order
trigger("order:created")

admin.should_see_new_order("a book")
admin.confirm_the_last_order

How can we implement an example side scenario?

With my current implementation it looks like this.

class SearchProductScenario

  def initialize(test_framework)
    test_framework.subscribe("product:created", self)
  end

  def execute(options)
    product_name = options[:product_name]
    user         = options[:user]

    user.visit_main_page
    user.fills_in "query", :with => product_name
    user.clicks("Search")
    user.should_see(product_name)
  end
end

I've got a simple TestFramework class that is responsible for instantiating all of the scenarios. It's not fully automated yet. It works with RSpec now, but it's framework independent.

We use this approach in one of our projects. It's too early to say if it works well in development. In my opinion it will help with most of the problems stated at the beginning of this post. Scenarios should be shorter and focused on the core of the feature in test. Tests should run quicker as they can share the state with their base scenarios. Fixtures or factories are no longer needed. It's all at the cost of being disciplined in writing scenarios in a certain way.

What do you think about this approach?

Some good news for people interested in this concept! We've been discussing the ideas with other DRUG (Wroclaw Ruby User Group) people and we're going to release a gem which will include this idea. The gem will contain the best things from Cucumber and Steak, so the name is natural: bbq :)

Thanks for reading!

If you read this far you should and subscribe to my RSS

Object oriented acceptance testing

In my last blog post I highlighted the two problems that I have with Cucumber for acceptance testing:

1. It's difficult to get the customer to write/read Cucumber scenarios
2. Cucumber scenarios most of the time are procedural programming.

The story

Being aware of the problems mentioned above I started experimenting with my side project to find a better way. I prefer trying out new ways on my side project instead of on the client work. The project is a social network app for board game geeks. As with every social app the interaction between users is quite high. People can vote for good games, comment on other people votes, write game reviews etc. I started writing tests in the following format:

user = TestUser.new
user.visit_main_page
user.click "Login"
user.fills_in :login, :with ="andrzej"

As you see it's a fairly normal Ruby code with API similar to what Capybara/Webrat offers out of the box. In fact, I use Steak with Capybara.

Ruby code

When you compare it to the Cucumber style, the main difference is that it's Ruby code. If it's so hard to convince the customers to write scenarios then I see no point in inventing and using a new language. We all love Ruby.

Programmer-friendly


If it's mostly programmers who read the scenarios, then Ruby is the best choice. It's programmer-friendly and I think many programmers would parse the Ruby code above more easily than the Cucumber way:

Given a new user
When he visits the main page
And he follows "Login"
And he fills in "Login" with "Andrzej"
..

User object

In most apps the concept of User is very important. Why not making it an object? It simplifies a lot and you can hide some implementation details in the (Test)User class. You can create methods and reuse the implementation in an easier (IMO) way than reusing step definitions in Cucumber.

Everything is an object

In fact when you start working this way, you quickly realize that there are more objects appearing in your test world. You will probably have a Website object which can keep some implementation details, like the path to admin panel or methods responsible for setting up the application state.

You can also have objects encapsulating certain pages, if it's useful, or objects for some of your website widgets - sidebar, search_box etc.

Object oriented programming


As you see it's all about the good old OOP that we use in our production code. This solves my problem with Cucumber being too procedural. I can write code like:

user.should_see("Popular games", :inside => sidebar)

Whatever OOP style you like, you can use in your tests. Sounds good to have all pages being subclasses of Page? Write it so.

Human readable

I complain about the fact that it's hard to get customers read/write the scenarios. Cucumber is human readable, no doubts about it. However, if you look at the OOP code - is it not human readable? It takes discipline to have all your methods well named, but it's so worth it. 

I claim that this kind of code is still readable to our non-programmers stakeholders.

Documentation

If you agree that it's readable, then you can actually think of your scenarios as the documentation. Nothing different here, as compared to Cucumber. If you organize your test objects nicely then you not only have a scenario view but also you can see what are the widgets responsible for or what is possible for the Guest user and what is possible for the logged-in User.

Interaction

I found it hard to implement interaction between users in a Cucumber style of testing. I mean things like this:

andrzej = TestUser("andrzej", "password")
john = TestUser("john", "password")
andrzej.logs_in
andrzej.writes_a_review("Agricola", "content")
andrzej.logs_out
john.logs_in
john.visits_reviews_page
john.should_see "content"
john.comments("nice review!")
john.logs_out
andrzej.should_receive_email("nice review!",  "john")

I think it's readable and if you hide the details in methods then the scenario reads very well. You can easily introduce new users into the interaction.

Personas

Personas is a concept in which you find the different types of users who interact with your app. You can give them names, describe what they do, what matters to them etc. This concept can help you think about new features in terms of which persona would use the feature in what way.

In your scenarios you can then use the persona to highlight which aspect of the feature you test. If you worry about security of the new feature then have a user in mind who can hack into your website and write scenarios to ensure it's not possible.

little_bobby_tables.visits_main_page
little_bobby_tables.fills_in :login, :with => "Robert'); DROP TABLE Users;--")
db.should_have_table("users")

Common.rb

When I started with this kind of acceptance testing I didn't think it was so useful at first. When I showed it to my colleagues they quickly started using it as well. Special thanks to Paweł Pacana, who used it in our Ruby User Group website and published it to Github. Since then we started using this approach in our client projects and it works really well. I think it brings back the joy of writing acceptance tests.

I call this approach "object oriented acceptance testing" but my friends started calling it "common.rb" as the main test file that I used. Here is an excerpt from the first version of my common.rb file:

class TestUser
  attr_accessor :steak
  attr_accessor :email
  attr_accessor :password
  attr_accessor :login
  

  def initialize(steak)
    @steak = steak
    yield if block_given?
    self
  end


  def click(*args);     steak.click_link_or_button(*args) end
  def choose(*args);    steak.choose(*args) end
  def select(*args);    steak.select(*args) end
  def fill_in(*args);   steak.fill_in(*args) end
  def visit(path);      steak.visit(path) end
  def attach_file(*args);      steak.attach_file(*args) end
  def should_see(text); steak.page.body.include?(text).should == true end
  def should_not_see(text); steak.page.should_not(steak.have_content(text)) end
  def should_receive_email(text) 
    steak.find_email(@email, :with_text => text).should_not == nil
  end

If you want to see more, look here:

https://github.com/drugpl/drug-site/blob/master/spec/acceptance/support/common.rb

All of the scenarios are kept in spec/acceptance, so look around how it works.

Not a gem, an approach


I was asked to release it as a gem. After some thinking I decided not to do that, as this would make it limiting. I prefer to think about it as an approach, not a specific technology. It all comes down to: Use object in your acceptance tests

The technology I choose is Capybara + Steak. Steak is a very simple DSL around RSpec. Since I started Capybara already provided a replacement for Steak, so it's no longer needed.

This approach is not web specific. In fact, we have a project which consists of a server written in Ruby with EventMachine which has no views but just exposes an API. We write test scenarios using objects that simulate clients connecting to the API. It works very well with typical acceptance tests but also with load testing.

Summary


As you see the approach is not revolutionary and I know that I'm not the first developer who uses it. There are not so many articles about it and I hope it can become more popular. Obviously, it doesn't solve all of the problems that exist in acceptance testing.

We still need to think how to make acceptance tests run quickly or solve the problem of preparing the application state for testing. I think we can do better than using Factories or Fixtures in acceptance tests and I have some ideas for simplifying it. I will present how it works in my next blog posts. Stay tuned. Thanks for reading.

If you read this far you should and subscribe to my RSS

2 problems with Cucumber

The most popular technology for acceptance testing in Rails applications is Cucumber.

I used to be a big fan of Cucumber. We've been using it in every project for our customers.

From my point of view Cucumber delivers 2 important features:

• A special language readable to non-programmers (Gherkin)
• Acceptance tests - embedded browser driver - Webrat/Capybara etc

When I started using Cucumber (and prior to that I used Rspec Stories) it was mostly because of the acceptance tests part. Cucumber offered the whole philosophy in the framework, a philosophy based on the importance of communication with customers through the scenarios. The communication was based on a language called Gherkin.

Cucumber became a whole framework for managing requirements. The implementation is based on step definitions which we could implement in Ruby. Cucumber also gives us tools for running the acceptance tests, tools like tagging, features like documentation etc.

Thanks to tags we could run only the tests that are responsible for a specific part of the system. Another use case is to tag new scenarios with @wip and those are not run.

The biggest Cucumber project we have at the moment contains almost 600 scenarios. The biggest project I'm aware of has 2000 scenarios. It's a lot of "code".

With every new project I hoped that Cucumber can become one of the communication tools between our team and the customer. Only once I managed to get the customer write the scenarios, however the quality wasn't good and they had to be rewritten by a developer.

Every now and then we could send the scenario to the customer to get their approval that they are correct. In this cases the customer was able to read them easily. However, it was difficult to convince the customer to review the scenarios on a daily basis. They preferred writing a document or using a ticketing system like Redmine. The result is that we have requirements in 2 places - tickets and scenarios. It's not perfect, but it works well enough.

My conclusion for now is:

1. It's difficult to get the customer to write/read Cucumber scenarios

We gain a lot from the acceptance tests feature of Cucumber. Every functionality is covered by scenarios and it creates a nice regression weapon in cases where we need to change something in the application. Upgrading from Rails 2 to Rails 3 was one of such occasions. Unit tests would not be enough.

Recently I noticed that it's getting harder to manage the scenarios written the Cucumber way. There are many patterns how to keep the steps definitions, how to split the scenario etc.

I realized that it's the same kind of problem when you write a procedural code with no objects.

2. Cucumber scenarios most of the time are procedural programming. 

With procedural programming it's hard to keep data and behaviour together. For simpler apps procedural programming is fine but for a bigger codebase I personally prefer object-oriented programing. I suppose it would be possible to include some object-orientation into Cucumber but I'm not sure if it would feel right and if it's worth the effort.

Summary

I see two problems with Cucumber and its usage in our projects:
It's hard to get customer working with scenarios and it's hard for developers to work with a bigger Cucumber codebase.

If you have customers working with Cucumber - congratulations! If you manage to deal with the procedural style of Cucumber - congratulations!

Is there any solution? 


I started experimenting with a new way of acceptance testing based on Capybara and an object-oriented approach. It spread very quickly among the Wrocław Rails developers so it seems to be an idea worth investigating. I will present how it works in my next blog post. Stay tuned.

If you read this far you should and subscribe to my RSS

MVC and OOP, are we doing our best? - talk proposal

Here's my Euruko 2011 talk proposal which was rejected by the organizers. Let me know if the topic sounds interesting to you, I'd be happy to present it at some other conference.

Euruko 2011 talk proposal

It's a result of many years of my research/practice in OOP and MVC. The recent discovery of DCI can change a lot about the way we create MVC applications and Ruby is the best fit for DCI, in my opinion.


"MVC and OOP, are we doing our best?"

Ruby is one of the best OOP languages - are we using it the best way in our MVC applications? My observations show that there are common places where Ruby developers break the traditional OOP rules like the Law of Demeter or SOLID. It results in code of worse quality than it could be.

Early on, we agreed that we should move the logic from controllers into models. It's all fine, but most often it results in fat model classes. How to deal with the complexity?

We put everything into classes. DCI claims it's not OOP, it's class-oriented programming.

DCI (Data Context Interactions), a new architecture paradigm focuses more on object behavior and fits very well with Ruby MVC frameworks like Rails or Sinatra. I will talk about DCI and present some examples. DCI was invented by Trygve Reenskaug, the same scientists who formulated MVC. One of its main ideas is to inject roles to objects runtime (using mixins), thus making the base classes much thiner.

DCI brings some fresh air into the MVC architecture and solves many of the existing problems, like:
- fat model classes
- difficult reusability
- testing problems
- instead of one big object-oriented design we can deal with many small ones

This talk is based on several of my blog posts. Some of the ideas were already presented at SRUG and DRUG (Silesia and Lower Silesia Ruby User Groups) meetups. The blog posts and the talks triggered many interesting discussions.

http://andrzejonsoftware.blogspot.com/2008/07/mvc-how-to-write-controllers.html
http://andrzejonsoftware.blogspot.com/2010/05/rails-mvc-how-to-deal-with-fat-models.html
http://andrzejonsoftware.blogspot.com/2011/01/code-reuse-in-ruby-why-composition-is.html
http://andrzejonsoftware.blogspot.com/2011/02/dci-and-rails.html
http://andrzejonsoftware.blogspot.com/2011/02/application-or-set-of-resources.html

http://srug.pl/assets/andrzej-krzywda-rails-and-dci.txt



BIO

Andrzej Krzywda is the founder of Arkency, a Ruby on Rails shop (6 developers) based in Wroclaw, Poland. He also teaches Ruby on Rails at the University of Wroclaw. Andrzej has 11 years of IT experience. He has been involved in many different projects, including big telco, small startups, consulting companies etc. He worked with Java, .Net, Python, PHP but since 2004 he feels most comfortable with Ruby. Andrzej has presented at conferences like RuPy 2007, RuPy 2008, PyCon 2008, SFI 2006, SFI 2008, KKIO 2004 and lots of smaller meetups like DRUG or SRUG.
andrzejkrzywda@gmail.com

Rails: Many classes in one file

Some time ago I asked if there is any way of having many classes in one file and being able to work normally in the development environment (reloading classes between requests).

Luckily, Jan Dudek (thanks!) came up with some nice solutions. The best of them uses ActionDispatch::Callbacks in combination with require_dependency. Here is an example:

config/initializers/load_classes.rb

ActionDispatch::Callbacks.before do
  require_dependency "app/models/events.rb"
end

I have tried it in one project and it works pretty well in all environments.

Why would I want to keep many classes in one file?


I agree that this technique breaks the Rails convention of keeping one class per file. Possibly, it can confuse some developers on where to find classes definitions. In this case it's important to communicate with the team whether it's worth trying and explaining how it works before using it.

For me this technique is useful when I have many Rails models, all very short, related to the same feature. The example above (app/models/events.rb) keeps definitions of Event, Attendee, Attendance, AttendanceType, EventBoard. All of them are short and instead of jumping between many files I've got one file which keeps it all in one place. It's hard to measure whether it's good or bad, works well for me so far.

An application or a set of resources?

My last post about DCI and Rails triggered many interesting discussions. One of the topics that came up very often can be summarized to this question:

Do you think of your web application as an application or as a set of resources?


My answer is clear: it's an application.

I noticed that not everyone agrees with it.

In the Rails world a resource is a very popular word. Mostly because of the REST architecture that we follow in our apps. Whenever I can I use the REST approach, but I think of it as a way of using URL's and accessing the application. I find it a nice way of structuring my controllers. REST is just an interface. From outside and for my users it's an application, not a set of resources. They use resources but they access them through an application which to me is an important distinction.

This topic caused that I decided to publish the assumptions that are needed for a DCI architecture, at least how I understand it:

1. As developers we model the real/business world.
2. If a concept appears in the model we reflect it in the code.
3. Most of the web apps scenarios has the pattern of "a user does something with the app"
4. If user and app are so important in the domain then we make them classes/objects.
5. It results in big user/app classes.
6. The problem of big classes can be solved by using the idea of "injecting roles" runtime.

Usually point 5. discouraged people from reflecting the user/app importance in the code. The alternative is chosen which is things like this in the controller:

@discounted_products = Product.discounted

whereas I prefer this:

@discounted_products = shop.discounted_products 
# shop is like the application object here

Anyway, point 5 can be now solved by the idea of injecting roles and DCI as described in my last blog post and in my opinion we can start using the user and application objects more often. Once you start using them, many of the OOP decisions you need to make become easier, because you just reflect the domain in the code.

DCI and Rails

I got really excited about DCI recently.  I'm still not sure if I get it right, but let me try to describe the way it works and how it can improve our Rails applications.

Here is the short version of this blog post:

#controller action 


current_user.extend Buyer 
current_user.add_to_cart(product_id, quantity)


DCI stands for Data Context Interaction. It was invented by Trygve Reenskaug, the same scientists who formulated MVC. It's a new paradigm, but it's very similar to OOP and fits especially well with the MVC architecture that we use in Rails applications.


Let me start with the evolution of Rails applications, so that I can explain current problems with the Rails model layer (in my opinion).

Very early we (the Rails community) agreed that it's better to move the logic down to models and keep controller very thin. I also wrote a blog post on how to write Rails controllers which is still my opinion on this topic.

As a result we had to cope with so called "fat models". This topic was an area of my research for years. It resulted in some techniques that I described in the "How to deal with fat models" blog post.

One of the things that I personally dislike is the overuse of Global State and Class Methods in Rails controllers. I have seen it everywhere and it my opinion in bigger apps it can result in maintenance problems. You can read why it's bad in this article by Hevery Misko. I must admit that with ActiveRecord it's very hard not to use Class Methods, but at least we can minimize it.

I was researching more OOP techniques that can be used to keep a nice OOP design in Rails model classes. I found the composite pattern (resulting in delegation) to be the closest to what I want to have. I asked the Ruby community why we don't use composition so often. The answers confirmed my observations that Ruby mixins (modules) are much more popular.

Anyway, it doesn't really matter - the result of "traditional usage" of mixins and composition in Ruby is very similar - you end up with a class which has lots of responsibilities. It doesn't matter that most of the responsibilities are delegated to other objects, actually it leads to a problem called self schizofrenia. This is especially visible in classes like User (in a social network app) or Article (in portal apps), which are very often a central place of application logic.

There are two techniques that I found useful during my research.

1. Current user pattern

This pattern was just reflecting the fact from a real life - a user interacts with our website so whenever a user does something there is a method in the User class responsible for that. Some methods can be grouped into "roles" and extracted to another class or a module, but they're declared in the User class.

2. Website pattern

This is the other side of the current user pattern. The user is interacting with a website, so it sounds like a good idea to have some kind of Website (or Application) class. This class encapsulates all the data and behaviour that is accessible from the website.


As you are probably seeing it now, it results in huge User and Website classes.

Just to be clear - both the current_user (in the Devise/Authlogic meaning) object and the website object are accessible from ApplicationController, so any controller could have access to them. The advantage here is that you don't need to introduce Global State and with the website pattern you can eliminate calls to class methods.

It was always kind of a smell to me that any controller takes the whole user object just to interact with a very small subset of the user interface. Let's say that you have a PostsController which needs to display all the posts. It takes the website object just to call website.posts, but theoretically it can call any other unrelated method. Static languages deal with it by communication using interfaces. It is a solution but it's still not perfect.

Here is where DCI comes in.

One of the things DCI claims is that most of the time we don't do object oriented programming but we do class oriented programming. It's more visible in the Java/.Net world but I think the same applies to most of us - Rails programmers. We put our declarations in classes and then instantiate objects with lots of unnecessary stuff.

DCI suggests that we keep our core model classes very thin. Zero logic, only data, if anything.


The logic/behaviour should be kept in roles. In Ruby we can nicely use mixins for that.


In an e-commerce app you would have User class but the buying/cart logic would be in the Buyer module. The User class knows nothing about cart, buying.

In a social network app you can create a GroupUser role and keep all the group related logic inside.

You can also extract some roles from the Website class. Some candidates in an e-commerce could be ProductRepository, OrderDepartment, NewsletterManager, Blog etc.

This logic is then injected runtime, at the object level, in the context of a specific use case.  In my opinion contexts or use case fits very well with the rails thin controllers rule.

Usually, one controller action is a specific use case. One example may be adding some product to a cart. The action could look like this:

current_user.extend Buyer
current_user.add_to_cart(product_id, quantity)

Another example, with a website:

class EventsController
  def index
    website.extend EventsBoard
    @events = website.recent_events
  end
end

As you see we use .extend method here which allows extending an object with a module. It all happens at the object level.

To me, this technique is most crucial for DCI (from a Rails perspective). Extending the object with a role at the controller level simply solves all of the OOP problems I had to deal with. The class doesn't know about all the roles, there's no problem with huge classes or object schizofrenia. You just create roles which are usually small modules and that's it.

DCI suggests that we keep use cases as classes. If my understanding is correct the place for creating use cases would be at the controller level. Notice that usually a single request is some kind of a use case. However, usually the code is so small in this area that I'm not sure it deserves a class. I can see the advantage of seeing all the interaction in one place so I just need to get used to this kind of thinking.

Terminology

In order to fit DCI into the Rails terminology I assume that we can call our thin model classes the Data layer. Our controllers actions can be wrapped into a class and called Contexts. Those actions take the Data, inject the roles (as Ruby modules) and run some Interactions.

Problems


I have already tried this approach in one small application. It worked pretty well. One of the problems that appeared was a situation when I actually want the object playing almost all of the available roles. This kind of situation often takes place at main pages or in some kind of dashboards.
I solved it by using the Cells framework. In short, Cells let you create mini-controllers (cells) that can be responsible for one part of the view. It can access the controller and execute a "mini-action". In my case I used in a way that every cell extends the website with the role that is associated with this view. It worked pretty well and I'm quite happy with the result.

Summary


In my opinion DCI fits very well with the way we create Rails apps. It simplifies the Model layer a lot. If you're happy with your current OOP design then you will probably not gain too much from DCI. If not, wikipedia entry is a good start for getting familiar with this concept at a more theoretical level. The whole concept is actually very simple and intuitive. It brings the user perspective to your code, which in my opinion is a good thing. We're here to model the real world/business etc.

Still looking for perfect Rails app architecture? Let us guide you through options - we've got dozens Rails projects finished and want to share the knowledge with you.

Sign up to be notified when our "Rails apps architectures" report is finished.

 If you read this far you should and subscribe to my RSS.