mutant and minitest within a Rails application

2015 was the year of mutation testing for me. Luckily, thanks to the effort of Markus Schirp, the Ruby community has the mutant tool.

In short - mutant is totally "production"-ready and may help you code with more confidence, by testing your tests.

What mutant does is it parses your code, builds the Abstrax Syntax Tree and then it makes some mutation, like changing true to false. After that it runs the tests. If the tests still pass, then you have an alive mutant and it's bad.

When it comes to details, mutant supports rspec. Rspec is usually the testing framework of choice for many Ruby devs (or maybe Rails devs?). While I see the value of RSpec, I'm also sceptical about the additional layers of abstractions.

In short - I'm not a fan of rspec. However, I'm a fan of mutant and that was a more important thing to me than using minitest. At Arkency, we've even had situations that projects switched from minitest to rspec, "because mutant".

As for December 2015, there's a pull request which has the minitest integration. It's not released yet, though. So, if you want to use it, you need to use this particular branch.

https://github.com/mbj/mutant/pull/445

How do you make it work with a minitest-based Rails app?

Add this to the Gemfile:

gem 'mutant', git: "git://github.com/mbj/mutant.git", branch: "feature/minitest-integration"

Now if you try to run mutant with a controller test (as an example) by:

RAILS_ENV=test bundle exec mutant -r ./config/environment -I test --use minitest FuckupsController#index

, you will probably see this error:

undefined method `cover_expression' for FuckupsControllerTest:Class (NoMethodError)

(and yes, I do have a project with a controller called FuckupsController)

The thing with the minitest integration is that you need to be more explicit with what the test is actually testing. This helps mutant to understand what it needs to run.

I had to add this to the top of the FuckupsControllerTest:

class FuckupsControllerTest < ActionController::TestCase
  def self.cover(expression)
    @expression = expression  end
  def self.cover_expression    unless @expression      fail "Cover expression for #{self} is not specified"    end
    @expression  end
  cover 'FuckupsController'

In the later steps, I'll move it to some base class and make all the tests inherit from that. In that case, you just need to be explicit with the "cover 'Foo'" thing.

Now, when I run the same command:

RAILS_ENV=test bundle exec mutant -r ./config/environment -I test --use minitest FuckupsController#index

I get a nice report like:

Mutant configuration:

Matcher:         #

Integration:     Mutant::Integration::Minitest

Expect Coverage: 100.00%

Jobs:            4

Includes:        ["test"]

Requires:        ["./config/environment"]

Subjects:        1

Mutations:       60

Kills:           60

Alive:           0

Runtime:         11.03s

Killtime:        24.06s

Overhead:        -54.17%

Coverage:        100.00%

Expected:        100.00%

Active subjects: 0

Mutant configuration:

Matcher:         #

Integration:     Mutant::Integration::Minitest

Expect Coverage: 100.00%

Jobs:            4

Includes:        ["test"]

Requires:        ["./config/environment"]

Subjects:        1

Mutations:       60

Kills:           60

Alive:           0

Runtime:         11.03s

Killtime:        24.06s

Overhead:        -54.17%

Coverage:        100.00%

Expected:        100.00%

In short it says, that the FuckupsController#index action is 100% mutation-covered.

The minitest integration is not officially released for good reasons. Not many projects were tested with it. You can contribute by trying it on any of your minitest code and see if it works.
All feedback welcome at https://github.com/mbj/mutant/pull/445 ot feel free to ask/comment here and I'll pass it along to the mutant team.

----

PS. We now have sale on our Arkency books - http://blog.arkency.com/products/ - get a 30% discount with the HOLIDAYSALE2015 coupon. One of the books is about refactoring Rails controllers which is a perfect fit for the mutation testing idea.