The Rails-way and bigger projects

The whole idea of having a data-structure that you retrieve from the database and carry through all the layers to the UI makes the development very quick at the beginning.

The coupling it provides, at the later stages of development makes it very difficult to split the app into any kind of modules. It's also difficult to test anything in isolation. That's why Cucumber, Capybara, Selenium are so popular, as they help you in writing integrated tests.

Integrated tests have their place, but it's impossible to test everything through all the layers. This results in slow builds - a typical syndrome in Rails projects.

The Rails way makes sense for smaller projects. What I'm disagreeing with is using "The Rails way" in bigger, serious projects.

The Rails itself has nothing inherently wrong. I'm not attacking Rails, I'm only attacking 'the Rails way'. You can use Rails for bigger projects, if you know what you're doing.

What I'm teaching at my Rails class

It's now the 4th time, I'm teaching the Rails class at the University of Wroclaw. The main goal of this class to prepare people to work in a Rails-based company. The students are the best of the best, so the goals can be high.

Every year, I introduce some changes to the program of teaching to reflect better what is happening in the Rails community.

This year I'm going to focus more than before on the integration with mobile, native apps and on the art of dealing with legacy projects.

We start with extremely quick Ruby lessons, based on the Ruby koans exercises (students do it on their own).

The first 3 lessons are about "the Rails way", which I make it clear - is good only for toy projects. This is where I teach about ActiveRecord-oriented programming, coupling everything with everything, making things impossible to test - you know, the usual Rails stuff.

Once we know the enemy and know the pains it provides, we learn how to avoid that.

This is where good, solid OOP lessons come in, service objects, repositories, domain objects, etc.

After that, we enter the "frontend" module. We learn how to create Single Page Apps using the dominant JS frameworks (Angular) and some no-framework techniques (HexagonalJS. This also includes techniques like Pusher to have server->client communication possible.

We also talk a lot about the way, a Rails backend can expose data through the API to the JS or mobile app clients. Mobile apps will take more time this way, as I want to show how to best cooperate with mobile developers.

There's a lesson about proper Rails deployment, things like Chef, Jenkins, Continuous Deployment etc.

Throughout the class, we'll not only create new Rails apps (lots of them), but I also want to teach how to find yourself in a legacy codebases, like Redmine or Discourse, how to refactor out from the mess, fix bugs, add new features - that's probably the hardest part of the class.

Testing is an ongoing topic in the class, as well. We'll learn about unit testing of different layers, integrated tests and their pains.

I hope it will be fun, as always :)

Implicit Rails features

This post is part of a series of blog posts. So far, there were 2 blog posts, which aim at helping you improve the modularity your Rails applications:

1. Typical Rails bugs

2. ActiveRecord overdose

3. Implicit Rails features

The speed with which you start every new Rails application is amazing. 

Within a few days you can have a fully working prototype of almost every application. Obviously, the prototype, by definition, may not be fully ready to production use.

With a really small codebase, the prototype does a lot of things. It's all thanks to many built-in Rails features. 

I call them implicit features. They come almost for free. 

• CRUD
• validations
• maintaining created_at and updated_at
• displaying errors
• displaying flash messages
• redirects between pages
• pagination
• uploads
• sending emails
• attr_accessible/attr_protected (security)
• CSRF - security

There's one downside to the implicit features. 

They're rarely documented and covered with tests.

As with every app, there comes a moment, when you need to start providing changes. Sometimes, you may want to rewrite some layers of the application. There's a trend recently that we try to find new ways of working with Rails apps, that is less dependent on ActiveRecord. Some developers rewrite their apps to Sinatra or Padrino. 

It sounds great in theory, but it's harder in practice.

It's good to be aware of such hidden functionalities. At some point, they can become explicit features, with their own tests, so that it's never broken.

If you know more such features, share them in the comments.

ActiveRecord overdose

In the previous post, I highlighted the typical Rails bugs, that I encountered during my research on better ways of working on Rails apps. The bugs were often security-related:

The main common point of the bugs is the ActiveRecord overdose.

ActiveRecord overdose is a surprisingly popular pattern that includes:

• putting everything into the model class
• fear of adding new non-AR classes
• over-relying on convention over configuration

When you look at a typical Rails model, it's a mix of the following responsibilities:

• persistence 
• business logic 
• adapters 
• view object 
• form object 
• others 

The Rails-way ActiveRecord is an enhanced version of the original Active Record pattern. The original pattern assumed only persistence and some simple business logic. The Rails version extended it with the view/form responsibilities. In many cases it's also extended with the adapters.

• persistence 
• associations 
• queries - scopes 
• saves 
• simple validations 
• business logic 
• state-machine 
• calculations 
• tracking changes (touch) 
• complex validations 
• some callbacks 
• feed/timeline/activity 
• factories methods (class methods) 
• adapters 
• service (coordinating multiple adapters) 
• search 
• external api 
• facebook 
• twitter 
• exception tracker 
• metrics tracker 
• image storage 
• pdf 
• mailer 
• notification 
• view object 
• first_name + last_name 
• foo.to_s 
• image_path 
• http boundary details (urls) 
• i18n 
• form object 
• accepts_nested_attributes_for 
• sanitizing 
• attr_accessor 
• others 
• acts_as_foo (persistence, logic, validations) 
• concerns 
• includes 

The core responsibility of an ActiveRecord model is being a representation of a db record. That means handling the persistence and having the basic logic. This usually works great in typical CRUD apps. Once things get more complex, it's time to reconsider the active record pattern. 

The best way of escaping from the ActiveRecord overdose is to do it step by step. I'll focus on that in my later posts.

Typical Rails bugs

As part of my research on improving Rails application, I noticed a pattern in the bugs that are quite characteristic to Rails in several applications.

They have certain 'visible' things in common:

• often security related, like leaking some information to unauthorized users
• they live somewhere in the area of business logic/persistence
• they are not so easy to fix in the existing codebase
• they tend to exist in groups, similar bugs in different areas
• they are easy to miss
• often they appear during the requirements changes

Overdose of ActiveRecord

The reasons they exist is a combination of different things. The main common point of the bugs is the over-usage of ActiveRecord. It's easy to fall into this problem, I did it many times, as well.

You start with a new Rails app, adding new features very quickly. New requirements pop up. You use the popular Rails techniques to deal with them:

• validations, which often turn into conditional validations
• Single Table Inheritance
• state-machine
• accept_nested_attributes
• models callbacks
• virtual attributes
• external gems that provide value magically, by using ActiveRecord (implicit dependencies)

Every technique is fine on its own. When you start using all of them in the same area, problems appear. 

Examples

I started noticing this trend, while reviewing the code of our potential Arkency customers. We sometimes do the "rescue missions" and help with legacy code. I can't use those examples here, so I started reviewing some of the popular open-source Rails applications: Discourse, Spree, Redmine, Gitlab. It didn't take me much time to discover the same bugs.

Just to be clear, all of the projects are awesome and I'm grateful to the people behind them. Most of the times, they deal with this kind of problems very quickly. I don't want to blame them for anything - they just serve here as examples.

Discourse

Let's start with Discourse:

1. Digest mail ignores secure groups

http://meta.discourse.org/t/digest-mail-ignores-secure-groups/7288

People receiving the digest mail can easily read posts not meant for them. That's because the digest mail ignores the secure groups a member has access to or not.
Quite a problem as I unfortunately found out.

2. Non-authenticated users see private topics in 404 page

http://meta.discourse.org/t/non-authenticated-users-see-private-topics-in-404-page-was-mobile-view/7419

Discourse correctly prevented me from seeing the topic and instead showed a 404 page. On that 404 page, the lists of "Latest Topics" and "Recent Topics" showed private topics. I could click those links, which resulted in seeing the same 404 page again.
The 404 page should not show private topics.

3. Auto-suggest topics shows private topics

http://meta.discourse.org/t/auto-suggest-topics-shows-private-topics/7418/3

We've got Discourse running with private categories. When a user without access to the private categories type a new topic, they are presented with topics in categories to which they don't have access.

Comment: Did you notice the pattern here?

Accidentally, I've had a small conversation at HN with one of the Discourse founders. He said:


Those private topic bugs are not the result of ActiveRecord. We added a group layer on top of existing code and missed some places where queries did not respect it.

Had we used raw SQL instead of an ORM we would have had the same issues. All projects are open to this style of bug. The correct thing to do is report, close them quickly and add tests to prevent them from happening again (which we do.)

I have completely no 'science' proof here, it's just based on my experience with hundreds of Rails projects. However, I disagree that all projects are open to this style of bug. 

It's very typical to Rails projects. 

The temptation of globally accessing every model/record from every place in the project makes it very easy to introduce such bugs. When you add new requirements (as they did) it's practically impossible to find all the places which should be changed.

Do I blame ActiveRecord here? No, ActiveRecord is a good library for persistence. It has its issues, it's huge, even the maintainers consider it a legacy code. Apart from some edge cases, it works more or less ok. I think the problem is relying on ActiveRecord for basically everything.

Spree

Let's see some other bugs, this time in Spree:

1. Fix wrong discount calculation with flat percent promotions

https://github.com/spree/spree/pull/3054

Fix wrong discount calculation with flat percent promotions when there are more than one line item in the order.

This error happened because the order instance here:
https://github.com/spree/spree/blob/master/core/app/models/spree/order_updater.rb#L24

is not always the same instance in memory here:
https://github.com/spree/spree/blob/master/core/app/models/spree/calculator/flat_percent_item_total.rb#L15

Adding the inverse option to the relationship makes sure you have the same object instance in both places.

Comment: Here we have another problem related to ActiveRecord. In more complex situations (cyclic associations), you can have the same 'record' in memory as two different instances, leading to errors.

2. Admin products loads entire dataset to determine the total count when paginating

https://github.com/spree/spree/issues/2679

When kaminary determines the total count of records it runs the following code
@collection.except(:offset, :limit, :order).count

The issue is that this sequence loads entire dataset to determine the count. This makes heavy load when products have large number of items.

The reason of this behaviour is group_by_products_id here https://github.com/spree/spree/blob/1-2-stable/core/app/controllers/spree/admin/products_controller.rb#L94

Comment: This time we have a combination of ActiveRecord and an external gem - kaminari (paginator). The end result is slowness of admin panels with thousands of products - they all will be loaded to memory. It's not the worst bug I've seen, but it's typical.

3. Default Tax Does Not Calculate Taxes Correctly if there is a Promotion

https://github.com/spree/spree/issues/2493

4. Taxes should be re-calculated after promotion adds an adjustment

https://github.com/spree/spree/issues/1735

Comment: the titles should be enough. This kind of bugs is usually related to the ActiveRecord callbacks usage. In complex scenarios it's not that easy to track all the places that need to be called after some changes.

Redmine

Let's now look at one Redmine bug:

1. Time entries of private issues are visible by users without permission to see them

http://www.redmine.org/issues/12287

Comment: Does it ring a bell? Again, a leak of private information.

Summary

We have seen some of the bugs, that I called typical Rails bugs. They seem to have certain things in common. I've seen them so many times, that they are the first things I look for, when I get access to a new project ticketing system. 

Is there any way of defending against them? I think so. The best step to start with is to be aware of such problems. Try to spot them in your projects, see what history is behind the bug, spread the knowledge in your team.

In my next blog posts, I'll focus on techniques that should help decrease the number of such bugs. Stay tuned. In the meantime, you may want to follow me on Twitter and subscribe to the Rails Architectures Guide, that I'm working on.

Single Page Applications fight - video

During the last wroc_love.rb conference we've organised a "fight" between 4 different approaches to Single Page Applications. Below is the video (0.5h) and a very short and simplified transcript from this event.

The participants, from the left-side:

Patrick Mulder - Backbone
Piotr Sarnacki  - Ember
Andrzej Krzywda (me) - Hexagonal
Adam Pohorecki - Angular

moderator - Jan Filipowski

1m39s

Can anything be a Single Page App?

(and discussion about SEO)

Patrick: only for some apps

Drogus: for interactive apps, not for static apps that need to be indexed by Google

Andrzej: for all the apps

Adam: we need better tools on the backend in order to have a better developer experience

Piotr: Andrzej, would you write wikipedia as a SPA?

Andrzej: Yes.

Adam: What about SEO?

Andrzej: It's a myth that SPAs are not indexable. Google has an official document for that. It is additional work.

Patrick: You can use PhantomJS for SEO.

Adam: We've used PhantomJS for that, took a couple of hours.

8m28s

Can you introduce the framework you're representing here?

Andrzej: HexagonalJS is not a framework, it's an architecture guide. Uncle Bob (OOP) + Raganwald (AOP) + Hexagonal Architecture. It's more like a set of guidelines

Piotr: EmberJS - it does as much as it can for managing your SPA. it's not minimalistic, for good reasons (you don't have to write much code). 

Patrick: Backbone, very small, minimal. you don't bring too much dependencies. you can choose what you want. 

Adam: Angular, Two-way data binding is the key feature. Synchronization of HTML and your JS. Choose framework with data-binding. Ember is essentially a language, forces its own object model. Backbone as well. Hexagonal or Angular is pure JS.Your application can live outside of Angular.You could use Angular with a Hexagonal app. Backed by Google, more mature than Ember.

Andrzej: Data binding is not as important as you may think. Manual control is not that difficult. Hexagonal doesn't have data binding jilt-in. Apps are not about data changes, but about actions

Piotr: I agree that data binding is important. Both, Angular and Ember are good for big applications. Nested views, zombie views (backbone). Angular has its own compiler for DOM elements. You need to know how the compiler works. Angular uses dirty checking. Angular claims that your UI shouldn't have than 2000 elements. You're screwed if you have more.

Adam: We built a 100 rows, 500 columns table. We managed to do it with Angular. It's not that hard to do the UI part. Directives are like helpers for html. 

22m6s

Do you have to take the mindset of the creator?

Adam: Most of the things in Angular are optional.If you don't use the HTML part, you lose a lot.

Andrzej: It's a problem of all of us here, including Hexagonal. We use different terminology for different things. All the frameworks are far from being Rails, we're not mature enough. All the frameworks are in their infancy years.

Patrick: It's easy to work with Rails and Backbone

Piotr: Ember has its own class model. It's done mainly for making it easy to optimise. Ember is about Convention over Congifuration. Rails in JS. Ember is like Rails, Backbone is like Sinatra. At some point you have to use more plugins if you start with something minimal. A lot of pain if you want to write a big app with a small framework.

Andrzej: I disagree. I would use a framework for a prototype or for a small app. Once the app grows, it has its own world. I don't want the framework to go into my way. I want to have the control over the applications. I'd use Ember for smaller apps, not for bigger apps.

27m23s

How to be involved in the community, how to learn the framework?

Adam: Angular has really good docs. All development happens on github.

Andrzej: Hexagonal has just started. It got the name just recently. We've used it for over a year. We have a strong team behind it. Our goal is to show the guidelines.

Piotr: Ember is 1.0RC now. Docs are better now. IRC is very active. Google Group.

Patrick: JavaScript and Ruby devs can help.

END

If you're interested in Single Page Apps, you may want to follow me on Twitter.

Why do projects take so long?

Some projects take a long time to develop. In different contexts, the definition of "long" may vary. It was always interesting to me, what are the reasons for it. Recently, though, it became my obsession.

I've started collecting some thoughts on this topic.

Ask yourself and your teammates about the last project you've been working on. "Would we be able to implement the same result in 1 month? In 1 week?".

Surprisingly often, I'm answering to myself - yes, that would be possible, given some assumption. I've had similar answers from my colleagues.

Where does it come from?

Is it the known tendency to underestimating a problem by developers? I don't think so, as we're talking about a project, that has already happened. We know what is to be done. We are aware of edge cases.

"This is time it will be different, faster"

This is another popular trend among us, developers. Most of us are optimistic, even the experienced ones. It's not some kind of a student-syndrome (I'll do it tonight), but a well-thought statement. Is it really different and faster? Not always…

Where does this optimism come from? Good developers become better at solving technical problems. With each problem solved, we know that we solved the problem forever. We love solving problems, every day we're smarter by this attitude. We're progressing and the optimism may come from that.

It's almost impossible to get a second chance of working on the same project. What would that even mean from a business perspective? We rarely get that chance.

Is it the technology that let us think, that we could do better this time? That's a serious candidate. Every time, I switched to better technologies or to better architectures, I noticed the speed increase. I was more confident. I saw the results.

I remember a situation, back in 2004/2005. I was involved in preparing an estimate for a new project. The estimate was about 6 months for 4 developers (assuming ASP.NET). Very shortly after that I learnt Rails. I realised, that the project is perfect for Rails. It was a CRUD-based web app. I kept convincing the management to give it a try. No luck. Finally, I had a chance of coding live in front of some managers. I implemented the main path in about 1 hour (I was well prepared). That convinced them. We've finished this project in 1.5 month, with one junior programmer and me being mostly a PM and a Rails mentor.

What's the lesson? It's hard to tell. I'm not trying to say that only Rails was the reason. Motivation? The passion behind it? Trying to prove something? The focus? I think it was a combination of all of the above aspects.

Is it easy to repeat those aspects? Do they sound familiar?

Have you ever attended a hackathon?

There are many kinds of hackathons. What's most important in this context, is how much can be done in a very limited time. If you look at the results of Rails Rumble, some of the apps are quite big. Many features, everything is polished.

Hackathons are proves that we are able to develop apps quickly.

How can we apply hackathon lessons to our every-day projects? Is it possible?

What else can help us developing software faster?

I would love to know your opinions.