Typical Rails bugs

As part of my research on improving Rails application, I noticed a pattern in the bugs that are quite characteristic to Rails in several applications.

They have certain 'visible' things in common:

• often security related, like leaking some information to unauthorized users
• they live somewhere in the area of business logic/persistence
• they are not so easy to fix in the existing codebase
• they tend to exist in groups, similar bugs in different areas
• they are easy to miss
• often they appear during the requirements changes

Overdose of ActiveRecord

The reasons they exist is a combination of different things. The main common point of the bugs is the over-usage of ActiveRecord. It's easy to fall into this problem, I did it many times, as well.

You start with a new Rails app, adding new features very quickly. New requirements pop up. You use the popular Rails techniques to deal with them:

• validations, which often turn into conditional validations
• Single Table Inheritance
• state-machine
• accept_nested_attributes
• models callbacks
• virtual attributes
• external gems that provide value magically, by using ActiveRecord (implicit dependencies)

Every technique is fine on its own. When you start using all of them in the same area, problems appear. 

Examples

I started noticing this trend, while reviewing the code of our potential Arkency customers. We sometimes do the "rescue missions" and help with legacy code. I can't use those examples here, so I started reviewing some of the popular open-source Rails applications: Discourse, Spree, Redmine, Gitlab. It didn't take me much time to discover the same bugs.

Just to be clear, all of the projects are awesome and I'm grateful to the people behind them. Most of the times, they deal with this kind of problems very quickly. I don't want to blame them for anything - they just serve here as examples.

Discourse

Let's start with Discourse:

1. Digest mail ignores secure groups

http://meta.discourse.org/t/digest-mail-ignores-secure-groups/7288

People receiving the digest mail can easily read posts not meant for them. That's because the digest mail ignores the secure groups a member has access to or not.
Quite a problem as I unfortunately found out.

2. Non-authenticated users see private topics in 404 page

http://meta.discourse.org/t/non-authenticated-users-see-private-topics-in-404-page-was-mobile-view/7419

Discourse correctly prevented me from seeing the topic and instead showed a 404 page. On that 404 page, the lists of "Latest Topics" and "Recent Topics" showed private topics. I could click those links, which resulted in seeing the same 404 page again.
The 404 page should not show private topics.

3. Auto-suggest topics shows private topics

http://meta.discourse.org/t/auto-suggest-topics-shows-private-topics/7418/3

We've got Discourse running with private categories. When a user without access to the private categories type a new topic, they are presented with topics in categories to which they don't have access.

Comment: Did you notice the pattern here?

Accidentally, I've had a small conversation at HN with one of the Discourse founders. He said:


Those private topic bugs are not the result of ActiveRecord. We added a group layer on top of existing code and missed some places where queries did not respect it.

Had we used raw SQL instead of an ORM we would have had the same issues. All projects are open to this style of bug. The correct thing to do is report, close them quickly and add tests to prevent them from happening again (which we do.)

I have completely no 'science' proof here, it's just based on my experience with hundreds of Rails projects. However, I disagree that all projects are open to this style of bug. 

It's very typical to Rails projects. 

The temptation of globally accessing every model/record from every place in the project makes it very easy to introduce such bugs. When you add new requirements (as they did) it's practically impossible to find all the places which should be changed.

Do I blame ActiveRecord here? No, ActiveRecord is a good library for persistence. It has its issues, it's huge, even the maintainers consider it a legacy code. Apart from some edge cases, it works more or less ok. I think the problem is relying on ActiveRecord for basically everything.

Spree

Let's see some other bugs, this time in Spree:

1. Fix wrong discount calculation with flat percent promotions

https://github.com/spree/spree/pull/3054

Fix wrong discount calculation with flat percent promotions when there are more than one line item in the order.

This error happened because the order instance here:
https://github.com/spree/spree/blob/master/core/app/models/spree/order_updater.rb#L24

is not always the same instance in memory here:
https://github.com/spree/spree/blob/master/core/app/models/spree/calculator/flat_percent_item_total.rb#L15

Adding the inverse option to the relationship makes sure you have the same object instance in both places.

Comment: Here we have another problem related to ActiveRecord. In more complex situations (cyclic associations), you can have the same 'record' in memory as two different instances, leading to errors.

2. Admin products loads entire dataset to determine the total count when paginating

https://github.com/spree/spree/issues/2679

When kaminary determines the total count of records it runs the following code
@collection.except(:offset, :limit, :order).count

The issue is that this sequence loads entire dataset to determine the count. This makes heavy load when products have large number of items.

The reason of this behaviour is group_by_products_id here https://github.com/spree/spree/blob/1-2-stable/core/app/controllers/spree/admin/products_controller.rb#L94

Comment: This time we have a combination of ActiveRecord and an external gem - kaminari (paginator). The end result is slowness of admin panels with thousands of products - they all will be loaded to memory. It's not the worst bug I've seen, but it's typical.

3. Default Tax Does Not Calculate Taxes Correctly if there is a Promotion

https://github.com/spree/spree/issues/2493

4. Taxes should be re-calculated after promotion adds an adjustment

https://github.com/spree/spree/issues/1735

Comment: the titles should be enough. This kind of bugs is usually related to the ActiveRecord callbacks usage. In complex scenarios it's not that easy to track all the places that need to be called after some changes.

Redmine

Let's now look at one Redmine bug:

1. Time entries of private issues are visible by users without permission to see them

http://www.redmine.org/issues/12287

Comment: Does it ring a bell? Again, a leak of private information.

Summary

We have seen some of the bugs, that I called typical Rails bugs. They seem to have certain things in common. I've seen them so many times, that they are the first things I look for, when I get access to a new project ticketing system. 

Is there any way of defending against them? I think so. The best step to start with is to be aware of such problems. Try to spot them in your projects, see what history is behind the bug, spread the knowledge in your team.

In my next blog posts, I'll focus on techniques that should help decrease the number of such bugs. Stay tuned. In the meantime, you may want to follow me on Twitter and subscribe to the Rails Architectures Guide, that I'm working on.

Single Page Applications fight - video

During the last wroc_love.rb conference we've organised a "fight" between 4 different approaches to Single Page Applications. Below is the video (0.5h) and a very short and simplified transcript from this event.

The participants, from the left-side:

Patrick Mulder - Backbone
Piotr Sarnacki  - Ember
Andrzej Krzywda (me) - Hexagonal
Adam Pohorecki - Angular

moderator - Jan Filipowski

1m39s

Can anything be a Single Page App?

(and discussion about SEO)

Patrick: only for some apps

Drogus: for interactive apps, not for static apps that need to be indexed by Google

Andrzej: for all the apps

Adam: we need better tools on the backend in order to have a better developer experience

Piotr: Andrzej, would you write wikipedia as a SPA?

Andrzej: Yes.

Adam: What about SEO?

Andrzej: It's a myth that SPAs are not indexable. Google has an official document for that. It is additional work.

Patrick: You can use PhantomJS for SEO.

Adam: We've used PhantomJS for that, took a couple of hours.

8m28s

Can you introduce the framework you're representing here?

Andrzej: HexagonalJS is not a framework, it's an architecture guide. Uncle Bob (OOP) + Raganwald (AOP) + Hexagonal Architecture. It's more like a set of guidelines

Piotr: EmberJS - it does as much as it can for managing your SPA. it's not minimalistic, for good reasons (you don't have to write much code). 

Patrick: Backbone, very small, minimal. you don't bring too much dependencies. you can choose what you want. 

Adam: Angular, Two-way data binding is the key feature. Synchronization of HTML and your JS. Choose framework with data-binding. Ember is essentially a language, forces its own object model. Backbone as well. Hexagonal or Angular is pure JS.Your application can live outside of Angular.You could use Angular with a Hexagonal app. Backed by Google, more mature than Ember.

Andrzej: Data binding is not as important as you may think. Manual control is not that difficult. Hexagonal doesn't have data binding jilt-in. Apps are not about data changes, but about actions

Piotr: I agree that data binding is important. Both, Angular and Ember are good for big applications. Nested views, zombie views (backbone). Angular has its own compiler for DOM elements. You need to know how the compiler works. Angular uses dirty checking. Angular claims that your UI shouldn't have than 2000 elements. You're screwed if you have more.

Adam: We built a 100 rows, 500 columns table. We managed to do it with Angular. It's not that hard to do the UI part. Directives are like helpers for html. 

22m6s

Do you have to take the mindset of the creator?

Adam: Most of the things in Angular are optional.If you don't use the HTML part, you lose a lot.

Andrzej: It's a problem of all of us here, including Hexagonal. We use different terminology for different things. All the frameworks are far from being Rails, we're not mature enough. All the frameworks are in their infancy years.

Patrick: It's easy to work with Rails and Backbone

Piotr: Ember has its own class model. It's done mainly for making it easy to optimise. Ember is about Convention over Congifuration. Rails in JS. Ember is like Rails, Backbone is like Sinatra. At some point you have to use more plugins if you start with something minimal. A lot of pain if you want to write a big app with a small framework.

Andrzej: I disagree. I would use a framework for a prototype or for a small app. Once the app grows, it has its own world. I don't want the framework to go into my way. I want to have the control over the applications. I'd use Ember for smaller apps, not for bigger apps.

27m23s

How to be involved in the community, how to learn the framework?

Adam: Angular has really good docs. All development happens on github.

Andrzej: Hexagonal has just started. It got the name just recently. We've used it for over a year. We have a strong team behind it. Our goal is to show the guidelines.

Piotr: Ember is 1.0RC now. Docs are better now. IRC is very active. Google Group.

Patrick: JavaScript and Ruby devs can help.

END

If you're interested in Single Page Apps, you may want to follow me on Twitter.

Why do projects take so long?

Some projects take a long time to develop. In different contexts, the definition of "long" may vary. It was always interesting to me, what are the reasons for it. Recently, though, it became my obsession.

I've started collecting some thoughts on this topic.

Ask yourself and your teammates about the last project you've been working on. "Would we be able to implement the same result in 1 month? In 1 week?".

Surprisingly often, I'm answering to myself - yes, that would be possible, given some assumption. I've had similar answers from my colleagues.

Where does it come from?

Is it the known tendency to underestimating a problem by developers? I don't think so, as we're talking about a project, that has already happened. We know what is to be done. We are aware of edge cases.

"This is time it will be different, faster"

This is another popular trend among us, developers. Most of us are optimistic, even the experienced ones. It's not some kind of a student-syndrome (I'll do it tonight), but a well-thought statement. Is it really different and faster? Not always…

Where does this optimism come from? Good developers become better at solving technical problems. With each problem solved, we know that we solved the problem forever. We love solving problems, every day we're smarter by this attitude. We're progressing and the optimism may come from that.

It's almost impossible to get a second chance of working on the same project. What would that even mean from a business perspective? We rarely get that chance.

Is it the technology that let us think, that we could do better this time? That's a serious candidate. Every time, I switched to better technologies or to better architectures, I noticed the speed increase. I was more confident. I saw the results.

I remember a situation, back in 2004/2005. I was involved in preparing an estimate for a new project. The estimate was about 6 months for 4 developers (assuming ASP.NET). Very shortly after that I learnt Rails. I realised, that the project is perfect for Rails. It was a CRUD-based web app. I kept convincing the management to give it a try. No luck. Finally, I had a chance of coding live in front of some managers. I implemented the main path in about 1 hour (I was well prepared). That convinced them. We've finished this project in 1.5 month, with one junior programmer and me being mostly a PM and a Rails mentor.

What's the lesson? It's hard to tell. I'm not trying to say that only Rails was the reason. Motivation? The passion behind it? Trying to prove something? The focus? I think it was a combination of all of the above aspects.

Is it easy to repeat those aspects? Do they sound familiar?

Have you ever attended a hackathon?

There are many kinds of hackathons. What's most important in this context, is how much can be done in a very limited time. If you look at the results of Rails Rumble, some of the apps are quite big. Many features, everything is polished.

Hackathons are proves that we are able to develop apps quickly.

How can we apply hackathon lessons to our every-day projects? Is it possible?

What else can help us developing software faster?

I would love to know your opinions.

Angular or Ember? Congratulations, you've already won.

At the wrocloverb conference we've decided to have a Single Page Applications "fight". (The format of fight worked pretty well last year for the JRuby vs Rubinius battle).

We're having Ember, angular, Backbone and hexagonaljs representatives. As I'm representing the last one, during the last few days, I've did more research on the other alternatives.

There are huge differences between all the choices. Some of the frameworks are like Rails, some are more like Sinatra. Hexagonaljs is not even a framework.

I have been making notes on the differences, collected jokes on each of the opponents. It's going to be a great fight :)

Anyway.

I realised one thing. If you're wondering which of the frameworks, non-frameworks to choose, you have already won. You are already in the world of Single Page Applications. There's no wrong choice in here. You will be fine with any of the frameworks.

There's a warning, though. I know many people who entered this world and they don't want to come back to the non-SPA reality.

Congratulations, that was a good choice!

Non-framework approach to JavaScript apps - hexagonal.js

This week we have officially launched and announced hexagonal.js.

The announcement was welcomed with a fair amount of interest. Jan Filipowski at the Arkency blog did a great job of explaining the basic ideas.

I'd like to cover the topic of hexagonaljs not being a framework.

What is a framework?

Some time ago I asked "What is a framework?". I liked the following answer:

In the case of a framework, your code is called by it. In the case of a library, your code call it. Rails is a framework, and provides libraries too.

I have a love/hate relationship with frameworks. That's why I struggled for months, what to do with the ideas that we're now calling hexagonal.js. I didn't want it to be a framework. That would ruin the whole experience.

Let me tell you a story on how it all started and why we didn't want it to be a framework.

The technical story of haxagonal.js

The first ideas were created when we were working on social web games at GameBoxed. Given the scale we were operating on (sometimes 4-5 social apps/games a month) we needed a high degree of reusability. Reusability requires good modularity. Often, we've had apps that have the same "logic", but they had much different GUI, sounds, FB integration etc.

We got the idea of use cases as objects from the DCI approach (DCI is much more than that!). We've experimented with the concept that it's the use-case that drives everything. Driving something (like playing sounds) meant calling it. Calling it directly meant creating a coupling between the use-case and the SoundAdapter, which didn't sound like a good idea.

One approach was to use events. The use case would trigger an event and the adapter would bind to it. It resulted in less coupling. It wasn't bad. To me, it wasn't yet the final solution, though. I didn't like the fact, that the use-case objects were full of "trigger someEvent" statements.

When I looked at the events, I realised that they happen usually at the beginning or at the end of a method in the use case.

Before, After....

Wait, wasn't that what AOP was about?

Introducing some simple AOP techniques (thanks to awesome Raganwald's library!), we have achieved exactly what was needed. This is what we currently call a Glue code. It's almost like a configuration, in most cases. No logic, just declarations.

Hexagonal architecture

We started with MVC in mind. The most basic rule of MVC is that when the model changes, the views are notified about it. After some time, we've realised that it's not only about views, unless you try to fit "playing sounds" into the View layer ;)

In a hexagonal architecture, GUI is just one of the adapters. It's usually the most important one, but technically, still, just an adapter.

In many apps, the GUI is the only adapter.

ServerSideAdapter, PusherAdapter, SoundAdapter, FirebaseAdapter, FacebookAdapter, LinkedInAdapter - those are just examples of the adapters, that we used so far.

It's not a framework

Given all of the above, we were struggling on how to bring this idea to people, without making them depend on any framework. The whole idea is a bit anti-framework.

Hexagonal.js is not even a library. It's mostly an architectural style adapted to JavaScript apps. We provide many examples (and more in progress..), that can guide you. You can copy them, fork them and have a good starting point.

It's probably a slightly unusual approach to deliver some ideas to you. It's not a framework, not a library.

Hexagonal.js is a set of good patterns that are extracted from real-world production apps. It was used to build dozens of apps, with a good speed of development.

You can look at the examples we already have. Look at the use case object, the glue and the adapters. It's a pattern that reappears in every hexagonaljs app. There is also an example of using it from a Rails app.

Tell us what you think about this approach. We've published it to bring more ideas. We're open for suggestions!

Thanks for reading.

Frontend first

Some time ago, I blogged about the different phases of mental transition when working with JavaScript frontends. Here's the shorten version:

1. No-JavaScript phase

2. JQuery explosion

3. Page and Widget objects

4. Single Page Application

Once you're in phase 4 and you create apps this way, you realize that you don't need the backend in order to create the Single Page App. Even more, you shouldn't start with a backend.

Frontend first

This is the additional phase. Frontend first. In most of the typical situations, it's much better to start with a Single Page Application and use LocalStorage or some kind of a InMemoryServerSideAdapter for the first phases of the development.

Only after you know how the frontend works and what kind of server interaction it requires, it actually makes sense to start working on the backend.

We've tried this approach in several Single Page Apps and it worked great for us. It now seems like the most natural way of working.

to learn more about Single Page Applications.

What is a framework?

The title question made me struggle for months.

1. What is a framework?
2. What is the difference between a library and a framework?
3. If I use a framework as a library, does it make it a library?
4. Is Rails a framework?
5. Is Backbone a framework?

Can you help me with your answers?