The Four Architectures that will inspire your programming

This is the Big 4. The architectures, that may help your work. They may influence the way you think about programming.

1. The Hexagonal Architecture

(also known as the Ports and Adapters)

Originally described by Alistair Cockburn in http://alistair.cockburn.us/Hexagonal+architecture.
Enhanced and popularised by Steve Freeman and Nat Pryce, thanks to their book: http://www.growing-object-oriented-software.com/

2. Clean Architecture

Originally described by Ivar Jacobson, but popularised by Uncle Bob: http://blog.8thlight.com/uncle-bob/2012/08/13/the-clean-architecture.html

3. DDD/CQRS

DDD: Domain-Driven Design
CQRS: Command Query Responsibility Segregation

Two separate concepts, but together create quite a unique architecture.
DDD is originally invented by Eric Evans. CQRS is often popularised by the work of Greg Young.

4. DCI

DCI: Data Context Interaction

James Coplien, Trygve Reenskaug
Desribed in James' book: http://www.leansoftwarearchitecture.com/ and here: http://www.artima.com/articles/dci_vision.html


What I think about them

I don't treat them as competitors. Learning the concepts behind them (still in progress) inspired me to bring some new ideas to our projects. Usually, they're not all or nothing.

The most mind-blowing architecture is definitely DCI. This is the future of programming. The vision of objects being different things in different contexts, while still keeping its identity is fantastic. Unfortunately, DCI is also the hardest one to use in day-to-day practice. Some of the concepts required (objects with roles) are hard to achieve in most popular languages.

The most practical architecture is possibly DDD/CQRS. There is a lot of example projects in many different languages. I love the pragmatic approach of splitting the system into Commands and Queries. At first, I couldn't get the point, but after more reading I see the beauty of it. I sometimes laugh that DDD could stand for (data) Duplication-Driven Design. It opened my eyes to the idea, that we can have multiple subsystems, each with its own storage. Some data will be duplicated this way, but it's often not a problem. The data is eventually consistent.

Another nice thing about DDD is the idea of Bounded Contexts. It's a terrible name, but a really useful concept of a module that encapsulates one little world of your project. Usually, you have one root object in each Bounded Context, which is called Aggregate. The same 'object' can exist in different BC, but will have a different name. Its identity will be kept via a unique id.

CQRS brings the concept of Commands being the most complex parts of your app, while Queries are usually very simple and don't require many layers.

What I like about the Clean Architecture is its focus on clear dependencies and on boundaries. Boundaries should be kept outside of your application. The communication between boundaries should happen via Data Transfer Objects. In the middle there's the Interactor, which represent a UseCase.

Last, but not least, we've got the Hexagonal Architecture. This is what inspired us mostly to create the JavaScript non-framework called http://hexagonaljs.com/. It's so easy to extract GuiAdapter and StorageAdapter, leaving the middle hex boundary-unaware.

HexagonalJS is a result of inspiration from all of the above architectures. We took the UseCase/Context part from the DCI architecture with a little bit of the Clean Architecture. The idea of adapters is taken from the Hexagonal Architecture. The middle hex is often implemented using the building blocks taken from DDD. Our unique addition to this setup is a little bit of Aspect Oriented Programming.

Often, when I'm in doubt how to approach a new project, I'm trying to stand far from the Upfront Design. However, it doesn't hurt if I imagine the project in each of the architectures. This gives me new ideas about the domain I'm implementing.


If you liked this blog post, you will enjoy following me on Twitter.

Turn a Rails controller into a Single Page Application

Single Page Apps are becoming more popular. When you see them in practice (GMail, Twitter, Facebook), it's quite cool. Then you look at your current project and you see how impossible it is to make it a Single Page App. Who's going to pay for that? It sounds like months of work, just replacing the whole existing functionality.

The thing is, it doesn't need to be the whole project at once. It's not all or nothing. You can make multiple small SPAs.

Where to start?

Look at your current Rails views/controller. Is there a clear widget, like a search bar, that could be extracted? That's a good candidate.

There's also another approach. Usually when you have a typical CRUD controller, it's always the same group of views. There's a list of things, that you can see, edit, update, add new items, etc.

Such controller is a good candidate for a SPA.

You end up with one view (index), which loads the JS. After the JavaScripts are initialised, an AJAX GET request is made to /index.json. This retrieves all the data for the list. Apart from that you need an API endpoint for creating, updating and deleting. Note that you don't need an endpoint for new/edit. They become JavaScript views, but they don't need to exist on the server-side. 

You turn the existing Rails views into handlebars templates (or whatever that renders the HTML client-side). They get rendered at appropriate moments by the JavaScript code.

If you prefer, you can use a framework for that. You can also do it on your own, as it's very easy. We usually go the hexagonaljs way - manual, with a nice structure and typical adapters: serverSide and guiAdapter. 

What does it give you?

• Single Page Apps are architecturally better solutions than Rails views
• it's easier to test
• once you learn it, it's quite easy to write
• there's strong decoupling between your frontend and the backend
• the backend is simpler
• the resulting SPA has less 'features' than a Rails view
• by default, no URL for each 'view', just for the main one
• good enough for most cases
• for SEO you need to do additional work on the backend
• not always needed

It doesn't need to be all or nothing

You can test the water by choosing some parts of your project and turn them into JavaScript apps. We're in the process of doing that in a large Rails project - I think, we're now with about 15 small Single Page Apps. We keep adding new features, it doesn't slow us down. Thanks to the faster build (it's easier to test decoupled components rather than a monolith), we're actually saving time.

The Rails-way and bigger projects

The whole idea of having a data-structure that you retrieve from the database and carry through all the layers to the UI makes the development very quick at the beginning.

The coupling it provides, at the later stages of development makes it very difficult to split the app into any kind of modules. It's also difficult to test anything in isolation. That's why Cucumber, Capybara, Selenium are so popular, as they help you in writing integrated tests.

Integrated tests have their place, but it's impossible to test everything through all the layers. This results in slow builds - a typical syndrome in Rails projects.

The Rails way makes sense for smaller projects. What I'm disagreeing with is using "The Rails way" in bigger, serious projects.

The Rails itself has nothing inherently wrong. I'm not attacking Rails, I'm only attacking 'the Rails way'. You can use Rails for bigger projects, if you know what you're doing.

What I'm teaching at my Rails class

It's now the 4th time, I'm teaching the Rails class at the University of Wroclaw. The main goal of this class to prepare people to work in a Rails-based company. The students are the best of the best, so the goals can be high.

Every year, I introduce some changes to the program of teaching to reflect better what is happening in the Rails community.

This year I'm going to focus more than before on the integration with mobile, native apps and on the art of dealing with legacy projects.

We start with extremely quick Ruby lessons, based on the Ruby koans exercises (students do it on their own).

The first 3 lessons are about "the Rails way", which I make it clear - is good only for toy projects. This is where I teach about ActiveRecord-oriented programming, coupling everything with everything, making things impossible to test - you know, the usual Rails stuff.

Once we know the enemy and know the pains it provides, we learn how to avoid that.

This is where good, solid OOP lessons come in, service objects, repositories, domain objects, etc.

After that, we enter the "frontend" module. We learn how to create Single Page Apps using the dominant JS frameworks (Angular) and some no-framework techniques (HexagonalJS. This also includes techniques like Pusher to have server->client communication possible.

We also talk a lot about the way, a Rails backend can expose data through the API to the JS or mobile app clients. Mobile apps will take more time this way, as I want to show how to best cooperate with mobile developers.

There's a lesson about proper Rails deployment, things like Chef, Jenkins, Continuous Deployment etc.

Throughout the class, we'll not only create new Rails apps (lots of them), but I also want to teach how to find yourself in a legacy codebases, like Redmine or Discourse, how to refactor out from the mess, fix bugs, add new features - that's probably the hardest part of the class.

Testing is an ongoing topic in the class, as well. We'll learn about unit testing of different layers, integrated tests and their pains.

I hope it will be fun, as always :)

Implicit Rails features

This post is part of a series of blog posts. So far, there were 2 blog posts, which aim at helping you improve the modularity your Rails applications:

1. Typical Rails bugs

2. ActiveRecord overdose

3. Implicit Rails features

The speed with which you start every new Rails application is amazing. 

Within a few days you can have a fully working prototype of almost every application. Obviously, the prototype, by definition, may not be fully ready to production use.

With a really small codebase, the prototype does a lot of things. It's all thanks to many built-in Rails features. 

I call them implicit features. They come almost for free. 

• CRUD
• validations
• maintaining created_at and updated_at
• displaying errors
• displaying flash messages
• redirects between pages
• pagination
• uploads
• sending emails
• attr_accessible/attr_protected (security)
• CSRF - security

There's one downside to the implicit features. 

They're rarely documented and covered with tests.

As with every app, there comes a moment, when you need to start providing changes. Sometimes, you may want to rewrite some layers of the application. There's a trend recently that we try to find new ways of working with Rails apps, that is less dependent on ActiveRecord. Some developers rewrite their apps to Sinatra or Padrino. 

It sounds great in theory, but it's harder in practice.

It's good to be aware of such hidden functionalities. At some point, they can become explicit features, with their own tests, so that it's never broken.

If you know more such features, share them in the comments.

ActiveRecord overdose

In the previous post, I highlighted the typical Rails bugs, that I encountered during my research on better ways of working on Rails apps. The bugs were often security-related:

The main common point of the bugs is the ActiveRecord overdose.

ActiveRecord overdose is a surprisingly popular pattern that includes:

• putting everything into the model class
• fear of adding new non-AR classes
• over-relying on convention over configuration

When you look at a typical Rails model, it's a mix of the following responsibilities:

• persistence 
• business logic 
• adapters 
• view object 
• form object 
• others 

The Rails-way ActiveRecord is an enhanced version of the original Active Record pattern. The original pattern assumed only persistence and some simple business logic. The Rails version extended it with the view/form responsibilities. In many cases it's also extended with the adapters.

• persistence 
• associations 
• queries - scopes 
• saves 
• simple validations 
• business logic 
• state-machine 
• calculations 
• tracking changes (touch) 
• complex validations 
• some callbacks 
• feed/timeline/activity 
• factories methods (class methods) 
• adapters 
• service (coordinating multiple adapters) 
• search 
• external api 
• facebook 
• twitter 
• exception tracker 
• metrics tracker 
• image storage 
• pdf 
• mailer 
• notification 
• view object 
• first_name + last_name 
• foo.to_s 
• image_path 
• http boundary details (urls) 
• i18n 
• form object 
• accepts_nested_attributes_for 
• sanitizing 
• attr_accessor 
• others 
• acts_as_foo (persistence, logic, validations) 
• concerns 
• includes 

The core responsibility of an ActiveRecord model is being a representation of a db record. That means handling the persistence and having the basic logic. This usually works great in typical CRUD apps. Once things get more complex, it's time to reconsider the active record pattern. 

The best way of escaping from the ActiveRecord overdose is to do it step by step. I'll focus on that in my later posts.

Typical Rails bugs

As part of my research on improving Rails application, I noticed a pattern in the bugs that are quite characteristic to Rails in several applications.

They have certain 'visible' things in common:

• often security related, like leaking some information to unauthorized users
• they live somewhere in the area of business logic/persistence
• they are not so easy to fix in the existing codebase
• they tend to exist in groups, similar bugs in different areas
• they are easy to miss
• often they appear during the requirements changes

Overdose of ActiveRecord

The reasons they exist is a combination of different things. The main common point of the bugs is the over-usage of ActiveRecord. It's easy to fall into this problem, I did it many times, as well.

You start with a new Rails app, adding new features very quickly. New requirements pop up. You use the popular Rails techniques to deal with them:

• validations, which often turn into conditional validations
• Single Table Inheritance
• state-machine
• accept_nested_attributes
• models callbacks
• virtual attributes
• external gems that provide value magically, by using ActiveRecord (implicit dependencies)

Every technique is fine on its own. When you start using all of them in the same area, problems appear. 

Examples

I started noticing this trend, while reviewing the code of our potential Arkency customers. We sometimes do the "rescue missions" and help with legacy code. I can't use those examples here, so I started reviewing some of the popular open-source Rails applications: Discourse, Spree, Redmine, Gitlab. It didn't take me much time to discover the same bugs.

Just to be clear, all of the projects are awesome and I'm grateful to the people behind them. Most of the times, they deal with this kind of problems very quickly. I don't want to blame them for anything - they just serve here as examples.

Discourse

Let's start with Discourse:

1. Digest mail ignores secure groups

http://meta.discourse.org/t/digest-mail-ignores-secure-groups/7288

People receiving the digest mail can easily read posts not meant for them. That's because the digest mail ignores the secure groups a member has access to or not.
Quite a problem as I unfortunately found out.

2. Non-authenticated users see private topics in 404 page

http://meta.discourse.org/t/non-authenticated-users-see-private-topics-in-404-page-was-mobile-view/7419

Discourse correctly prevented me from seeing the topic and instead showed a 404 page. On that 404 page, the lists of "Latest Topics" and "Recent Topics" showed private topics. I could click those links, which resulted in seeing the same 404 page again.
The 404 page should not show private topics.

3. Auto-suggest topics shows private topics

http://meta.discourse.org/t/auto-suggest-topics-shows-private-topics/7418/3

We've got Discourse running with private categories. When a user without access to the private categories type a new topic, they are presented with topics in categories to which they don't have access.

Comment: Did you notice the pattern here?

Accidentally, I've had a small conversation at HN with one of the Discourse founders. He said:


Those private topic bugs are not the result of ActiveRecord. We added a group layer on top of existing code and missed some places where queries did not respect it.

Had we used raw SQL instead of an ORM we would have had the same issues. All projects are open to this style of bug. The correct thing to do is report, close them quickly and add tests to prevent them from happening again (which we do.)

I have completely no 'science' proof here, it's just based on my experience with hundreds of Rails projects. However, I disagree that all projects are open to this style of bug. 

It's very typical to Rails projects. 

The temptation of globally accessing every model/record from every place in the project makes it very easy to introduce such bugs. When you add new requirements (as they did) it's practically impossible to find all the places which should be changed.

Do I blame ActiveRecord here? No, ActiveRecord is a good library for persistence. It has its issues, it's huge, even the maintainers consider it a legacy code. Apart from some edge cases, it works more or less ok. I think the problem is relying on ActiveRecord for basically everything.

Spree

Let's see some other bugs, this time in Spree:

1. Fix wrong discount calculation with flat percent promotions

https://github.com/spree/spree/pull/3054

Fix wrong discount calculation with flat percent promotions when there are more than one line item in the order.

This error happened because the order instance here:
https://github.com/spree/spree/blob/master/core/app/models/spree/order_updater.rb#L24

is not always the same instance in memory here:
https://github.com/spree/spree/blob/master/core/app/models/spree/calculator/flat_percent_item_total.rb#L15

Adding the inverse option to the relationship makes sure you have the same object instance in both places.

Comment: Here we have another problem related to ActiveRecord. In more complex situations (cyclic associations), you can have the same 'record' in memory as two different instances, leading to errors.

2. Admin products loads entire dataset to determine the total count when paginating

https://github.com/spree/spree/issues/2679

When kaminary determines the total count of records it runs the following code
@collection.except(:offset, :limit, :order).count

The issue is that this sequence loads entire dataset to determine the count. This makes heavy load when products have large number of items.

The reason of this behaviour is group_by_products_id here https://github.com/spree/spree/blob/1-2-stable/core/app/controllers/spree/admin/products_controller.rb#L94

Comment: This time we have a combination of ActiveRecord and an external gem - kaminari (paginator). The end result is slowness of admin panels with thousands of products - they all will be loaded to memory. It's not the worst bug I've seen, but it's typical.

3. Default Tax Does Not Calculate Taxes Correctly if there is a Promotion

https://github.com/spree/spree/issues/2493

4. Taxes should be re-calculated after promotion adds an adjustment

https://github.com/spree/spree/issues/1735

Comment: the titles should be enough. This kind of bugs is usually related to the ActiveRecord callbacks usage. In complex scenarios it's not that easy to track all the places that need to be called after some changes.

Redmine

Let's now look at one Redmine bug:

1. Time entries of private issues are visible by users without permission to see them

http://www.redmine.org/issues/12287

Comment: Does it ring a bell? Again, a leak of private information.

Summary

We have seen some of the bugs, that I called typical Rails bugs. They seem to have certain things in common. I've seen them so many times, that they are the first things I look for, when I get access to a new project ticketing system. 

Is there any way of defending against them? I think so. The best step to start with is to be aware of such problems. Try to spot them in your projects, see what history is behind the bug, spread the knowledge in your team.

In my next blog posts, I'll focus on techniques that should help decrease the number of such bugs. Stay tuned. In the meantime, you may want to follow me on Twitter and subscribe to the Rails Architectures Guide, that I'm working on.