Thursday, September 17, 2015

Start from the middle

The order in which requirements are implemented can have a huge impact on the success of the project. I often recommend a radical approach of starting from the middle to address this issue.

What does it mean?

It can work on different levels. It works on the whole project, but it can also work on a single feature. 

Don't focus on providing the data

Assume all the data involved in the feature already exists. Don't focus on the way the data is provided. Don't implement the importing feature at the beginning. Focus on what is possible to do with the data in the context of the project (or feature).

Let's say you implement a system which shows the project and its tasks (a project management tool). Starting from the middle would mean focusing on: 
  • how the project/task views looks like, maybe hardcode the HTML/CSS
  • what you can do with the tasks
    • prioritise
    • assign to people
      • all the consequences of assigning a task (emails, notifications)
    • complete the task

Consequently, you're NOT focusing on:

  • how the people are imported to the system (do they need to register? are they imported?)
  • how the projects are created
    • it's not a common task to start projects
  • authentication
  • persistence
You try to focus on the actions that are in everyday use of this project (potentially).

What if you really need to use the real data?

This obviously depends on the context of the system. Given the above example, you probably have 2 or 3 initial users of the system. Ask them to provide the data in any way they like. It might be Excel, Google Doc, just an email, whatever. You can quickly insert the data to the system, using the console/database.

Thanks to that you will see what kind of data those people *actually* use. Do they have 100 projects or just 3 of them? Do they have 5 employees or 500? This will guide you how to implement the importing, later on.

Most importantly, you don't waste time on implementing the less crucial parts of the app. Most likely, the importing is not your business advantage.

In the first days, you don't even need the proper persistence - focus on manipulating the existing data, but it doesn't need to be persisted (in the first days).

You don't need the proper authentication, yet. Everybody knows you're able to implement authentication, once it's actually the most important part. Focus on the risky parts, not on the ones, you know how to do.

Technical decisions

In Rails, we often don't know, what the app is about, but we know it's going to be Rails, Devise, RSpec, Postgres.

In Java, it's the same, Spring + JPA/Hibernate + Oracle, before we know anything here.

If it's a web app, consider starting with the frontend app, but don't go into Angulars/Embers or any other heavy framework. Make it a lightweight HTML + some simple views. They can be jQuery-based or React. You will change it later.

Starting from the middle is not only a prototyping tool. Starting from the middle is a business thinking tool. 

What in our app/feature is the most important/risky part?

How can we fake the other parts? How can we defer some of the less important decisions?


BTW, one practical tip - when a software project starts, many people care a lot about many things, especially the technical decisions. You may spend days debating rspec vs minitest or rails-api vs sinatra. Somehow, those are the most important decisions at the beginning. Everything changes after 2-3 months. Suddenly, introducing minitest is not a big deal. It's no longer so important that one part of our app will be React-based and not Angular. It still matters, but the discussions are not so heated. 
Use it to your advantage - suggest deferring decisions "until the next week/month". Try to move the discussion to the core of the requirements, at this stage of work.

Let's say, you're adding an analytical feature to the app. You want to collect a lot of data about the users behaviour and then implement a dashboard which show nice things.

Collecting analytical data is not really risky (maybe the performance part). Focus on what kind of nice things you want to display. What should be highlighted in red? Do we needs some alerts here? You will quickly realise that those things can guide the collecting phase. It will be easier to decide on the ways the collecting needs to be done. Do we need to collect data in batches? How real-time does it need to be?

Detecting actors early

What's most important, by focusing on how we want to use the data, we need to start thinking in terms of actors. Who is going to use the data? Do we have access to those people? The earlier we try to find them, the better. We try to localise the places, which are not under our (developers) control. Fix them first. Then you're on your own implementing the stuff that is fully under your control.

Sometimes, it's just a small gain. You start the feature by visualising the data. You send it to the people involved. They can start discussing it (often takes days). Meanwhile, you can work on the persistence or authentication foundations. When they're back with some ideas of changes, you will need to do them, but at least you were not fully blocked.

If you did it the other way round - spending 3 days on persistence/backend/authentication and then implementing the main screen - you would need to wait longer now. This is using concurrency to your advantage.

Another example - you have a working system. Now, it's time to make it a SaaS. You need to make it a multi-tenant app. This is a big deal. This is not a 3 days work. There's lots of things to consider. Starting from the middle would require thinking in terms of already existing data. Don't focus (yet) how to let new companies/tenants to register. Try to turn the existing system into a tenant system, from the middle. Does it mean, we need to assume we already have subdomains? How do we persist/isolate the tenants data? Try to hardcode other tenants data. This will all guide you how to turn the app into a SaaS in a safer way. 

Starting from the middle is pivot-friendly

If you work in a fast-changing environment there might be situations where a decision to drop the idea appears. You work on a feature X, you spend 1 week on it. Suddenly, the board decides that X is no longer important. We now need Y. If you started from the middle, there's no inconsistent features that you need to disable because they totally don't make sense. Yes, you still need to clean up the code, but at least there's no need to remove the import form, as there was no such thing yet.

I use this technique to blogging. I timebox 30 minutes and I start from the middle. What are the most important messages I want to send? Make them headers. Add 1-2 sentences to explain them. When the time is over, the blogpost is shippable. In fact, it's always shippable. If I have more time, I expand the explanation.

When to stop working in the middle?

Once you have all the unknowns clear - then it's time to go back, to the beginning. We're now totally sure what our app is for. How it solves the users problems. We know the actors. We know the data specifics.

Now it's a good start to implement the missing infrastructure. 

In one of my projects, we implemented the whole app as a Single Page App with no backend at all. We collaborated with the actors, the app was tested and validated. Once all the frontend was settled, we knew the data we needed. Implementing the backend took us only 2 days. It was crystal clear. We started with the client needs first, then we knew what we needed. If we started with the backend, we would be guessing about the needs.

This is a lot like TDD, but on a different level.

Gradual migration of persistence

Let's say you start with a JavaScript frontend. You show "the middle" screens. You implement the most important actions (frontend-only). This is enough for the basic testing. 

Now the users may want to actually "save" the data somehow, so that when they're back the next day, the data is still there.

You can implement it using the local storage. The simplest technique? Just dump all the data graph to the local storage after every change. It's not the proper way, but it solves the *current* problem.

Later on, the users may want to access the data from different devices. You now need to have some backend. You can even follow the same strategy and just dump the data to the backend as a json blob.

The next requirement is usually to be able to collaborate on this data with other users, so they can work on it at the same time. Now it might be time to be more granular with how the data is persisted. You can now extend the API with more endpoints for different actions.

Ultimately, you may need to make the collaboration smoother by pushing the data to the clients using WebSockets.

Code is not a stone

Starting from the middle is a great exercise for continuous refactoring. All the code keeps changing and you're not afraid to do so. It becomes a natural habit. Your work become a sequence of small steps, gradual improvements.

Outsourcing less important parts

Starting from the middle teaches about outsourcing some work to other tools or even other teams. Do we even need our own authentication? Is Twitter login enough? Do we need our own persistence? Do we need to implement the import forms? Maybe there's already a tool which the users can use to "click" all their data and then export it to xml? Maybe there are some standards in this industry?

Optimise for the developer brain

What I often see is a number of features in the "importing" process. Those are features that may be useful, but only at the beginning of the user lifecycle in the project. However, they complicate the UX forever. You have some checkboxes which make sense to be displayed on each line item, but it's only realistically use in the first days. Even if they're already implemented, I recommend considering to drop them. Code is maintenance cost. Even if the code is not changed that often it brings the mental overhead to load the codebase to the developer brain. 

It may sound stupid to optimise the features for the developer brain. However, it's less stupid, when you think that it's the developer who is then able to come up with simpler/faster solutions, thanks to the fact that the whole thing is now simpler.


Starting from the middle is like a framework of business thinking. It constantly challenges you on the best ways of using your time. 

Are you working in the middle now? 

DCI, DDD and the concept of roles

All of the architectural approaches in software engineering aim to solve similar problems. One of the techniques that is often used is the concept of roles.

When you do OOP, you may use interfaces (especially with statically typed languages). Interfaces help us thinking in roles - in this context this is the role the object plays.

The DCI architecture is opposing the current class-oriented thinking. They aim to introduce the full OOP which is object based, not so class-based as we're used to now. To simplify things a bit, in DCI you should be able to implement an interface on an object basis, not on the class level.

For example, if you have an ebook object, you might want to temporarily treat it as "a digital product" in the Shipping context (where it's delivered via email to the customer). Not all ebooks are being delivered at the same time, so there's no point in all of them (at the class level) having the shipping abilities. For the time, when it's important to ship the digital product, the ebook has a new role. When it's used in another context (listed in a catalog?) it doesn't have the role - it doesn't have some methods.

That's the DCI conceptual way of using roles.

Now, what is the DDD way?

In DDD, we use the concept of bounded contexts. They represent the boundaries in which a different language is used. In a way, it's similar to DCI Contexts.

In DDD, we would have the Catalog context and the Shipping context as well. Similarly we will have the concept of ebook and digital products. They are not called roles, here, though. Also, it's probably more of an implementation detail, but most DDD implementations would have different objects (different identities) for the ebook (as in Catalog) and for the digital product (as in Shipping).

The ebook and the digital product represent the same thing and very likely they are aggregate roots in their bounded contexts. Keeping them as separate objects (often, with their own separate persistence) requires some synchronisation mechanism. If the ebook title changes, it may need to be changed in all contexts. In DDD, domain events are one way of ensuring the (often eventual) consistency.

As you see, different kinds of architectures need to solve similar problems. Here, both DDD and DCI use the role-oriented thinking. The implementations are different, though.