A Learning Organization


A lot of talk has gone around about the importance of becoming a “learning organization”. But what does that mean and how can it be achieved? Is an organization safe because it provides a professional development stipend in its benefits package?

Providing time and resources to your employees for professional development is great. But it is the very first step in a long journey. While having a workforce that is capable of adapting to changing practices and techniques is necessary, it is not sufficient to keep you competitive in the modern software landscape. Sending your workers to conferences and buying them books will not make you a “learning organization”.

Becoming a learning organization is about continuous experimentation. The primary type of learning that you will need to be doing is not the type that can come from books or conferences or classes. The competitive knowledge that you are after is mostly hidden. It is knowledge that either does not yet exist, or exists as a trade secret locked deep within your competitor’s corridors.

The kind of knowledge that a learning organization acquires is a deep, mostly hidden knowledge. It is the knowledge that informs you of what will provide the most value to your customers. It is the knowledge that enables you to increase your customer base while keeping your existing customers. This knowledge is something that only your customers can provide to you.

Your customers may not even know that they possess this knowledge. They give up this knowledge through increased use of your platform. They give it up through increased satisfaction and NPS scores. They give you this knowledge when your UX tooling informs you that a new feature allows your customers to complete the same workflows in 28% less time. They give up this knowledge when they spend 15% more annually on your products and services or when the internal system you built increases communication between departments and reduces mistakes by 18%.

To be a learning organization you must experiment. You must experiment faster than your competitors, or you will ultimately lose.

The results of those experiments become the secret knowledge that gives you an edge in the marketplace (but only for a little while: there is always someone else experimenting). It is the foundation upon which you build the future of your company. It is how you provide radical value to your customers.


To be a learning organization you must experiment often. To experiment often you must reduce the cost of experimentation. To reduce the cost of experimentation you must reduce the size of the experiments. You must automate everything about the experiment that can be automated. You must reduce or eliminate wasted effort and unplanned work. You must constantly work to identify bottlenecks within the delivery workflow and eliminate or control them.

“The bottleneck should be the creation of good ideas”

Eric Ries and Steve Blank (quoted in Beyond the Phoenix Project)


I would argue that continuous experimentation lays at the very heart of all of agile and all of DevOps. If you could give the product team a magic wand that allowed them to instantly create and deploy new ideas and just as quickly eliminate them if they are not successful you would have finally hit the target of everything that agile and DevOps principles are aiming for.

We will of course likely never get there. In the meantime, though, we can aim to get as close to continuous experimentation as possible. Small improvements and features deployed continuously with strong feedback loops and telemetry provide the foundations for an organization that is continuously learning; enabling you to tap into the arcane and prized knowledge hidden deep within the customer’s experience. The knowledge that your competitors currently lack.

A Learning Organization

Coupled Microservices

There are a few primary high-level advantages and one major drawback that occur to me when I think about making the transition from monolith to “microservices” (I use quotes here for these reasons). Some of the major advantages are logical isolation, ease of deployment and independent scaling on your infrastructure. I will go into more detail on each of these, but that is not the point of this post. My goal here is to point out how you can easily lose most of the advantages that microservices confer and be left with only the drawback(s).

Logical Isolation

Logical isolation means that you have separated one logical, cohesive “chunk” of your application into its own independent system. It reduces cognitive overhead for the developers working on that system by allowing them to focus on the function, structure and features of this service alone. This service can be independently managed by a small team without as much regard to the system as a whole. This makes these teams more autonomous, reducing the burden of the “mythical man month” problem and enabling faster parallel development of the system as a whole.

Ease of Deployment

Because each service is smaller and maintains a high level of independence from the rest of the system, services can be deployed much more easily. The communication between teams and team members that is necessary around each deployment is reduced (ideally to nearly zero). Changes can become much smaller, and smaller is significantly better in software development. Smaller changes can also be deployed more often thus enabling one of the major goals of devops — continuous delivery. More deploys means that we significantly reduce the average amount of time between a developer introducing a bug to the codebase and the discovery of that bug. This leads to a much lower mean time to resolution as the developers that worked on the buggy portion of the codebase worked on it more recently. Lower mean time to resolution leads to happier customers and happier, more productive developers who are freed up to work on more important things. All in all, ease of deployment should be a major goal of any agile team and having many smaller, independent services helps to enable this.

Independent Scaling

Independent scaling refers to the fact that smaller services can more easily be split up and grouped according to usage and infrastructure requirements. Services that receive more usage and traffic can be given more raw horsepower to do their job. Services that are used infrequently can be given much less in the way of compute resources, or they can be grouped together. All in all, having many small services gives you a lot more flexibility in how you build your infrastructure, often enabling quite a bit in cost savings.

Drawbacks

One “drawback” to microservices is that when you transition from monolith to microservices, you move from a centralized system to a distributed system. This is usually touted as an advantage, but if you lose the other advantages, this quickly becomes a liability.

Distributed systems are inherently more complex, overall, than centralized systems. They are more difficult to reason about. Consistency of data comes to the forefront as an issue that you need to worry about. Tracking down errors becomes more difficult as your logging systems are (by default) also now distributed. In a distributed system, you need to be much more deliberate about your logging and event tracking systems to track data and events as they propagate through your system (so you can quickly isolate errors and track down bugs).

It can also be more difficult (and expensive) to find developers with expertise in distributed systems. While developers are increasingly becoming accustomed to working in such environments and thinking in a distributed manner, it is still a relatively new way of thinking for your average-sized company.

How To Be Left With Only The Drawbacks

When you have the advantages of logic isolation, ease of deployment and independent scaling the issues that come with a distributed system are often worth the trade off. In addition, they can be mitigated to some extent by deliberate effort and a thoughtful implementation of your architecture and the support systems that you provide your developers and operations people.

So how do you ensure that you gain the advantages of logic isolation, ease of deployment and scaling?

The simple answer is: decouple your services.

You must ensure that your services remain independent to the highest degree possible. There are many ways to achieve this, and much of the Old Wisdom™ of good software engineering and systems design can be applied here. But I want to point out one of the most important factors: versioning.

Version your services. Without versioning, you end up with a highly coupled distributed system and you lose many of the most important advantages that microservices provide. You essentially end up with a distributed monolith, the worst of all possible scenarios

If your services are not versioned, you lose logical isolation. Developers on one team have to think about how their changes will affect developers on another team. This increases their cognitive load and developers are resigned to thinking about the system as a whole again. Cross-team communication is now much more critical and you lose much of the team autonomy and team parallelization that microservices are supposed to allow.

Deploys become a coordination nightmare. Teams will spend hours coordinating their deploys so as to ensure no interruption to other services. Once again, cross-team communication is increased and autonomy is decreased. Cross-team blaming is also reintroduced, as you will very often have to release different apps simultaneously. When something fails, both sides are apt to blame the other (even if politely…). If you have versioned your services, however, they can (and usually will) be released independently.

What about independent scaling? Well, the good news is that even if your services are deeply coupled, you can still scale the machine(s) they rely on independently. However, if your services are still tightly coupled, how much money will you actually save every year through independent scaling? Does this savings outweigh the cost of the added cognitive overhead (read: more developer time/developers) that your poorly designed distributed system has created?

Conclusion

The conclusion is simple: if you are building out a microservices architecture, decouple your services or else you will likely end up with a more costly distributed monolith. Implement good systems design and thoughtful abstractions and by all means, version your APIs.

Coupled Microservices