Chris' Blog

Learning to Love DDD - A Tale of Two Products

2023-01-11T00:00:00Z

Learning to Love DDD - A Tale of Two Products is a conference talk I started delivering in 2022.

In some respects, it's a love letter to Domain-Driven Design after years of building systems both with and without the practice. See below for videos of the talk at a few conferences.

Talk Description

Over the last 16 years, I've been fortunate enough to help launch two successful businesses as a hands-on CTO/co-founder & developer/architect.

The first, Flexischools, provides online services to schools & parent communities, and has supported millions of Australians over the years.

When we launched Flexischools, I was relatively inexperienced - I did my best to follow the guidance I could find online, but I frequently found myself struggling to incorporate the burgeoning growth in complexity of its feature-set into an increasingly tightly coupled code-base with a very small team of developers.

In 2015, I discovered DDD, and when reading the Blue Book, I felt a light bulb going off on every page. When we launched our second product, LanternPay, I vowed not to make the same mistakes.

LanternPay is a healthcare & disability claiming & payments platform supporting hundreds of thousands of Australian care providers and recipients to ensure prompt and equitable access to care.

In this talk, I'll reflect on the comparative experience of launching, operating & scaling both products and the impact DDD had on the technology & the business.

Some key questions we'll explore:

How to use DDD when you're launching a new product into a new market and there are no domain experts to consult?
Why it's not a good idea to use the same object/table to store both the size of a school uniform t-shirt and the presence of a slice of cheese on a sandwich
How to make the business case for a major multi-service refactor when your DDD-inspired domain understanding evolves mid-project

Videos

NDC Oslo 2022

Conference Schedule | Slides

This one is good for newcomers to DDD as it doesn't assume any pre-existing DDD knowledge.

https://www.youtube.com/watch?v=QqvYIIobZ6A

KanDDDinsky 2022

Conference Schedule | Slides

Being a Domain-Driven Design focussed conference, this one is a bit shorter as I skipped/glossed over some of the more introductory material.

https://www.youtube.com/watch?v=ApDjFCnvsj8

Engineering & Product: Join Forces!

2022-07-28T00:00:00Z

Sometimes I meet software engineers who say "Just tell me what to build and let me work out how to build it."

While I think I understand the motivation behind the statement it always makes me a little sad.

In this dynamic, the requirements are rarely informed by what's technically possible, and the solution is rarely informed by the organisation's real purpose.

My assumption is this is a form of learned helplessness- the outcome of having worked in environments that punish creativity and innovation, are strict about role boundaries and filled with command & control style authoritarian leadership.

So I have empathy for these engineers - you can't help where you've been.

But the mind boggles at the wasted potential!

The potential of inviting genuine collaboration; the potential for more innovative solutions to real problems, for aligning everyone with their work's purpose.

So my advice to my clients and mentees is: join forces! Invite your engineers into customer conversations. Immerse them in the domain, let them contribute to the product vision and don't punish them when an idea doesn't work out. Foster psychological safety by focussing on the lessons learnt and incorporate them into the next iteration.

Product & Engineering: stronger, more creative, more innovative, together!

Have you worked in environments that rewarded or prohibited such collaboration?

(Cross-posted to twitter and LinkedIn)

The Software Engineer as a Novelist

2022-07-21T00:00:00Z

In conversations with non-technical founders and leaders, I often come across the lament that the engineering department is an opaque box - requirements go in, working (sometimes not working) software comes out (but sometimes doesn't).

When imagining what's inside this box, many folk envision some kind of production assembly line - a conveyor belt transporting user stories, developers mechanically adding code as the story trundles past them.

With such a metaphor in mind it's understandable how confusing the unpredictability of engineering's output can be. This confusion is frequently quite frustrating - especially given how much of an organisation's scarce capital is involved.

Happily, I'm here to say that this metaphor could not be further from the truth.

Rather than an assembly line, I think things make more sense if you imagine that engineering teams are being asked to write a novel.

Immediately, you get a sense of the unknowns. Creativity, design, empathy are involved, and there's also plot coherency, narrative arc & language (architecture & software design) to consider - not to mention of course the dreaded "Writer's Block" (analysis paralysis).

To get this novel written faster, it's not uncommon to assign it to a team - or multiple teams.

Sometimes each team is assigned a chapter, and since they are working concurrently, the last chapter is written at the same time as the first.

Sometimes the front-end team is assigned the first sentence in each paragraph, the backend team is assigned the middle of the paragraph and the infrastructure team is assigned the last.

And sometimes the team assignment is as unwieldy as assigning each word in each sentence to different teams!

What to do about it?

There are a number of helpful approaches, but here are 3 ideas I've found incredibly useful:

The first is acceptance - that software is an inherently complex, creative, and unpredictable endeavour. Much like writing a novel, you don't always know how it's going to finish when you start, and it can take several drafts (iterations & refactors) before you're happy with the end result.

The second is to orient your engineers not around chapters, paragraphs, sentences and words (i.e. the technical elements), but around characters, themes and plots - in design parlance, your business domain: the concepts, stakeholders, processes and points of value.

The third is to exploit the ways in which software is NOT like writing a novel: it's hard to sell the first draft of a novel, but the first draft of a software system (an MVP) can still be valuable to a subset of your customers. Over time, you can expand the pool of prospective customers by incrementally enhancing your software.

The keys to this approach are:

effective customer-value-focused prioritisation to ensure you're delivering real value early, and
automated quality checks to ensure that as you expand the capabilities and support a broader customer base you can move quickly without breaking anything your early adopters are counting on

So next time you're asking a developer how long it will take them to complete a task, imagine you're asking a novelist how long it will take to finish a book - perhaps the conversation will take you in a different direction!

(Cross-posted to twitter and LinkedIn)

UNSW 2022 Co-Op Scholarship Induction Ceremony

2022-03-18T00:00:00Z

I was recently flattered to be invited to speak at the UNSW 2022 Co-Op Scholarship Induction Ceremony as an Alumni. What follows is the text of the written speech - as I don't refer to notes when speaking, what came out was slightly different but this hits the key themes.

Thank you to Kay and the Co-op program for the invitation to speak to you here this evening, and to Michelle for your stellar organisation and co-ordination during such unpredictable times!

Firstly - to the 2022 co-op scholars, congratulations! You have no doubt done great things to be included in the cohort, and I'm sure you will continue to do great things into the future. You are embarking on what will very likely be a life changing experience, and I speak as someone whose life was definitely changed by being part of this program.

It's a while ago now, but I vaguely recall participating in a ceremony much like this one back in 1997 - having just met and started to get to know my fellow co-ops at that point I had no idea that this group would become so important to me - since then I have lived with some of them, worked with many, started a business with one, and been in the wedding parties of two, just to mention a few key events.

The brief for this evening was to share the story of my career and any pieces of wisdom that I have picked up along the way that might be useful to new scholars. I feel uncomfortable describing anything I have to say as particularly 'wise' - I have done much, and learnt much, but there is so much more to do and learn and everyone needs to find their own path to peace, success & satisfaction in life. But I will intersperse the story of my career with some things I have come to consider to be true and let you decide if they are wise or not!

The first, speaking of the lifelong friendships formed in the co-op program, is that I have come to consider that everything is relational. This is a complex word, and it means many things. I mean it in the human sense - that the relationships with your peers in the program, with your sponsors, with your teachers and staff at the university will all be foundational to your lives. I also mean it in the technical sense that as engineers, technologists and business people you will be designing, building and engaging with systems & processes that exist in a context - the value of those systems will lie in how they relate to the broader world, and in the internal relationships between the components they are comprised of. Finally I also mean it in the socio-technical sense - technology is shaped by society, but it also shapes it - sometimes for the better as in how it helped us adapt and stay connected through a global pandemic, not to mention the rapid production of the vaccine. But sometimes in less positive ways, from the impact on mental health and widespread disinformation caused by the rise of social media through to the devastating environmental impact of technically exciting innovations like cryptocurrencies and blockchain which are exhibiting burgeoning energy consumption. Everything is connected, everything is relational.

But back to my career - I studied Electrical Engineering and enjoyed a nice mix of industry placements. The first was over the summer after first year with a firm called JTec which was building pioneering high speed internet (for the late-90s) - I wrote some programs that helped speed up some of their circuit board layout design & manufacturing processes, which I really enjoyed as I was working beside the users of the programs and I got to see the positive impact it had on their lives immediately.

After 3rd year, my first 6 months were at a company that is now known as Dematic. This was by far the most impactful for me - I worked part time for them through 5th year, and then seamlessly transitioned into full time work after graduation. The technical work was interesting - we were designing automated control systems for the logistics industry - programs that would, amongst other things, sense the movement of packages around conveyor belts and instruct actuators to send particular packages down particular conveyor belts into the backs of waiting trucks. All of this was cool, and I loved contributing to a technology that was so useful to society, but if I'm honest, it was the people and culture that I really loved, and which made me confident that I would enjoy my time there.

In between Dematic and 5th year, I also spent 6 months at Alcatel - another telecommunications firm, and once again I found myself working on systems that were helping to roll out "high speed" internet. At the time, Alcatel had the primary contract with Telstra and the components we were working on were being deployed into the green telecommunications boxes that you see around the city providing some of the very first ADSL connectivity - how exciting it was to feel I was contributing to a service that was so valued by the community.

Following graduation, I ended up back at Dematic full time, and after less than a year, they asked me to relocate to the US - Grand Rapids, Michigan, to join a global Research & Development team helping to design the next generation of automated conveyor systems. I had always loved travelling and couldn't say no. I landed in the coldest December I had ever experienced and had an absolute blast. For a time I'd spend my days working on interesting technical challenges, and my evenings night skiing at a small hill 1/2 hrs drive from the office. A bizarre juxtaposition for an Australian!

Professionally, it was also a fascinating experience - while the Sydney office had felt like a small family business, the US office felt very much like a division of a large corporation. I made some more amazing friends there, some of whom I'm still in touch with to this day and we did some great work together on this new product, but it helped me understand something about myself, which is that I'm very much a 'small business' type of person.

I greatly respect the things a large corporation can achieve, and the people who can work together across multiple distributed teams to make it happen - but it had become clear that that wasn't for me.

After two years, the Dematic office in Sydney asked me to return to be one of the first engineers in a new department called Systems Engineering. This was another amazing opportunity, and I learnt some important skills in that role, including an understanding of some of the engineering & project management practices used to build incredibly large systems, with budgets of hundreds of millions of dollars.

One of the great things about returning was reconnecting with Steve Austen - a fellow Co-op scholar who had also joined Dematic after graduation and who was still in the sydney office. We'd head out for lunch every day and gradually our conversations turned to aspirations of doing something novel - starting a business together. Steve's older brother Geoff had had a successful career in Finance and thankfully decided that funding us was something he was keen to do, and before we knew it we were in business together. It was difficult saying goodbye to Dematic, but they were very understanding and I'll always be grateful to the incredible colleagues & mentors I met there.

Somewhat predictably, given that our startup conversations had been over lunch, and the Dematic sydney office was in French's Forest, a place with no particularly good lunch options, our initial startup idea was to setup an online lunch ordering service. The concept was that you could punch in a lunch order online, it would be sent to a local cafe or restaurant, and by setting up relationships between the cafes and companies, the cafe would get to deliver a larger volume of meals in one go. Geoff, having worked in Finance was aware of a tax benefit scheme that meant you could pay for meals provided at work from your pre-tax pay, and bam, we had our business, launched in 2005 - only 4 years after graduation. It was called FlexiMeals and within 2-3 years it was well on its way to being highly successful - we had about 12 staff, and almost 20 client companies, some of which were quite large, including Macquarie Bank & IAG insurance using our services in their CBD offices.

Being such a small business, I really felt in my element. One minute I'd be writing some code, the next helping answer a customer service call, and in the afternoon making a sales call to a prospective client. I loved the diversity and the excitement of getting to see something that had started as an idle thought become tangible and generating a strong buzz and positive response from our customers.

Unfortunately, in 2008 the government changed the tax legislation and the pre-tax benefit disappeared. Many of our clients who had engaged us to offer a benefit to their staff felt that the value had gone, and dropped us overnight. It was a difficult time, but thankfully we had recently had some feedback from a customer that they would love to use our service to order their children's lunches at school, and we had just started our first pilot school when the tax laws changed. We decided that the time was right to pivot hard towards that opportunity, and a new business, Flexischools, was born.

Flexischools took off like a rocket - by the end of that year we had 10 schools, the end of the next year about 40-50 and a few years later several hundred. Today, Flexischools services almost 2,000 schools around Australia, hundreds of thousands of families at a time, processes almost 200,000 lunch orders a day - most of which are placed on our systems at almost the same time - around 7:45am - a fascinating engineering
challenge. And finally, with my daughter having just turned 6, I'm actually a Flexischools customer myself, and placed a sausage sizzle order for earlier today.

This is a good time to mention another thing I have come to consider to be true: perseverance and resilience are some of the most valuable qualities you can develop. When the tax laws changed, or when any number of other challenges threatened to derail our goals, we never once considered stopping. Our attitude was always 'well, this way isn't working, what other things can we try?'. The assumption that we would one day be successful was never questioned, and although it did take us quite a while in the end, I feel very proud of where this business has arrived at today.

I have to say, while that moment in 2008 was a crisis at the time, it was really a blessing in disguise. While FlexiMeals had been interesting, I had always somewhat questioned the value provided by helping high income financiers and insurers in saving a few dollars in taxes off their lunches each day. After launching Flexischools, one day we got a call from a school telling us that thanks to Flexischools, their canteen had switched in one year from needing parent fund-raising activities to stay open, to being able to donate money back to the school, which was spent on a new playground for the kids. Needless to say we were thrilled, and I once again felt the purpose in our work.

So - another thing I have come to consider to be true: there are at least three properties that need to be satisfied to be fulfilled in your career - you need to find the intersection of the things where 1) you enjoy the doing of the work itself - it's essential that the actual day to day is a pleasure and not a chore; 2) you need to find the things that you are actually good at and can do well and 3) you need to find the things that fill you with a sense of purpose. Work that satisfies all three is truly the most rewarding, and the work at which I believe you will be most successful.

The next major milestone in my career was in 2015. Flexischools was thriving, and a new opportunity landed on our laps. Although Flexischools was oriented towards school lunches, what we had built, in effect, was a general capability to handle complex multi-party transactions in high duty of care environments - unlike a traditional purchase between a buyer and seller, Flexischools had to handle purchases between a buyer (the student), the seller (the canteen or uniform shop), the funder (the parents) all through the lens of the rules and constraints setup by the school.

An analogue presented itself in the form of the National Disability Insurance Scheme - a new government initiative for helping people with a disability have more equitable access to life's opportunities. The vision was to invert the traditional funding model, where people with a disability were told what services they could access and when, without regard for whether the services were actually assisting them, into a model founded on the principles of choice and control - where participants in the scheme would be given access to a budget, and given sliding scales of freedom and autonomy to spend on the services and providers that would work for them - with the budgets oriented around individualised life goals.

It was clear that this was something we wanted to help with - although we were new to the field of disability and health support, we felt we had the technical skills to assist, and the desire to make a positive difference. We launched a new product called LanternPay - by this time we had a team looking after the Flexischools technology, and so I threw myself back into the space I love the most - early startup land, creating something real from nothing more than an idea. We built LanternPay from scratch, grew the team, and over the last 6 years it has helped 10s of thousands of people with a disability access a wider range of service providers as well as streamline payments and reduce the overheads of operating in the scheme. On top of this, we've expanded into the broader healthcare insurance landscape, helping tens of thousands of people who have been victims of traffic accidents or workplace injuries receive care when funded by state government support schemes and more recently expanding into private health insurance for some of the types of care which don't currently have electronic payments.

This was a very different journey to Flexischools - we were well funded from the beginning and playing in a different league - engaging with government and enterprise, and needing to really put on our grown up pants. But, it has been an immensely rewarding journey, both in terms of the impact we have had, and the fact that last year LanternPay was acquired by National Australia Bank to join forces with their existing healthcare claiming product, HICAPS.

The timing was a co-incidence, but the acquisition has certainly helped with the most recent career change for me - last year I decided that after 16 years of being on the coalface, a hands on CTO (I was writing code until the very end, as well as leading and managing a growing team of engineers) it was time for something different. We had luckily found a very skilled person to take on the CTO role, which allowed me to transition into being a non-executive board director which means I get to stay closely connected to the company and the people I love, but freed up my time to launch something new.

That something new is an independent boutique technology advisory service. I support a range of startups around Australia with technology strategy and advice to help them launch their great idea, mentoring and coaching their engineering teams and just generally trying to be helpful. Frankly, I'm loving it - this last year has been the happiest time in my career. I work part time - usually 3 or 4 days a week, so I get to spend much more time with my children and my hobbies (including contributing to open source projects, and some home automation engineering), and the rest of my time meeting and working with a huge range of energetic, passionate and creative people working on the next generation of great startups, teaching them, yes, but also learning from them, in a symbiotic relationship I value enormously.

And this brings us to the present day, so I'll close with a few more things I have come to consider to be true:

Learning beyond your degree is immensely valuable for many reasons. Reading literature, engaging with the arts, seeing live theatre and music, studying history, learning about the history of your field, but also the history of humanity - they will all furnish you intrinsic delight in their own right

They will help you become a student of humanity: human nature being what it is, patterns repeat and observing and identifying those patterns as they recur is incredibly powerful.

And they will help with another skill I have come to value highly - the skill of empathy. In the modern era where organisations are increasingly aware of the importance of psychological safety - that fortunate condition where those working side by side are not scared to speak candidly about issues, are empowered to try new things and experiment with solutions - in this world, empathy is becoming a more prized skill than any technical capability. Having empathy for your colleagues, for the people who came before you and the people who come after - having empathy for your users, for your customers, for those affected by hidden externalities in the systems you build and the systems you engage with - all of these will make you a more fully present and intentional contributor to your workplace and society.

Once again congratulations to all the new scholars - I wish you all the best with your degrees, your industry placements, and your careers wherever they take you - but most of all, I wish you the good fortune I have had, that you can say in 20-odd years that you are a lifelong learner - still learning, still discovering and still filled with excitement and anticipation for what you will get to learn next.

3 Startup Software Superpowers - Part 1 - Introduction

2022-02-25T00:00:00Z

One of the awesome things about working in a startup is that you get to take on an enormously diverse range of tasks. But how do you ensure you're delivering high quality work across all of them?

I love experiencing the full lifecycle of a product. When it's just you and a small team taking an idea from initial concept through design, planning, implementation, sales and operations you get to wear a lot of hats (or capes?) - something that can be immensely rewarding.

Working in a startup, you get to wear a lot of hats!

Over the years I've learnt three conceptual tools that have helped me be effective at different stages of the journey. I'll go further: they are like having superpowers. Once you learn and apply them, you'll feel like a superhero too!

In this series of posts we'll take a look at these superpowers and demonstrate how to make the most of them in a startup context.

Part 1 (this post) will briefly introduce the ideas, explain why they feel like having superpowers, and share links to other online resources with more details. Parts 2, 3 & 4 will take a deeper dive into each as we go on the journey of our fictional startup.

They are:

Our 'Super' Startup

If you're starting work on software for a startup, I'm assuming you and your partners already have a business idea.

To illustrate these superpowers, let's get to work for our fictional startup - and since we're talking about superpowers, and inspired by the recent flooding in northern NSW, lets build a product for some real-world super-heroes - the volunteers getting involved to help out those affected.

Our startup is codename E.A.S. - Emergency Alert System.

E.A.S. - Emergency Alert System

Our mission is help co-ordinate response efforts when disaster strikes:

Members of the public want to request help, and get alerts about nearby incidents so they can steer clear.
Response volunteers want to find out about people who need help so that they can attend and assist.

Future expansion might include an integration with Waze to help route traffic around incidents and an integration channels to support formal government emergency services.

With the scene set, time to launch our startup!

The Concepting & Design Superpower

As the technical co-founder of E.A.S., the first question you're probably asking yourself is: what do you build to make that idea a reality?

The concepting and design superpower is known as Event Storming and when you're first working out what's needed, this superpower will save the day.

Event Storming was invented by Alberto Brandolini as a way to foster collaborative domain modelling in organisations with isolation between IT and other parts of the business, such as customer support, sales and management. Over the years it has expanded, and is now frequently used in a range of scenarios, including envisioning a startup ecosystem.

Event Storming In Action

Event Storming is a startup superpower because of how well it helps you and your stakeholders collaboratively explore the uncharted landscape. It rapidly establishes a shared vision of what your system will do ensuring you're all in sync and can move quickly.

Event storming follows a fairly simple process:

Identify all the 'events' that occur in your system. E.g. for E.A.S. we might have:
1. Help Requested
2. Help Arrived
Get agreement on the time sequencing of events
Work out what causes the events - user interactions? Integrations? Time-based workflow?
Re-organise around key objects/entities (known as aggregates)

The result is an your starting model - one that is closely aligned with the language and domain of your stakeholders.

In Part 2 of this series, we'll follow this process for E.A.S., but if you want to check out some real life event storming now, there are great examples all over the web, like this article and these videos.

The really hard part is that in a startup, you don't know what's coming next - startups are all about exploring and learning, so you have to be prepared to evolve and iterate your ideas as you go - and that's where the other superpowers come in.

The Planning Superpower

Once you have an vision of what to build, you need a plan for building it. Unsurprisingly this is where many projects go off the rails.

The two most common mistakes are:

Doing a huge amount of up front planning trying to schedule the whole project.
Just getting started without planning ahead at all.

Put that way, it's obvious that success lies somewhere in the middle. This is where the planning superpower, known as User Story Mapping shines.

It's a startup superpower that helps you make the most of your initial investment by launching quickly, learning & iterating so you can rapidly find product market fit.

The genius of User Story Mapping is that guides you to plan just enough to implement a true MVP (Minimum Viable Product) --- that is, one that is both minimum AND viable. (Many folks miss one part or the other!)

For example, with E.A.S. you may be so focussed on minimum that you spend too much time on the general public requesting help, but not enough on volunteers responding. Your MVP will fall flat, as it doesn't deliver value - it's not viable.

Or, you might be so intent on viable that you lose focus on who the MVP is for. You spend a lot of time on features (e.g. a Waze integration) that your first customers don't care about, wasting time before you can launch - it's not minimum.

User Story Mapping helps you find your MVP

In Part 3 of this series we'll produce a user story map for E.A.S., but if you'd like to read more now, the first article by its creator Jeff Patton is a very clear explanation.

After user story mapping your concept, you have a realistic plan for incrementally building, with plenty of opportunities to incorporate lessons learnt.

Now how do you build the software so that you can rapidly respond to all the twists and turns that lie ahead?

The Software Development Superpower

The software development superpower is one that many have heard of, but less frequently used: Test Driven Development (TDD).

If you're familiar with TDD, you may be surprised I recommend it for startups. There is a perception that TDD is only useful if you have stable requirements, or only doable if you have the luxury of time and capital to spend writing tests.

Neither sound like the dynamic, exploratory, fast-paced world of startups.

And in my experience, neither could be further from the truth.

If you're not familiar with TDD, it's the practice of building software in very small cycles, each cycle consisting of three steps:

Write a failing test
Make the test pass in the simplest way you can
Refactor to improve your code & design while keeping all tests passing

This is often visualised as Red -> Green -> Refactor - a familiar loop I included in my logo because I find it so valuable!

The inner cycle of the site logo represents TDD

It's a simple idea, but hard to grasp the implications of until you've given it a try. Dave Farley has a great video demonstrating the technique for beginners. Kent Beck originally developed the idea and wrote the seminal book on the topic. GeePaw Hill is also well worth following for excellent advice and content.

There are many positive benefits of TDD, but in a startup context the most important one is that it allows you to continuously refactor.

The early days of a startup are when your design is most likely to need to change/evolve rapidly. By ensuring you diligently refactor during each TDD cycle, you will find your code absorbs these changes more easily - and you will have high confidence that you haven't broken anything that your early customers are relying on.

This is important because your early customers are the ones you need to impress the most - both by delivering a reliable service, and by demonstrating that you are listening closely to their feedback.

If Part 4 of this series, I'll illustrate some key refactoring efforts the E.A.S. developers need to make and the impact of TDD.

Summary

If you want to be a startup superhero try out these tools:

Event storming to explore an initial concept and design
User Story Mapping to find the MVP
Test Driven Development to keep your implementation moving quickly

Even if you're not in a startup, these tools are well worth learning. They are superpowers in many situations; this blog post just focusses on their application for startups.

What are your Superpowers?

This is not an exhaustive list of software superpowers by any means!

What are yours? Share them in the comments below, on my Twitter or LinkedIn

Getting your Superpowers

If you'd like to experience what it feels like to have these superpowers, send me a message - I offer workshops & training in all the above topics and would love to share my experience with you.

What I Think About When I Think About Coupling

2020-10-28T00:00:00Z

The phrase “loosely coupled” is used alongside “microservices“ so often you'd be forgiven for assuming you get loose coupling for free, just by “doing“ microservices. (Spoiler: you don't.) But what actually is coupling? How loose is loose? Should we also be worried about cohesion? And why do we care?

Introduction

Talking about Coupling & Cohesion

Before we begin it needs acknowledging - talking about coupling and cohesion is hard! Informal conversations often rely on intuition and analogy, and while academic research has given us reams of classification schemes and complex formulas, they are infrequently applied in commercial development.

In my experience, part of the problem is that over the years the industry's understanding of coupling and cohesion has expanded dramatically, leading to a number of different ways of thinking about the concepts. While all of these ways are linked by a common thread, that commonality is not always apparent, and the reason why coupling is important can become disconnected from its definition. I've had conversations with other engineers where it felt like we were speaking completely different languages when trying to discuss the merits or risks of a particular design with respect to coupling.

Talking about Coupling & Cohesion can feel like speaking different languages

When talking about loose coupling, we have the added challenge of relativity - loose is often not used as an absolute. Frequently it is used to talk implicitly about looser coupling, compared with what came before. A system might be described as loosely coupled - and it may well be, next to a previous iteration of the same system - yet it may still exhibit many characteristics of tight coupling.

v2 may feel like loose coupling compared with v1

The aim of this blog post is to summarise a number of the most common ways of thinking about coupling and cohesion, and hopefully draw out the common thread that links them all.

Why do we care?

The concepts of coupling & cohesion are not new. They were developed by Larry Constantine in the late 1960s and 70s as part of a broader idea called "Structured Design".

In 1979, Constantine and Edward Yourdon published Structured Design: Fundamentals of a Discipline of Computer Program and System Design.

Structured Design

The book contained explorations of these relatively new ideas, with the inspiration noted as:

For a given problem, the human error production and, therefore, the cost of coding, debugging, maintenance, and modification are minimized when the problem is subdivided into the smallest functional units that can be treated independently

With a focus on the cost of a software project, Structured Design had the economics of software development firmly in mind.

So why do we care? The first reason, historically speaking, is that the concepts were invented specifically to help design software that is less costly to implement and maintain.

But how did they propose to achieve this goal? They started by investigating what seemed an unnecessary portion of the cost - dealing with errors (aka bugs, faults, defects). In particular, they honed in on the root cause of those errors - the humans who introduced them in the first place, and their capacity to deal with software's growing complexity.

We care about loose coupling and high cohesion, because it helps us design software that is less costly to build and maintain.

As we work through this post, we are going to extend this map to illustrate how the different ways of thinking about coupling and cohesion relate to these goals.

Ways of Thinking about Coupling & Cohesion

The Basics

Before taking that thought further, it will help to establish some baseline understanding about the general structures involved.

Any system can be thought of as a collection of interconnected components (Constantine's "functional units"). In Structured Design, he referred to them as modules, but the principles apply to any type of component. In Object Oriented Programming, you might be talking about a class, or at a higher level you might consider a package, or assembly. In Microservices architectures, you're typically talking about an independently deployed and operating microservice.

With that in mind, we can start with some basics:

Coupling is about the relationship between two components.
Cohesion is about the internals of a component.

Since coupling is about relationships, it's important to note that it is directional. Components A & B are coupled if there is any kind of relationship between them but the impact of B on A might be far more significant than the impact of A on B.

Coupling is about the relationship between components, cohesion about the internals of a component.

Coupling

We're going to talk about coupling first, and for the purposes of this post, I've loosely grouped the ways of thinking about it into three conceptual models:

Software is written by people

In pursuit of more cost effective software development practices, Constantine noted that most of the cost of system development is in fixing bugs - in other words, correcting errors made by humans. To reduce the cost, he went looking for ways to reduce the bug-rate.

It's been observed that humans have an information processing limit of about 7±2 objects at any point in time. Constantine concluded from this that most sources of human-introduced errors are a result of attempting to code modules that are naturally too complex for humans to understand.

In Constantine and Yourdon's own words:

The key question is: How much of one module must be known in order to understand another module? The more that we must know of module B in order to understand module A, the more closely connected A is to B. The fact that we must know something about another module is a priori evidence of some degree of interconnection even if the form of the interconnection is not known.

...

The measure that we are seeking is known as coupling; it is a measure of the strength of interconnection. ... Obviously, what we are striving for is loosely coupled systems --- that is, systems in which one can study (or debug, or maintain) any one module without having to know very much about any other modules in the system.

Coupling as an abstract concept --- the degree of interdependence between modules --- may be operationalized as the probability that in coding, debugging, or modifying one module, a programmer will have to take into account something about another module. [1]

This definition clearly incorporates the limitations of human thinking - if a developer needs to keep facts about component B in their head while working on component A, then we increase the likelihood that we will exceed the 7±2 object limit - bugs will inevitably follow!

Coupling between ClassA and ClassB increases the risk of exceeding our human information processing limits.

Following this to its logical conclusion, we can reduce coupling by arranging our system design such that in order to work on a given component, we do not need to know much about other components. This has a strong link to the idea of designing for high cohesion - further details below.

An interesting aspect of this way of thinking relates to modern DevOps style of working in that errors while developing are one thing but errors at run-time become operational incidents. For the same reason that looser coupling makes things easier when designing and bugfixing, it also makes operational incidents easier to resolve as the response team will have less parts of the system to think about and can focus on the component with the fault.

To visualise this way of thinking about coupling we can map out the chain of reasoning:

Loose coupling helps us work within our limits, reducing the risk of errors, and the overall cost to build, maintain & operate software.

Interconnections Between Components

Constantine & Yourdan were quoted above: "The fact that we must know something about another module is a priori evidence of some degree of interconnection even if the form of the interconnection is not known."

In the search for ways of objectively measuring this degree of interconnection, researchers have necessarily needed to focus on scenarios where the form of the interconnection is known.

For example, an interconnection might be:

A call to a method in another class
A dependency on another package or assembly
A remote API call
A message exchange via a queue

The outcome of this research is probably given the most attention in the formal software engineering education and online documentation. For example, Wikipedia's article on Coupling and other online tutorials contain a laundry list of coupling types under various classification schemes as well as formula for computing a coupling metric based on these specific methods of interconnection.

These formula and classification schemes are thoroughly and expertly discussed elsewhere so I won't repeat them here, but I would like to discuss some of the extra benefits of loose coupling that have been identified through the research that produced them. These further benefits all contribute to reducing the cost to build, maintain and operate software systems.

By reducing the degree of interconnection between components (i.e. looser coupling), we achieve the following benefits:

Reduce error impact

When there is an interconnection between components, a fault in one component can cause failures in other components. For example, if ClassA calls a method on ClassB then does something with the return value, a bug in ClassB can manifest as an incorrect result in ClassA. Hopefully you have unit testing to catch and isolate these sorts of errors, but reducing coupling will reduce their likelihood altogether.

Improve runtime performance

At runtime, inter-process interconnections (such as remote API calls between microservices) carry a performance overhead, and introduce temporal coupling - that is, the execution time of the caller is affected by the execution time of the callee. This latter point is part of why asynchronous message queues are often recommended for inter-process communications.

Reduce incident impact

An incident in a component can spread to other components. For example, when a component is experiencing an incident, it may become unavailable to components that connect to it, or present degraded service quality. In the reverse direction, a component experiencing an issue can overload components that it calls out to. Reducing coupling can help isolate the incident and reduce its spread, improving availability and reliability.

To visualise this way of thinking about coupling, once again we can map out the chain of reasoning:

Reducing the interconnection between modules reduces error impact, improves runtime performance and reduces incident impact.

Change Blast Radius

The observation that loose coupling reduces the impact of errors was first published in a book based on Constantine's work called The Practical Guide To Structured Systems Design by Meilir Page-Jones.

This way of thinking about the impact of the relationship between components has come to be known as the "blast radius" of the component. It's usually defined in terms of the number of other components affected by an event in the component. The looser (or fewer) the connections a component has with other components, then the blast radius of the event is smaller. (Note: Page-Jones did not use the term "blast radius" - this is a more modern term for the same concept.)

Errors & incidents are types of events that can have a blast radius. Another type of event is a change.

It is a rare software system that is built once and never modified. Software systems, especially in the modern product-led Saas world, are living things, undergoing continuous evolution. Inspiration for changes can come from anywhere - a startup pivot, a strategic review, customer feedback, operational efficiency goals, just to scratch the surface!

When embarking on a change, one has to start by considering the component(s) most obviously responsible for the functionality that needs to change - and then work out if any of the changes impact other components. Components that are tightly coupled are more likely to require changes to accommodate the initial change; conversely, components that are loosely coupled are less likely.

Change blast radius refers to the scale of impact of a change - in this example, components A and D are in the blast radius of component E.

In the image above, when component E is updated to v2, components A and D are broken - they are in the blast radius of this change. To complete the change successfully, they must also be updated - adding cost and time to the change, especially if those components are owned by other teams.

The change is only complete when components A and D are updated.

The analysis of change impact is so significant that Page-Jones continued his research in this area and ended up coining a new term just to describe it: Connascence. Connascence has also been analysed in formal research and https://connascence.io has a good summary of the classification schemes that have been produced.

This way of thinking about coupling has become one of the most popular informal understandings I've seen (although the term connascence appears less well known). In fact for many engineers, this has become the default, and most intuitive way of thinking about coupling. For example, this tweet from Kent Beck defines coupling exactly in this way:

https://twitter.com/KentBeck/status/1116379072781619200

I do think that this focus on the impact of change reveals one of the most challenging aspects of striving for loose coupling - in many scenarios, predicting the future types of changes is almost impossible, particularly when in response to a startup pivot, or a strategic review. We'll consider these types of changes more in a future post about the Logical View.

A formal concept associated with this idea of change is the Instability Index - a metric derived by analysing the number of "inbound" vs "outbound" dependencies, with the goal of predicting the "instability" of the component - or how likely it is going to need to change when other related components change.

Similarly to errors, there is a run-time impact of connascence, particularly in the world of microservices. One of the goals of microservices is to have independently deployable units, owned by autonomous teams. If two microservices have a high degree of connascence, then it's more likely that a change in one microservice will necessitate a change in another - reducing the likelihood that the change in the first service can be safely deployed without coordinating the deployment of the consequent change in the other service, and thus reducing team autonomy.

Once again, we can visualise this way of thinking by mapping out the chain of reasoning:

Reducing the change blast radius reduces the cost and complexity of changes, and enhances team autonomy.

Cohesion

Cohesion naturally follows as a sort of inverse of coupling:

'Intramodular functional relatedness' is a clumsy term. What we are considering is the cohesion of each module in isolation --- how tightly bound or related its internal elements are to one another.

...

Clearly, cohesion and coupling are interrelated. The greater the cohesion of individual modules in the system, the lower the coupling between modules will be. ...

Both coupling and cohesion are powerful tools in the design of modular structures, but of the two, cohesion emerges from extensive practice as more important. [1]

In other words, if all the functionality in a given component is related to the same idea then while the developer is working on that component there is less risk of other unrelated stuff taking up space in the 7±2 object limit and crowding out the relevant material.

When class A contains functionality related to both foo and bar, we run the risk of exceeding our human limitations.

It may come as a surprise that Constantine and Yourdan considered cohesion as more important than coupling, considering the industry focus on coupling, and the many ways it has come to be understood. I believe it makes some sense, though, given that they were focussed on the introduction of errors by humans. The other benefits of loose coupling had not yet been articulated.

While the word cohesive is less frequently used to describe microservices, this idea is at the heart of the advice to organise your services around "business capabilities" -- the intention is to ensure the functionality within each service is all related -- i.e. that the service is cohesive.

We can visualise the impact of cohesion similarly to coupling:

Highly cohesive modules can reduce the risk of errors, and consequently the cost to build and maintain software.

Conclusion

To bring together all our maps of the different ways of thinking about coupling and cohesion:

The complete coupling/cohesion landscape.

Given that most conversations focus on a narrow band of this landscape, it starts to make sense why these are such difficult topics to get your head around, particularly for people new to the field!

In summary, there are many different ways to think about coupling & in this post we've explored three:

"Software is written by humans" - how much can we keep in our heads at one time
"Interconnection between components" - the actual interconnections between components and their characteristics
"Change blast radius" - changes, and the way they necessitate further changes in related components

In my experience, these different ways of thinking are at the root cause of many unfruitful conversations about coupling -- but while they appear different, hopefully it now makes sense that they are in fact all related. The common thread is the strength of relationships between components and the benefits of reducing the strength of those relationships. At different times and in different scenarios, you may find one way of thinking or the other more useful, and I encourage you to keep in mind the broader perspective so you can make an informed choice about when it makes sense to adopt a particular approach.

Stay tuned for future posts (coming soon!), in which we will apply these different ways of thinking to concrete code and system designs -- making good on the spoiler above to explain in detail why you don't get loose coupling for free just by "doing" microservices.

When returning to the series of posts about the 4+1 views, we will also analyse coupling & cohesion through the lens of each view, examining how different types of coupling are more relevant in different views, and how increasing cohesion in one view can actually decrease it in another.

History of the 4 + 1 Architectural View Model

2019-02-28T00:00:00Z

This post explores the history and legacy of the 4 + 1 views. See 4 + 1 Architectural View Model: Introduction for the more technical overview.

Origin Story

In 1995, Philippe Kruchten was working at Rational Software Corp, at the time the pre-eminent vendor of software development tools.

With years of experience designing large scale complex software systems across telecommunications, aerospace, transport and defense, he had observed that software architecture diagrams often failed to provide clarity as to the actual system design. It was frequently confusing as to what the boxes, lines and arrows really represent, and stakeholders struggled to discover the information they needed.

Boxes and lines and arrows... but what do they mean?

He proposed a solution in a paper published that year - Architectural Blueprints—The “4+1” View Model of Software Architecture - to organise the description of a software architecture using a set of concurrent 'views', each addressing specific concerns, for specific stakeholders.

Historical Context

Comparisons between software architecture and the architecture of buildings have been made since at least 1968, when the Nato Science Committee sponsored a Working Conference on Software Engineering. Amongst many other topics the attendees discussed how to discover the "right attitude to design". Peter Naur (the "N" in BNF) suggested:

software designers are in a similar position to architects and civil engineers, particularly those concerned with the design of large heterogeneous constructions, such as towns and industrial plants. It therefore seems natural that we should turn to these subjects for ideas about how to attack the design problem.

The following decades contained numerous efforts to grapple with the large scale design of software systems with many foundational principles proposed and refined, but applied sporadically.

It was only in the early 1990s that the term "Software Architecture" came to the fore. In 1992 Dewayne Perry and Alexander Wolf published "Foundations for the study of Software Architecture":

The 1990s, we believe, will be the decade of software architecture. We use the term "architecture", in contrast to "design", to evoke notions of codifcation, of abstraction, of standards, of formal training (of soft-ware architects), and of style.

So it was in the context of a discipline with decades of development, but only nascent formalisation, that Kruchten proposed his 4 + 1 views.

It's no surprise given the historical analogues with building architecture that he would refer to the views as blueprints.

Legacy

In 1996, Kruchten was appointed the Director of Process at Rational Software where he led the development of the "Rational Unified Process" (RUP). The 4 + 1 views are a key concept in the RUP section on Analysis and Design.

While Kruchten was working at Rational, his colleagues were tackling the challenge of communicating about software on a different tack - the outcome was the Unified Modelling Language (UML).

The 4 + 1 views are a generic approach that does not prescribe any specific notation or diagramming structure (although Booch notation is used in the paper); UML fills in the blanks with prescriptive definitions of how to notate certain types of diagrams. Although the 4 + 1 views are not specifically referenced in UML, the intention of the RUP was that it would work in concert with UML as the notation for diagrams. It seems no coincidence and there is a neat alignment with the 4 + 1 views, with many UML diagrams being well suited for use in a specific view.

These days the 4 + 1 views are rarely utilised - technology and tech culture have moved on.

Where did the 4 + 1 views go?

Perhaps the fledgling rumblings of Service-Oriented Architecture (SOA) in 1998 heralded the end? In the context of Kruchten's paper, systems were large complex beasts with often custom implementations of both hardware and software. The rise of SOA aligned with the web as the communications medium of choice, the adoption of commodity hardware, and increased search for 're-usable' components (often pre-packaged components from vendors). It's possible that engineers started to see less value in explicitly documenting the separate views.

Or perhaps it was the rise of Extreme Programming (XP) and Agile? They have paved the way for a new kind of development process, one which eschews heavy formal frameworks like RUP (although Kruchten maintains that RUP is agile-compatible). Were the 4 + 1 views the baby thrown out with the bathwater?

4 + 1 Architectural View Model: Introduction

2019-02-27T00:00:00Z

Have you ever worked on a microservices system experiencing challenges like these?

I've separated my system into loosely coupled microservices - why is integration between them still so hard? I'm spending all my time implementing circuit breakers, bulkheads & retry patterns when I'd rather be writing business logic!
My autonomous microservices teams are still blocking each other from delivery - why?
We still struggle with version incompatibilities between microservices... we've had to invest heavily in consumer driven contract testing to ensure we don't break things
Making an update is so hard - we have to update code in multiple repositories containing our shared libraries and wait for a nuget or npm publish cycle in each of them before we can test the integration of changes

Or perhaps you're a startup who doesn't want to over-engineer their launch solution, but also doesn't want to end up with a big ball of mud, or unable to scale their system when the business starts doing well?

Either way - the "4 + 1 Architectural Views" may help.

This is the first in a series of posts about the views. Throughout the series, I hope to demonstrate how useful the 4 + 1 Views can be in understanding complex distributed systems, as well as facilitating more nuanced approaches to system design.

This first post will provide a brief introduction to each of the views, with deep dives into each view coming soon.

Origin Story

The origin and context of the 4 + 1 architectural views model is a fundamental part of the history of Software Architecture itself - a separate post explores that history.

The Views

The "4 + 1 Architectural Views" were proposed in 1995 to solve increasing challenges communicating about software architectures.

The purpose of separating the architecture into multiple concurrent views was to isolate and illustrate different aspects of the design with information specifically oriented towards different stakeholders.

Each view should:

define the system in terms of Components, Connectors and Containers
use appropriate styles, forms and patterns for each view
consider applicable constraints in each view
consider the relationship between the views

The following diagram from the wikipedia article illustrates the views and relationships between them.

The 'classic' 4 + 1 views

For any given system, you may not need to document all the views, and there may be other, more relevant views for your system. I recommend creating the minimal documentation to explain architectural decisions, and ensure you are ready and willing to evolve your architecture (in each of the views) as your solution matures.

Logical View

The logical architecture primarily supports the functional requirements — what the system should provide in terms of services to its users.

Logical View

The logical view illustrates how the system is decomposed into specific areas of functionality. You will likely need to diagram the system at various levels of abstraction.

If you are using Domain-Driven Design (DDD) then at the highest level (the most abstract) you might show domains, sub-domains and bounded contexts and the flow of events between them. In the Domain Blue Book, this is called the 'context map'. At the lowest level (the most concrete), you might illustrate aggregates, aggregate roots, entities and relationships.

However, DDD is not the only way to design software - for some systems it may be better to illustrate functional modules, perhaps with a simple entity-relationship diagram.

Deep dive into the Logical View: coming soon...

Process View

The process architecture takes into account some non-functional requirements, such as performance and availability. It addresses issues of concurrency and distribution, of system’s integrity, of fault-tolerance, and how the main abstractions from the logical view fit within the process architecture [1]

Process View

The process view gets us to closer to an illustration of running software. Components in the process view are real executables that make up the run-time of your system.

For example, each of the following would be considered a 'process':

IIS Web Application Pool
Single-page app running in a browser
Linux Daemon
Database engine running stored procedures or SQL scripts
etc.

Processes can be thought of as a unit of 'tactical control'. An individual process can be deployed, started, recovered, reconfigured, scaled out, and shutdown.

The purpose of the process view is to start to capture both the flow of inter-process information exchange (e.g. REST API calls, messaging via a bus) and the sequence and timing of these inter-process communications. This allows us to consider questions such as concurrency and reliability.

Deep dive into the Process View: coming soon...

Physical View

The physical architecture takes into account primarily the non-functional requirements of the system such as availability, reliability (fault-tolerance), performance (throughput), and scalability. The software executes on a network of computers, or processing nodes (or just nodes for short). The various elements identified — networks, processes, tasks, and objects—need to be mapped onto the various nodes. [1]

Physical View

Even with the advent of Cloud computing and virtualised hardware, the physical architecture view still plays a critical role in understanding system performance and capacity.

Components are either processing nodes (servers, virtual machines, docker containers, serverless configurations, etc.) or networking channels (routers, firewalls, proxies, load balancers, etc.) and the diagram should illustrate which nodes host which processes from the process view. This allows us to start considering compute and network capacity, as well as latency and other performance considerations.

Deep dive into the Physical View: coming soon...

Development View

The development architecture focuses on the actual software module organization on the software development environment. The software is packaged in small chunks — program libraries, or subsystems — that can be developed by one or a small number of developers. [1]

Development View

I often think about the development view as the "source code" view. As such, you can consider it as having two 'sub-views', or perspectives:

Physical perspective, comprised of:
- repositories
- file systems
- IDE projects/solutions & structure (e.g. Visual Studio Solutions)

The physical perspective helps you 'find' the code. There is a concept of 'nearness' which can be helpful when considering what code you want to keep 'near' what other code to enhance cohesion.

Logical perspective, comprised of:
- modules (e.g. assemblies, packages)
- dependencies between modules
- layering of modules

The logical perspective helps you understand how the code modules work together. It allows you to understand static dependencies between the modules, as well as the flow of execution through the actual source code. This should help with understanding a run-time model - i.e. which modules will execute within which processes in the process view.

Deep dive into the Development View: coming soon...

Scenario/Use Case View

The elements in the four views are shown to work together seamlessly by the use of a small set of important scenarios —instances of more general use cases—for which we describe the corresponding scripts (sequences of interactions between objects, and between processes) [1]

Scenario View

The scenario, or 'use case' view helps tie the architecture together. Different diagrams help to illustrate the flow of activity in a system and illustrate which logical, process, physical and development components are working together to facilitate the outcome.

Deep dive into the Scenario/Use Case View: coming soon...

Why should I care?

In this series of posts, I hope to demonstrate that many challenges facing teams today are a consequence of blending and ignoring the distinctions between the views - trying to solve problems in one view with design patterns in a different view, or by tightly aligning the design in each view. My experience is that if you allow yourself to evolve your solution in each view independently, many of these challenges become much easier to reason about.

This means you can plan ahead to scale appropriately, you can more easily trace performance or scalability issues back to the source, and you can understand why different teams (owning different components in the development view) have complex developer-time interactions affecting autonomy and progress.

After exploring each of the views in depth in the coming posts, we'll return to this idea in more detail and explore how specific view alignment (or mis-alignment) patterns cause common risks and challenges.

Introduction

2019-02-24T00:00:00Z

Hi, welcome to my blog - it's a place for me to share my experiences co-founding and helping grow two technology businesses to scale. I'm a developer at heart so it will mostly be about the tech, but it will also be about the intersection of technology, business, leadership, society and ethics.

Along the way, I hope to explore the history of our discipline. Over the years I've noticed how frequently the solution to a problem I was facing had been developed many years before. So began my fascination with the trajectory of ideas: how they are shared, diluted, abandoned, and reinvented, each time with a new spin in a new context.

About the Name

For a while I was calling this site (and my coaching/advisory business) "Dev Cycles". Although I no longer use that branding, this is an explanation of the name, as I still find it interesting! Dev Cycles is a reference to the software development life-cycle, which describes a general process of software development from concept to outcome.

The banner image above is tangentially related - it's a slice from a diagram of epicycles in the Geocentric model - an ancient idea in the history of astronomy (which turned out to be very wrong!) trying to explain what was at the time inexplicable - the motion of the planets and stars.

The Geocentric Model

Apart from the name association, the metaphor continues:

The epicycles themselves relate to the cyclical nature of technology trends - fads come and go like dancing planets.
I hope to use this blog to tell stories about the evolution of ideas, and the journey from epicycles to the general theory of relativity is surely a gold standard in the field!
I have often felt like I was trying to solve a technical challenge with a conceptual framework that was as ill-equipped to understand the work ahead of me as epicycles were to explain the motion of the heavens.

_{Image: Cassini_apparent.jpg by James Ferguson (1710-1776), based on similar diagrams by Giovanni Cassini (1625-1712) and Dr Roger Long (1680-1770); engraved for the Encyclopaedia by Andrew Bell. - Encyclopaedia Britannica (1st Edition, 1771; facsimile reprint 1971), Volume 1, Fig. 2 of Plate XL facing page 449. Sourced from wikipedia.}

Join the Conversation

An aspiration of this blog is to explore a connection with the origins of ideas, so I value any personal stories and experiences that add flavour and context.

Please join the conversation and leave a comment - a critique, an observation, or a memory of your own experiences - all welcome!