After my unforgettable and fun vacation in Seoul and Sokcho, my original plan was to write about Systems Thinking and the book I read, "Zen and the Art of Motorcycle Maintenance" by Robert Pirsig.

But all it takes is a moment, which has lasted more than two weeks, and I feel my life has turned upside down. The last two weeks have been quite literally a car wreck. 

• First, the Sunday before the US elections, while pulling over for a paramedic, I got rear-ended by a monster pickup truck. I am fine, thankfully, but my car still is not. 

• Second, I really believed that the company I work for, the New York Times, would avoid the ULP strike by the tech workers by agreeing to a fair contract before the US elections. That did not happen. And I don't want to discuss the US election results because that hurt will not go away anytime soon. Meanwhile, my colleagues and I went on an 8-day ULP strike. Since the company refused to bargain with us while we were on strike, we decided to end the strike and return to work earlier this week. We would like to get the contract we deserve soon.

• I am trying very hard to return back to functioning, I must be succeeding because here I am, able to write some words! 

Estimates - Is it an Art or Science? 

My car has been in the auto body repair shop for the past two weeks, and I am still trying to get an estimate of its damages. I am learning that it's an interesting domain: 

• First, the insurance adjuster submits what they think is the damage to the car to the repair shop. Let's say this is $15000

• Next, the repair shop submits what they think is the damage back to the insurance company. Let's say it's $18000, and the shop estimates it would take 30 days to fix it. 

• Eventually, the insurance adjuster and the repair shop expert meet to assess the damage together. Let's say the insurance adjuster approves this new estimate from the repair shop. The repair shop then starts with the repair process. At least this is my understanding so far. 

Unforeseen Issues

But what if there is damage that can't be seen yet? Let's say the repair shop is in the process of doing the repairs they notice additional damage. Or they put the car on the frame machine to see if there is damage to the frame. In a unibody frame, it's not just the point of impact of the frame; some of the effects can also transfer to the back of the frame. So they'll have to inspect it to assess the full damage. They might start unassembling the car and notice more damage that was not originally on the estimate. Now what? 

Emergent Solutions 

Let's say, based on the new findings, it's going to cost an additional $20,000. They now need the approval of the insurance company to proceed. So they write up the new findings with the new cost and send a supplemental repair for approval. The insurance company will determine whether, at this point, it still makes sense to proceed with the repair or call it a total loss. And, of course, there are guidelines and rules to determine that. Let's assume it's not a total loss. The insurance company then approves the additional cost of repair. 

Real Talk or Crazy Talk?

Can you imagine if the insurance company started arguing with the repair shop, asking them—no—telling them that they would only pay the $18,000 and not the additional $20,000 because that was the original estimate? Does that sound ridiculous to you? It does to me, too. Thank heavens, reality does not operate like this. 

Isn't Complex Software Architecture Modernization the same? 

When you are in the process of modernizing legacy software, you are in the realm of complex software. When you look at it from the outside, you see some obvious things, and based on experience and expertise, you come up with an initial estimate. This is the repair shop's original estimate of $18,000

As you are in the process, you start uncovering additional complexity. This complexity could be similar to some hidden damages the repair shop discovered after un-assembling the parts, or this could be significant frame damage that the repair shop discovered. What do you do? You need additional approvals to proceed at this point. 

Good Leaders Ask the Right Questions

The right questions are asked if you are in a healthy software development environment when you bubble these problems up. How complex is this problem? What are the different ways of solving this? What are the tradeoffs? Are there workarounds or alternate solutions to this? 

Suppose the questions start leading to how you did not see this complexity, how you missed this, why it is taking so long, and you are beholden to the original estimated dates? May the force be with you and your team to give you the strength to see this through. Hopefully, your modernization journey will be a short and successful one. 

To Proceed or call it a Total Loss?

I have been involved in both types of modernization projects. Cases where the supplemental cost gets approved, and work proceeds until the next step,  rinse and repeat until the project has come to completion. I've also seen total loss, i.e.  the project got canned because it was too expensive to proceed, and it would cost more than it was worth. It's not easy deciding this, and there are frameworks and decision workshops to try and choose the direction. 

Is this a complex context or a complicated context?

I urge you to read the excellent article from David J Snowden and Mary E Boone on, “A Leaders Framework for Decision Making”. Using the Cynefin framework, motorcycle maintenance or auto repair might fall under the complicated context. In order to fix the car, the expert technician, after having listened to how the collision happened, also has to analyze and test multiple factors, such as unseen damage, frame damage, etc., to determine the best course of action for the car. 

However, in a complex context, you can only decide whether or not it is right or wrong in hindsight, i.e. after you've tried the thing. Does this jive more and more with complex legacy modernization software projects? I.e. you try something and learn that the integration you thought would work doesn’t work or you uncover a completely new behavior you never knew existed and you now have to account for it? 

There is no fixed path when modernizing a complex legacy system. There is no rulebook to follow. You try, experiment, discover, solve, and move on to the next piece until you're done. 

Denial - Anger - Bargaining - Depression - Acceptance??

When modernizing applications, if we understand that this effort falls somewhere between Complex and Complicated Contexts, we can implement the right sort of gauges to see how to proceed and determine success. Applying the process of estimates for a complex context is wrong. It’s like trying to use a hammer as a solution for every screw or a lug nut! Sometimes you need a screwdriver and sometimes you need a wrench!

Curveballs in modernization projects are just a reality. You can't foresee every outcome ahead of time. No amount of upfront analysis would lead you to a perfect data model. There will be new learnings almost every step of the way. Data Models will have to change based on the complexity you have uncovered. So it's a matter of when, not if, for curveballs. When a curveball hits and the estimate changes, take a step forward instead of getting angry or blaming folks or trying to do a what-if analysis on how best to bring the schedule in.

When you’re dealing with curveballs, which Ron Westrum’s defined model of culture does your organization adopt? Power-Oriented where messengers who inform about the curveballs are shot and failure leads to  scapegoated, Rule-Oriented where the messengers who inform about the curveballs are ignored and failure leads to justice or Performance- Oriented where the messengers who inform about the curveballs are trained and failure leads to enquiry? 

As long as the problem you're trying to solve is still relevant and still meets the business needs, take it one step at a time. Ultimately the software you are trying to modernize is for users. 

Tip for Leaders Leading Modernization Initiatives

To quote from the same article, "Leaders who don't recognize that a complex domain requires a more experimental mode of management may become impatient when they don't seem to be achieving the results they were aiming for. They may also find it difficult to tolerate failure, an essential aspect of experimental understanding. If they try to overcontrol the organization, they will preempt the opportunity for informative patterns to emerge. Leaders who try to impose order in a complex context will fail. Still, those who set the stage, step back a bit, allow patterns to emerge, and determine which ones are desirable will succeed". 

A New Hope

Either way I will be ok, whether the repair company is able to fix my car or the insurance company deems it a total loss. The system works. I wish I could say the same for modernization projects and the awful industry practice about how much ceremony we attribute to estimates, completely forgetting that an estimate means an approximate of the actual. I hope more and more software companies and leadership use the proper framework for measuring success. I hope this leaves you with some ideas on implementing changes or at least questioning the practices if they seem wrong. 

"I've wondered why it took us so long to catch on. We saw it, and yet we didn't see it. Or rather we were trained not to see it. Conned perhaps into thinking that the real action was metropolitan and all this was just boring hinterland. It was a puzzling thing. The truth knocks on the door and you say, "Go away. I'm looking for the truth." And so it goes away. Puzzling."

― Robert Pirsig

Matthew contributes significantly to software development, sharing his many decades of learning as an API developer, architect, and product manager to the community through his newsletters. So, when Matthew asked me sometime in May, it was a resounding yes! of course!

I started to think about what article I wanted to write for API Notes. I decided to write about APIs and backward compatibility, a topic near and dear to my heart.

Here's the published article:

Surviving API Breakage: Lessons from Modernizing Legacy Systems

Thank you, Matthew, for your patience with me, your regular check-ins, and your support in this writing process.

Thanks to my dear friends Simon Cropp and Chris Richardson for helping me make this article awesome.

Why should we even attempt to document decisions?

Whether building new services or changing existing ones, we are constantly making choices in software development. Often, our choices are guided by what we know about the business, such as its complexity or our understanding of it when making the decision.

George Box said, "All models are wrong, but some are useful." Our models and designs become incorrect when we have a new understanding of the problem or uncover a different facet of the problem. Fast-forward six months or a few years. No one knows why the software behaves a certain way or why someone made this particular choice.

Are you having a moment of Deja Vu? When working on a project, look at a design or code and think, "Why did they do this? Why did they not consider option X?" I have been there. Decision Records shine here not just as a documentation tool but also to preserve the context of our design choices and why they were made.

It's not the destination; it's all about the journey. Decision Records are all about documenting the journey rather than just the decision. What question was trying to be answered? What choices were considered? Were there any known constraints to the chosen approach? Documenting these facets gives much more insight into the decision-making process. When a vital complexity is uncovered later, we can revisit our choice and see if the decision needs to be altered, given the new data.

I wanted to be kind to my future self, so I started tracking the decisions in my project. 

How I started

My design friends at the Times had an awesome one-pager template for pitching an idea or a potential project. Olivia Cheng introduced this to me. This one-pager frames the problem and the initial discussions that have turned ideas into potential projects. The goal is to orient the team, leadership, and stakeholders. The elements of this one-pager are:

• What is the Problem? Here, you describe the problem you are trying to solve and the critical implications.

• What is the Opportunity? What does solving this problem unlock for the business?

• List of your goals

• Success Metrics. How do you measure this?

• What are the Challenges? A brief description of each challenge.

• Team and Stakeholders impacted.

I knew the real intent of this template was for when you pitch an idea, but it has many of the correct elements of what I was looking for.

My first iteration of documenting decisions

Patterns and Pattern Language have taught me that a solution for any problem always has a context. So, I modified this template to contain the following elements. This was a free flowing document with Headings for each of these elements.

Context: I started with the context in a paragraph. The context describes the specific area applicable to the problem or dilemma we must solve.

Problem: What are we trying to solve specifically? I try to be as specific as possible here. Since people have read the context, they can now understand the questions much better.

Opportunity: Here, I describe why solving this problem is essential and how it improves things for the business or the user.  

Success Metrics: How do we measure this? What are some of the things we can measure?

Impacted Teams: Who are the folks going to be affected by this decision?

Known Challenges and Constraints: Do we know of any specific issues or things we will have to live with by making this decision? I document any important callouts here.

Open Questions: Initially, when attempting to solve the problem and looking at solutions, there could be many open questions that arise that we need to find answers to. I documented them in this section.

Recommended Decision: After deliberating and reviewing the findings, I use this section to document the agreed-upon decision.

Appendix: If I had several options that got evaluated as a result of answering this question, I documented each option as an Appendix along with its pros and cons.  

Toooooo Long!

I created a lot of my initial decision records in this fashion. It was highly informative. I have some of these documents from more than a year ago, and my present self is so thankful to my past self for doing this. But as I started to create more decision records using this template, I ran into these issues:

Too verbose: It had a lot of information. I didn't need to document things to the extent I was attempting to.

What’s the Status!?: Seeing the current standing on this decision by opening the document and scrolling to the end is too much time and effort. It's a decision document; we need to be able to see this quickly!

Adoption: I was the only one using and creating this. My colleague, Alex Silverman, who is a product manager gave me feedback that it needed to be easier to read and less verbose.

Although it was a fantastic start, after trying this approach for several months, I knew I had to evolve it. After all, if no one else is adopting this, then it's a big fail.

Iterating on the approach

Documenting Decisions is not new; it's a solved problem. I wanted to see how others in my space are doing this. I researched and read several takes on decision records. The one I found particularly interesting was Andrew Harmel-Law's: "Scaling the Practice of Architecture, Conversationally." In this article, among other important things, he talks about Decision Records as a thinking and recording tool. I was glad to find a lot of commonality in what I was doing and what he was describing. I loved the idea of the "Status". This would most definitely solve one of my problems I encountered.

I also remembered my Design Team's Research Plan document. The information was presented in a visually pleasing way, grouping the right information in a table and it was so easy to read and get to the right information points immediately! And this solved my other problem. Also, it wasn’t just a boring table, it was a visually pleasing table.

Once I made the changes, I immediately chatted with Alex, who had given me the initial feedback. He loved it. He and I reviewed and tweaked the document to group the related information. Spoiler Alert: The iteration described below is heavily used in my project. We have a template for it. And most importantly, I am not the only one creating these decision records.

Elements of my Decision Record

Status

When looking at the decision record, you must be able to immediately tell the decision's status. I took Andrew's suggested status but kept it to four values: Draft / Proposed / Adopted / Retired.

You can start the decision record when you only know the problem you are trying to solve. The status in this case is Draft. As you begin exploring the issue, you can start to document the other parts of the decision record, such as the Options Considered. When you're ready to recommend a choice, you can change the status to "Proposed ." and fill in the "Recommended Decision" section. You then circulate the decision record with the stakeholders/decision makers and have a meeting if that's how things are decided in your organization. Once you get their sign-off, you can change the status to "Accepted." When you have to revisit any accepted decision in light of new information, you can start a new ADR and mark this one as "Retired."

Action Items

Sometimes, you need to chat with other people. So, I added a list of checkboxes as a list of to-dos. This checkbox list reminds you of what you must do to move this decision forward.  

Question to decide on and Context

Question to decide on: Describe in a sentence what question you are trying to answer.

Context: Describe the context in which this question or problem applies. Anyone new to this problem or a new team member should be able to read the context and understand the question that is trying to be answered.

Recommended Decision, Supporting Arguments, and Consequences / Constraints

Recommended Decision: When starting with the decision record in the "Draft" status and you haven't explored all the choices, it is okay to leave this section blank. However, once you have considered all the options and your decision record is in the "Proposed" state, you can write down the decision you recommend.

Supporting Arguments: You note down why this option was favored amongst the other options.

Consequences / Constraints:  If there are any potential ramifications to this decision, whether positive or negative, document them here.  

Other Options Considered, Impacted Stakeholders, and References

Other Options Considered: When your decision record is in the "Draft" status, this is where you will be working most. As you explore each option, give it a name and start documenting the pros and cons.

Impacted Stakeholders: You can list the impacted stakeholders even when your decision record is in "Draft" mode. When you eventually propose the recommended option, i.e., and it is accepted, you can make a note of the specific people who have signed off on this decision.

Related References: Include any relevant artifacts in this section, such as meeting notes, documents, miro boards / figjam boards, and Slack threads, that are relevant to this decision-making.

What do you do with this decision record?  

You can create a markdown version or an Asciidoc template and check it in your source code repo. Alternatively, store the decision record as a Google doc in a folder designated for your team's relevant project.

Neurodivergent Friendly!

When I look back at the original iteration I had, my brain was struggling to quickly grasp all of the important points. When Alex, who also happens to be neurodivergent was feeling the same way, I knew I had to make changes. Not just to the parts of what goes into the decision record, but also how it is being presented. On a side note, when you accommodate the needs of neurodivergent folks, you end up making the whole thing better for everyone! The presentation and how the information is laid out makes a huge difference. Perhaps this is the reason for its successful adoption.

Additional References

• Andrew Harmel-Law’s, “Scaling the practice of Architecture, Conversationally”

• adr.github.io - There’s a wealth of information documented here including Michael Nygard’s gold standard of what should a light weight ADR contain.

• AWS Prescriptive Guidance - Using Architectural Decision Records to streamline technical decision-making for a software development project

Start documenting your decisions today!

Here's a link to the Google document that I now use as a template if you want to use this as well:

Product / Architecture Decision Record

Trust me, your future self will thank your past self! Do this now!

DDD is more than just a set of principles; it's a practical approach that empowers you to delve into the complexities of a domain. By understanding the business rules and problem space, you can design software that's functional and aligned with your business needs. It's about more than just writing code; it's about creating solutions that make a real difference for your business. 

There’s more than one way to solve a problem!

When I was in high school, I always loved math and, in particular, Algebra. I don't know about you, but the amount of time I spent memorizing the formula for solving quadratic equations was such that if you wake me up in the middle of the night and ask me the formula today, which is almost thirty years later, I can recite it in a blink of a second. I impressed my high school kid with this scary memory recall when teaching him Quadratics! However, there are multiple ways to solve a quadratic equation; the formula is just one way. If I've now made you curious about quadratic equations and the other different ways of solving them, here's Khan Academy to your rescue. Working in software, we pride ourselves to be very logical beings and yet succumb to strong opinions strongly held. If we remind ourselves that there are more ways and more perspectives to look at a problem more deeply, we might become more curious instead of being dogmatic. In that process, we are then nicer to each other and become better collaborators. 

When trying to understand the current system and how it works, There is a system. This system may comprise several services and components or be just one giant monolith with different surface areas. The primary reason the system exists today is that it is still bringing in revenue. And that's because the system is still helpful and serves the users' needs for which it was initially built. 

My challenges with the Big Picture Eventstorming

When I joined the NYTimes almost two and a half years ago, I had to quickly understand the domain of subscriptions and newspaper delivery. I had to learn how it works today and the pain points to help identify the pathways to modernizing the system. 

Challenge 1: Assembling a large group

Like any DDD enthusiast, I started with Big Picture Eventstorming. My initial hurdle was assembling a diverse group of domain experts, product owners, and other stakeholders for the meeting. Coordinating everyone's schedules, especially in the remote work environment, was a challenge. 

Challenge 2: Asking business folks to imagine the system as domain events

The next challenge I immediately encountered was asking everyone to imagine the system as a series of domain events. First, you need to explain what domain events are: significant things to the business, essential happenings that have changed state. Then, you tell them the semantics of it, i.e., you have to name these domain events in the past tense because events are statements of facts that have already happened. Naming things takes work! Remember these famous words?

"There are only two hard things in Computer Science: cache invalidation and naming things." - Phil Karlton

Yet we asked everyone to do this within the small time frame of the remote meeting, which proved difficult. The business folks thought this was very techie, and I agree. Instead of speaking in their language, we were dictating the terms of language and communication. 

Challenge 3: Cleaning up the duplicate events and the timeline

Then comes the cleaning up of the duplicate events and arranging them in the timeline because the first part of brainstorming all of the events is just that: lots of sticky notes in Figjam or Miro. You can get to this step after a few hours, but I didn't have the luxury of setting up four-hour or all-day workshops. 

Challenge 4: No clear understanding what services and components are part of the current state

I made some progress, but I needed to understand how subscribers were getting their paper delivered! I wanted to understand the different components and services that were part of this user journey and how they interact with each other. Big Picture Eventstorming did not help me with that. That may not be the intention of Big Picture Eventstorming, and I may have tried to use it incorrectly. 

Exit Big Picture Eventstorming. Enter Service Blueprints!

At this point, I wanted to try a different technique, Value Stream Mapping, which is an excellent way to document business processes but not user experience. This is when my good friend Olivia Cheng asked, why not try Service Blueprints? She pointed me to an article from nngroup.com on Service Blueprints, and I was immediately intrigued. This was a research method that helped in designing service experiences that delighted the user. And I was interested in trying a new way. Little did I know back then how useful it would be. 

When designing a system, it's crucial to understand how the user experiences it and what they feel. A Customer Journey Map is a visual tool that can help with this. Similarly, a Service Blueprint provides a holistic view of how all the different parts of the system work together to meet a user's need. By extending this technique to the software design field, we can visualize how all the different architectural elements work together to meet the user's needs, fostering a more user-centric approach to software design. 

What’s Next?

Stay tuned. I will continue to write about how I've managed to evolve the Service Blueprints from the design world and hack them into the Software Design world to document current state and gain insights to make the experience better. I have the wholesome approval of my friends in the Design space, who were kind enough to share how I'm using them in their community of practice! If you cannot contain your curiosity and are itching to read about Service Blueprints, my good friend Chris Richardson wrote a few articles on the topic.

Sometimes, when the problem is so complex, it can be overwhelming. And it doesn't matter at what point you start looking at the problem space—whether you were there when the company first decided its strategy on how to modernize it and where to spend the dollars or if you are looking at the problem space after the strategy is already set in place.

Reflecting on my past work in the application and architecture modernization arena, I've discerned a recurring theme. Projects that prioritized understanding the users and their needs from the outset had a significantly higher success rate than those that treated user needs as an afterthought during testing.

Often, it takes more work to focus on the architectural complexity alone. As Architects, we like to take comfort in familiar things. We've spent many years studying software design and architecture and still learning daily. Domain-Driven Design, Loose Coupling, Cohesion, and what style of architecture must we use for the given problem context: Layered Architecture, Vertical Slice Architecture, Micro Services? What are the different communication styles between these components and services: RPC, and Messaging? And what about the cloud: Azure, AWS, GCP? The list goes on and on. When we start looking at the System, we see services, components, databases, and all of the technical and infrastructure parts. These parts make sense for us, the techies.

One of my favorite Systems Thinkers, Dr. Russ Ackoff, said this, and it's stuck with me: "It is much better to do the right thing wronger than the wrong thing righter."

If we focus all our efforts only on these technological aspects of modernization, we could lose sight of why we are building or modernizing the System in the first place. What's the point of spending months and months creating all these services and components if the users who use the System are still stuck with the same problems? Can we call it modernization if we have replaced all of the legacy parts with new shiny things built with the technology of today if the users don't see any benefits in the behavior of the System?

We can take a systematic approach to modernization by first understanding the System's users and their needs. Once we have determined this, we can see how today's System fulfills those user needs. What are the components and services involved? Do these belong to the same team or different teams? What are the pain points from both the user perspective and architectural perspective? A visual approach works best to do this well and bring about a shared understanding of all parties involved.

I've learned and used many diagrams to describe the System: Data Flow Diagrams, UML, C4 Modeling, State Flow Diagrams, Sequence Diagrams, etc. These diagrams represent the parts of the System in a very techy, component-specific way. And then there are the arrows! In some of these diagrams, you have arrows that crisscross boxes and more arrows pointing to more boxes, which gives my brain a headache! But how do we connect these to the actual user using our System? I've learned that you can do both! We can be very intentional in the techy parts of the architecture while still starting with the user needs.

All the stars in the universe aligned for me and brought me to work closely with Olivia Cheng on the same modernization project briefly at the Times, and I was introduced to new ways of thinking from the realm of design about how to think about the user and user needs. My first introduction to Service Blueprints. Since then, I've been experimenting with the Service Blueprints approach for the past two years by tweaking the original concept from the design world and adding some architectural elements to the mix that help understand how the existing components work together to fulfill a user's need. So far, it has proven invaluable to my modernization work in understanding the System's current state of how a specific user need is being fulfilled.

Does this sound interesting? Come join my hands-on workshop at New Crafts on May 16th if you're in Paris. If not, stay tuned—I plan on writing a whole lot more on this topic in byte sizes!

A huge thanks to my dear friends Chris Richardson, without whom there probably wouldn't be a domainanalysis.io, and my dear friend Diana Montalion, who've been both constantly nudging, encouraging and nurturing my writing habit!

Someone said this, and it stuck with me: "You don't need to post every day or even every week to have a blog that matters." I won't inundate your mailbox with articles, but when I do write and publish, it's something I have been eager to share, however agonizingly slow I am in publishing it.

These articles are my take on analyzing the complexity of any given domain using structured approaches, such as Systems Thinking, Domain-Driven Design, and other methods that stem from the Service Design world, like Service Blueprints, I’ve learned along the way.

Subscribe now