Agile and Lean Project Management

Agile and Lean Principles

My belief is to succeed at the largest scale, Agile software development should use Lean manufacturing principles.  The command-and-control hierarchical organization of large companies is rapidly giving way to a more biological metaphor, one in which a corporate nervous system senses important signals from the market and supply chain and rapidly reacts. At the largest scale, this trend is being driven by principles of Lean manufacturing, which envisions the enterprise as a massive event-driven system, waiting to create products when customer orders arrive instead of building products in advance and holding them in inventory.

Software development has been similarly transformed by Agile development methods. In this arena, the command-and-control ethos is represented by the Waterfall development methodology, which follows a seemingly reasonable pattern of gathering requirements, designing a product, and building and testing. The problem is that requirements are almost always wrong. The Waterfall method does not allow for mid-course corrections, and the beginning-to-end cycle may span a year or even many years.

Agile posits a huge payoff for being highly suspicious of software requirements. In Agile methods, instead of building the whole product, you build the smallest possible useful part and give it to users, who tell you what is right and what is wrong. Agile development is an evolutionary conversation in which incremental steps of two to four weeks lead to feedback that allows requirements to be tested and adjusted. In addition, some functionality is available much sooner.

Quality also increases in Agile projects because using a working system exposes defects right away instead of leaving them to a final testing phase. Many other aspects of Agile are beneficial but beyond the scope of this discussion.

Agile's popularity has led to a problem: How do you apply it at scale? How can a large project run using Agile methods? One team working on one project in an Agile way is not hard to envision. But what about running 10 or 20 teams, each working on part of a product? How does the list for each iteration get sorted out and synchronized? How does the result of each iteration become integrated into the larger whole?

I've been discussing this topic for a few months with Ryan Martens, CTO of Rally Software, a leading vendor of software and training services to support adoption of Agile development. In addition, I saw a fascinating presentation about tracking Agile projects at the New York CTO Club last week in which Bruce Eckfeldt, the founder of Cyrus Innovation, an Agile training company, detailed various methods of tracking progress of Agile projects.

What surprises me is that most advanced practitioners of Agile now use more vocabulary from Lean manufacturing than from Agile. In essence, as a practical matter, good ideas from Agile are being absorbed into a new approach to software development that is Leaner than anything else. Someone else can name this phenomenon, but Lean and Agile are merging.

The fundamental recommendations of Agile methods are focused on creating a rapid feedback loop between the users supplying the requirements and the technologists transforming them into a solution. Agile methods also suggest a variety of engineering practices that increase the quality and stability of code.

Agile has much less to say about how to connect the work of many different teams, and that's where Lean has a huge impact. In a Lean manufacturing system, the work is broken into a set of value streams triggered by demand signals. The output of one value stream leads to others. Value streams may be executed sequentially or in parallel as needed. Eventually, everything is combined into the product. The suppliers for materials needed are alerted through a system of just-in-time replenishment of parts and components called Kanban.

Large organizations that are using Agile are applying a value stream approach in which various development teams are organized in sequential and parallel streams of work so that at the end of each iteration, you get a new version of the product. Sometimes different rhythms of iterations occur, with some work taking place in faster increments followed by the integration of everything every fourth or fifth iteration.

The Lean concept of Kaizen also has a strong influence on the way Agile is being practiced, filling a gap relating to continuous improvement. The evolution of Agile is primarily focused on evolving the product toward a better fit with requirements. In Agile, both the product and the requirements are refined as more is known through experience. Kaizen, a continuous improvement method used in Lean, focuses on the development process itself. When Kaizen is practiced in an Agile project, the participants not only suggest ways to improve the fit between the product and the requirements but also offer ways to improve the process being used, something usually not emphasized in Agile methods.

"The power of Agile is that it's self-adaptive, It teaches software teams how to deliver value to customers and how to improve themselves using techniques like Kaizen, allowing them to deal with unique and changing constraints and environmental factors."

In practice, Agile seems to be changing for the better by adopting Lean thinking in a large way. Customers get to market 50% faster and are 25% more productive when they employ a hybrid of Lean and Agile development methods. Given the way that Agile fits in to the Lean framework, it wouldn't surprise me if before too long Agile is considered a branch of Lean practice tailored for the software industry.

Let’s start from Agile architecture. Most of my discussions surrounding agile architecture have been focused on exploring how modularity helps increase architectural agility. I claim that modularity is a required (and to this point, missing) aspect of agile architecture. The basis for this claim follows:

·         Architecture is design, but not all design is architecture.

·         Design is architecture if the design is architecturally significant. That is, if it’s hard to change.

·         The goal of architecture is to eliminate the impact and cost of change.

·         The way to eliminate the impact and cost of change is through flexibility.

With flexibility comes complexity. We must therefore strive to increase flexibility while taming complexity.

Modularity helps us identify where we need flexibility by understanding the joints of the system where flexibility is necessary. Without modularity, we can’t identify the joints so it’s more difficult to understand where we need the flexibility. My posts titled Modularity & Architecture and Modularity by Example show visual examples comparing designs that are isolated and insulated to designs that span the joints of a system. I’ve also devoted extensive discussion to these ideas in a number of other posts, which I summarize in my Agile Architecture Presentation post.

Lean Principles:  Recently, I’ve been spending more time exploring lean software development principles and their relationship to agile architecture, and the best place to start when examining lean is with the Poppendiecks. I’m fascinated by the synergy that exists between the lean principles and agile architecture. In fact, when reading the second chapter of their book, “Implementing Lean Software Development: From Concept to Cash“, I was pleasantly surprised by an interesting discovery.

It seems that a study of software development practices by Harvard Business School professor Alan MacCormack revealed four  fundamental practices that lead to successful software development. These include releasing early, continuous integration, experience and instinct, and a modular architecture. So it seems I’m not alone in feeling modularity is a critical component of agile architecture. But the thrust of the discussion comes later on in Chapter Two, when speaking of deferring commitment.

Deferring Commitments, Reversibility, and Eliminating Architecture. Deferring commitment focuses on two fundamental factors - reversibility and irreversibility. In general, reversible decisions are those that can be changed while irreversible decisions are those that cannot be changed. We should strive to make irreversible decisions at the last responsible moment. For it is at this moment when we possess the most knowledge that will allow us to choose the most viable option. But are also advised that First and foremost, we should try to make most decisions reversible, so they can be made and then easily changed. For me, this captures the essence of eliminating architecture. If we are able to take a seemingly architecturally significant challenge and make it reversible, then we have effectively minimized the impact and cost of change to a point where change is no longer architecturally significant. Going forward, I intend to more fully explore additional synergies between lean software development principles and agile architecture.

Lean Principle First Deliver Fast : Deliver Fast.  In a way, that’s a funny principle to have.  I would have thought that’s stating the blinking obvious!  But the reality is that it isn’t.  All too often in software development, things seem to take ages.

It is common for people to think too deeply about future requirements that may or may not ever arise.

It is common for people to be blocked – maybe requiring help or information from others – and not to react to that problem with the kind of urgency it deserves.

It is common to over-engineer solutions, both in terms of the software architecture, and also the business requirements.

Why build a simple solution and get it to market quickly to be enhanced incrementally based on real customer feedback when you can build one gigantic monolithic beast of a system and build it all before you launch anything for your users? (for those that can’t detect sarcasm, that was meant to be ironic btw!).

When a team is held up waiting for someone, others in the team could potentially pick up the task everyone is waiting for and get it done, even if it’s not normally their role.  It’s important in agile teams to establish a good team spirit. It shouldn’t be the case that everyone sticks rigidly to their job specs.  ”That’s not my job” really isn’t a phrase I’d ever like to hear in an agile team.  If the team needs something done in order to achieve their objectives, the whole team should feel willing and able to help each other out, even if it sometimes means deviating from their usual specialty.

Speed to market is undoubtedly a competitive advantage.  There is considerable evidence that companies who gain ‘first mover advantage’ go on to be the most successful companies in their chosen sphere.  Companies can copy, and sometimes even come along later and do things better, but often it is the first to establish itself that wins the day and becomes the long term leader in its field.

Another advantage of delivering fast is that, if there is as little time as possible between the Product Owner stating the requirements and the team delivering the product, there is little chance of the requirements changing.  Less time minimizes the chances of changes in the market, changes in people, or even simply a change of mind.

So what is required to go faster?

Have the Right People.  As with any methodology, it’s important to have the right people.  Lean thinking in the manufacturing industry originally changed the way companies think about their people.  Instead of factory lines where the principle is to standardize everything to the extent that all people are performing small routine tasks and are essentially interchangeable, they moved towards the idea of having ‘thinking people’.  People who are able to think for themselves, solve problems, be proactive, be flexible, take appropriate actions, make decisions.  As per Lean Principle #2 – Create Knowledge.

Keep It Simple.  Another way to go faster, as I eluded to earlier, is to keep things simple.  Keep the requirements simple.  Keep the technical solution simple.  Find the easiest way to meet the users’ goals.  Don’t over-engineer.  Don’t spend too long considering future requirements that may never materialise.  Get something simple to market and build on it based on real customer feedback.

Work as a Team.  Really as a team, helping each other to make sure that the team achieves it’s objectives, whatever it takes.  Be flexible in the tasks you are willing to pick up.  When you commit to something, do everything in your power to deliver on it.

Eliminate Waste.  The Second  principle of Lean is to eliminate waste.  Sometimes it’s easier said than done, but this is clearly another way to deliver faster.

Build Quality In.  in order to go faster, you really need to build quality in.  A team that suffers from lots of bug fixing, or lots of breakages as changing one area affects another, or lots of post-delivery remediation work, can never go as fast as a team that is delivering good quality in the first place.

The third principle of lean software development is Create Knowledge. This one seems a bit strange to me, as it almost seems obvious and common sense. But then I guess we all know that common sense isn’t that common! Thinking about the fact that the origins of Lean are in manufacturing, where the traditional approach is to simplify and standardize everything to the point where no knowledge is required, like on a factory line, then I guess it’s clear why this principle was originally such a different way of thinking.

In software development, we all know that knowledge is paramount. We’ve all had situations where there’s only one developer that can work on something, at least productively, or even at all. Nothing beats the knowledge that’s created when someone actually writes the code. Although I’m not particularly an advocate of Pair Programming, this is a specific agile practice from XP (Extreme Programming) that helps to ensure that the inherent knowledge that comes from writing the code is held by at least two people, rather than one.

So, if knowledge is important, and helps the longer term productivity and flexibility of the team, you need to take some specific actions in the short term and put some specific things in place to ensure that you create it. Exactly what you do, as always, depends on your particular situation. Here are some things you could do on your projects to help create knowledge:

·         Pair Programming

·         Code reviews

·         Documentation

·         Wiki – to let the knowledge base build up incrementally

·         Thoroughly commented code

·         Knowledge sharing sessions

·         Training

Use tools, such as tinyPM, to manage requirements/User Stories

I’m sure there are lots more. But these things are often not given the priority they really deserve. What do you do – proactively – to create knowledge in your teams?

Defer Commitment.:  I’m not sure I really like the name of this one. It could easily be misunderstood. It doesn’t mean you should put off committing to anything indefinitely, or defer all decisions – that would obviously be a very bad idea.  What it does mean, is decide as late as possible, particularly for decisions that are irreversible, or at least will be impractical to reverse. Timebox critical decisions for the latest point they can be made without causing problems.

Obviously it is also important not too leave decisions too late. This can delay the team and make projects difficult. But, the later you can safely leave critical decisions, the more information you will have available to make the right decision when the time comes.

Deferring irreversible decisions means you keep your options open for as long as possible. By the time the decision needs to be made, there is every chance that you will know more about which of those options is the best route to take. It also gives you time to potentially explore the different options in more depth and experiment, helping to come to the right conclusion.

In areas of complexity or uncertainty, where things are very likely to change, this is especially important. In areas like this, as well as deciding as late as possible, you should also try to architect your solution to be flexible, in order to make fewer decisions impractical to reverse. Another example of deciding as late as possible in agile development methods is Sprint Planning, or iteration planning. In agile, you decide what features to include in each iteration and analyze them just in time for them to be developed. Keeping decisions about features and the development of those features close together helps to ensure that the right product is delivered, because it leaves less room for change.

In more traditional project management methods, in between the specification and any particular features being developed, there is a much longer period of time when change can occur. Changes in requirements, changes to the underlying system, changes in technologies, changes in people (either the product owner or the development team), changes in direction, or of course changes in the market.

Deciding too early, you run the very likely risk that something significant will have changed, meaning your end product might meet the spec, but it might still be the wrong product! This is one reason why so many projects fail. So, try (within reason) to architect your solution so that fewer commitments are irreversible. And defer commitment on irreversible decisions to the latest point possible.