Aliases: Adaptive Management …
Your company has been using a traditional approach to management of engineering projects. Now, senior management wants to change how they manage and execute projects so that they can better deliver software – faster and cheaper.
???
A number of organizations use traditional approaches (Linear Sequential Model or Waterfall based processes) for software development. Post funding approval, such processes begin with an obligatory requirements gathering phase followed by an analysis and design phase, a detailed design phase, development, system integration, user acceptance, and production deployment.
Traditional approaches consider obligatory that all requirements be gathered at the start of a project. Managers in such organizations implicitly assume that:
- Requirements can be well defined by the users of a non-existent product who know what they want.
- Requirements are static and will not change.
The subsequent creation of a detailed design then supposes that the solution to a poorly-stated problem (in the previous stages of the SDLC) can be defined with precision.
The waterfall method does not explicitly encourage prototyping at the design stage. This implies that design can be conducted in a purely abstract space, or put another way that trial rarely leads to error.
Eventually, the team starts to implement the design that has not yet been proven. The underlying assumption is that the technology will all fit nicely into place when the time comes. It seldom does. More often than not, projects flounder at the system integration stage as the previous implicit assumption is rarely true.
There are three problems with late discovered problems: (1) costs to fix issues grow exponentially over time, (2) it is often too late to take corrective action in time, and (3) they play havoc with the project schedule.
It is hard to accurately gauge the progress of a Waterfall project — reporting is subjective and is not focused on completion of features (the focus instead is on task completion). However, a true picture is required for evaluating options and setting expectations.
Oft burnt management tries to compensate for these failed and challenged projects by asking for more rigor up front in the belief that the PM and team did not do a good enough job planning the work — if they had planned better, there would be no surprises and no slippages and cost overruns. This is a vicious cycle and doesn’t address the real problem.
Companies that insist on heavy initial planning to ensure predictability and efficiency usually end up with projects that fail outright (cancelled prior to completion or delivered and never used) or are challenged (fewer features delivered than agreed and/or late). Details on the state of project delivery can be found in the Chaos Report on the Standish Group Web site. The Standish Group conducts bi-annual surveys of thousands of companies and projects and makes this data available every two years.
Therefore:
Use “agile” management techniques that are characterized by quickness, lightness, and fluidity (allow for changes in the business environment with minimal fuss).
While agile techniques vary in practices and emphasis, they share common characteristics, including iterative development and a focus on interaction, communication, and the reduction of resource intensive intermediate artifacts. In addition, all agile methods recognize that software development is inherently creative in all its aspects.
???
Defined/traditional processes can be characterized as “Plan, Plan, Plan, Do” processes. They can be characterized by:
- Plan what you expect to happen throughout the project
- Enforce that what actually happens is the same as what had been originally planned
- Use change control mechanisms to manage changes in users needs and business requirements
Agile processes on the other hand spread the planning throughout the project lifecycle. Rather than heavy up-front planning, agile methods rely on frequent course correction during the project.
Because traditional methodologies invest so much in up-front planning and up-front design, they are naturally biased against change. Waterfall based methodologies assume that variations are the result of errors (in the definition of requirements) and these incorrectly specified requirements lead to incorrect designs and eventual project troubles. Agile methods, however, assume that the external environment causes critical variations.
Agile methods welcome change. These processes adapt and thrive on change, even to the point of changing themselves. Changing requirements often imply a better understanding of the market, the product, and the technology being used. These changes allow the company and team to fine-tune what they are doing and delivering.
- Developing in iterations allows the team to adapt quickly to changing requirements.
- Working in close location and focusing on communication helps teams make quick decisions and act on them immediately, rather than wait on written communication.
- Reducing intermediate artifacts that do not add value to the final deliverable frees resources that can be devoted to the development of the product itself which can be finished sooner.
The problems due to the Waterfall process are listed below (a non-exhaustive list):
- Projects are significantly “challenged” — late and with reduced scope — and the problem is partially attributable to a subjective assessment of project status.
- Pushing integration and user feedback to the end ensures that deficiencies and errors will not be caught soon enough to be fixed.
- Business perception of long lead times for delivering value.
- Arbitrary order of feature delivery — no focus on delivering high business value items first.
- Applications delivered do not meet users’ needs because feedback from users and subsequent adaptation is delayed due to the sequential nature of the process.
- Focus on hand-offs (comprehensive up-front) rather than collaboration and quicker delivery. Written communication suffers from the inability to quickly clarify what was stated; this ambiguous and imprecise communication leads to undetected inconsistencies in requirements, designs, and implementations.
- Failure to attack risk early. Waterfall projects delays the mitigation of risks — « Big Bang » implementation and integration effort just before delivery date. Problems unearthed then are usually very expensive to fix as they have a cascading effect on other features, which also need to be modified.
Note that the problem is not with the Waterfall method itself. Defined processes have been used for decades with success in the engineering disciplines. The problem is the improper selection and inappropriate use of the method — you can use a fly-swatter and a baseball bat to strike objects, but the implement used varies depending on the circumstances. Any process properly applied in the correct domain will result in the desired outcome; our problem is that traditional predictive methods do not work well in most software development domains. Unlike most manufacturing and engineering disciplines, software development is more akin to continuous learning and invention. When you can’t define things well enough up-front so that they run unattended and produce repeatable, acceptable quality output (activities are not predictable, are non-linear, and are too complex to define in repeatable detail); the only solution is to control the process through inspection and adaptation.