Why Waterfall Project Management Threatens To Return
“Especially for complex and costly IT deployments, customers need to invest more in the up-front evaluation, even if that entails … detailed requirements analysis ….” Rob Preston, “Montclair State Sues Oracle – Another Train Wreck,” InformationWeek.com, June 13, 2011.
And so it begins. Let’s do more up front analysis to keep us safe. More documents in advance, assuming that this will prevent any such slipups. So has this kind of waterfall approach succeeded in the past? Sometimes. But this is often how it gets started. Something goes wrong and the solution is often just this “figure it all out in advance before anything is done or committed.”
When I started computer programming, I would argue that we were what folks are now calling “agile.” Actually, I would call it chaos, but that is because we were often too eager to please. If a customer asked if our software could do “X” we would change the code and show it to them within a day and say “do you mean this?” This would start a back and forth relationship where we’d “tweak” the system until it was pretty close to what they wanted. Requirements document? What was that? Design and code inspections? What for — see it works! Comments in the code? Well, we are still making changes, so we don’t want to spend too much time doing that yet.
When this pre-agile approach inevitably caused degradation of the code and operations, management would step in and add controls on who could change the operational system and who would approve it. Additional phases and checkpoints were introduced to ensure all due diligence was done before proceeding to implementation. Pretty soon, we had huge checklists and process documents on how things were to be done. To make even the smallest change required significant documents, procedures and approvals. All sorts of “support” organizations popped up to ensure we were following the process and to ensure “quality” by auditing and inspecting what we were doing. There were now more “support” people than there were software programmers.
In retrospect, we clearly went too far in the “figure it out all in advance” direction just as we had gone too far in the “code and go” direction before that. It was all motivated by things going wrong and the unproven assumption that if we figured it all out in advance, then things would be better. They weren’t and the trend today is agility in software development and project management because the waterfall approach was not consistently improving results.
Recommendations as in this editorial too often help to send us, once more, to an extreme. Playing it safe is, well, safe. But it doesn’t necessarily result in better outcomes, only more “self protection” when the results turn out to be not good. It takes courage to do a really good job and that often means sticking to what we know works and managing the risk, rather than taking the personally safest route. I would have liked Rob’s conclusions better if he talked about using a more incremental or agile approach to minimizing risks and ensuring we were getting what we asked for. Instead, he suggested “an ounce of protection” but we know that these ounces often grow to tons of overhead without any improvement in the outcome.
Is your project management threatening to become overloaded with up front activities that add questionable value?
9 thoughts on “Why Waterfall Project Management Threatens To Return”
Comments are closed.
Agile is simply a lean way of managing software development projects.
Upon starting the project, one has to identify the sweet spot between ‘over analysis’ and ‘shots in the dark’. You plan ‘just enough’ to envision how the project will be (on the long term) and the things you need to start your project.
Agile does not give clients a free pass to always change their minds. They will have to pay when they change their minds, but it teaches the developing team to be ‘open to change’.
Agile is not chaos, it is discipline. The incremental process requires technical excellence. The teams are always results-driven, and plan to adjust their process and thinking to adopt the new needs of their environment.
Osamah,
Yes, that balancing act at the sweet spot is one of the greatest challenges for a manager — and one usually gets better at it with experience. Our chaos was just our enthusiasm and while enthusiasm is great, it needs to be disciplined as you indicate. As I mentioned in the article “being open to change” was not our problem, but more mature heads decided we were too open and doing as much damage as good. Over time, the desire to be “safe” in every change we made probably got too safe (i.e., too many controls, restrictions, etc.). It also built a culture of “be safe” which helped to deflect some of the blame (“but we followed the process!”) when things did go wrong, which was not a good motivation for keeping such controls — which were at the heart of many of the reasons things went wrong.
Thanks for the thoughts.
Bruce
Without taking positions on either sides, I agree that we have been hitting extremities of the swinging pendulum – if waterfall was one extreme of the rigorous everything-upfront world, then agile is returning it back to as unregulated as it could be – and its stated simplicity often makes people doubt its potency claims. I firmly believe that in second decade of agility, which I like to call as the maturation period, is when agile will transform to become better aligned with business topline then just making an impact on a team’s performance.
Tathagat,
A few years ago when I tuned into Scrum software development, I fell in love with the technique. However, I always found it interesting when listening to the Scrum gurus/founders during question and answer periods. They would often admit all the normal “sins” of software development popping up when using Scrum (e.g., promising more than current velocity predicts, etc.). Scrum would go “bad” when these things occurred and if Scrum didn’t work well for a team, it was usually for these reasons (or they didn’t fully adhere to the Scrum process, etc.). Gee, that is just about exactly what every other technique/methodology evangelist had said about their approaches in the past. While I still really like the Scrum approach, it was a good reminder that doing something well, almost regardless of the technique, is the primary source of success with that technique. We have to manage around all the normal, usually human induced, problems in a process and if we can do that — which is often hard — we will have an effective process: no matter if it is agile, lean or waterfall.
I realized the simplicity of Scrum was starting to wane when I first heard of a “scrum of scrums” needed to manage large efforts. Again, nothing wrong with finding ways to scale, but the history of waterfall is dotted with add-on processes and steps to handle all the real world factors one can encounter.
Bottom line is I agree. The pendulum swings are usually driven by reality and often what looks like a great new approach is not much more than an existing idea in new clothes waiting to mature. This article was simply a reminder that this maturation process adds both good and bad, and when too much bad is added we can either jettison the approach (and all the good with it) and find something new or clean up the approach so it is lean and mean again. Starting with something new often seems much easier but usually with time we reinvent much of what we had already known, in an effort to make the new approach work in the practical world.
Thanks for the comment.
Bruce
You can go either way with a Lean approach. Lean requires cusotmer feedback and incremental product development in either an agile or waterfall methodology.
Only the sponsor or God knows all, to everyone else bring proof. So before you code for feature X in the above example, you’d ask a good portion of your current and potetenail customers, “if we add this kind of feature, would that solve any pain for you?”
If yes, go to next step. If no, kill the feature.
Next step, if the feature did this, this and this, would you pay this for it?
And so on and so on.
Michael,
Good approach. I would only add that many key breakthroughs don’t come out of the customers/users, but from the ingenuity of the folks building the capabilities (or even outside of the organization). So I would add the step that we “take a chance” and periodically build in new capabilities that we suspect the customer will like, but they won’t know it until they see it.
Regards,
Bruce
Did Waterfall project management ever disappeared?
Laurent,
No, of course not. I am guilty of using a headline that is partially intended to pique curiosity. I used the editorial to show how, in my experience, waterfall often went awry. The worst of waterfall, again in my experience, came from an overreaction (though well intended) to problems encountered on previous projects. The notion is we need to pay attention to what we are adding to our project process requirements to ensure they really add value.
How I personally view waterfall can better be seen in http://pmtoolsthatwork.com/perfect-project-management-methodology/. To the point, waterfall works just fine when used appropriately (as with any methodology or technique or tool).
Thanks,
Bruce