Waterfall theologies are directly responsible for the failure of ~70% of IT projects of any size and variety. Forrester estimates that 68% of projects fail due to poor requirements gathering; Gartner regularly publishes reports outlining how 50% or more of IT projects fail; and a Project Management Institute survey states that the average large IT project runs 45% over budget, 7% over time, and delivers 56% less value than expected. (Project Management Institute: Pulse of the Profession 2015). Waterfall theology is directly attributable for most of this carnage.
Apologists cite that much of the failure with Waterfall can be attributed to ‘poor requirements’ and a misunderstanding between the business and IT – the famous ‘impedance mismatch’. Client and vendor mis-understanding on scope and requirements leads to cost overruns, conflict and failed projects including failing on delivery time, reliability, security, budget overruns and poor end platform performance which negatively impacts end-users and the business processes involved. Ergo say its defenders, if we could just get the requirements correct, Waterfall would be great.
Wrong.
Waterfall as a top-down, totalitarian construct is doomed to failure. You will never resolve the ‘requirements’ issue, or the ‘delivery’ issue within the liturgical rites of Waterfall. The entire system itself is a mockery and needs to be dispensed with. Agile and ‘team based’ IT project management is the only possible solution to the issues which plague Waterfall and render it useless and dangerous. Why Waterfall is brutally insipid:
- Top down (Why the control freak fetish? Has a committee every invented anything?).
- Managerial types, drunk and delirious on paper, plans, more plans and planning the details of the future plans; none of which have any impact on reality. Y(ou need High Level Designs, statement of work, and the details will be discovered through iteration and hands-on proof of concepts).
- Very slow process premised on the totalitarian instinct of control and that if every move is ‘planned’ somehow the end product will be successfully deployed. (Wrong)
- Mismatch in skills and resources, with a lack of engineers on the team, dwarfed by a plethora of sweaty-faced project managers and sundry beautiful minds, who enjoy micro-management, which evidently fills a missing psychological need for control and relevance.
- Technologically illiterate project management who ‘run’ the projects and enjoy the sadism of torturing engineers with inane questions and demands.
- Projects unburdened by reality or the fact that projects in the real world follow an iterative not a linear process.
- Gantt Charts used (about as useful as writing your name on a sandy beach before the tide comes in), which no one reads, understands or care about (except top managers who pretend to care).
- MPP plans foisted on engineers by Project Managers, with withering detail none of which are predicated on technical statements of work (about as intelligent as hiring a carpenter to fix your plumbing).
- Work Breakdown Structures based on said MPP-Gantt detail with no engineering input (about as intelligent as trying to forecast the weather 2 months from now and then firing someone when said prediction is falsified).
- End of process reviews which reveal that nothing was built, nothing functions but the Gantts sure do look pretty and therefore the manager in charge is promoted even though the project is a spectacular failure. Yeah.
Waterfall Theology is the quickest way to IT irrelevance.
Agile on the other-hand, based on common-sense, mandates a cultural shift. No GANTTs. No WBS. No top-down control freaks. No endless meetings. Subject Matter experts and engineers run the projects. Agile enshrines speed, fast-fail, proof of concepts, the team, the individual, the informed, and the doer. It allows for appropriate documentation, iterative builds and fails; and learning. Doing something is more important than playing buzzword bingo in a meeting, to plan a meeting, on the committee meeting to commission another study. DO SOMETHING! That is the mantra of Agile.
Why Agile is necessary:
- Agile works best with dedicated teams who are trained on Agile
- Small teams are better who know each other and who form a team around the application, platform or artefacts that need to built
- No real hierarchy or HIPPOS (highly over paid person’s professional opinion)
- Ideally you are in the same office, where the team uses flip charts, magic paper, whiteboards, and the back of their hand to analyze, design, map out and solution problems.
- The backlog or tasks to be done, are ‘scrummed’ or reported on each day in a 30 minute meeting by the team members. The backlog is worked on, obstacles removed.
- The team ‘sprints’ in 2 week increments to complete discrete deployments within the context of the larger project.
- A Technical person or Owner, runs the Agile team, not the project manager.
- Agile supports DevOps which basically means that the team who built the system, will test, operate and own it! This is a significant mind-set shift. If you won the process, you will ensure it succeed.
- Business owners and clients are embedded into the process providing non-technical feedback about what users or the market actually wants. This will close the requirements gap and through iteration provide scope and requirements clarity.
- Agile development must be automated. Pick your poison. Jira, Jenkins, Chef, Puppet, Testing tools etc. Just automate as much of the process in devops-Agile as you can.
- Integrated testing must be built into the processes and engineer training.
- Governance of risk, documentation, requirements project (sprint) reviews; and change requests are fundamental to have success with Agile.
The 2 main issues with Agile are scope creep and documentation. The first results from embedding the business owner into the process. He will then accidentally-on-purpose offer up insightful ideas into making the application or platform ‘more user friendly’. A defined contract scope will stop this. If the contract scope is vague than you need to manage the parameters. Most projects do not have really well-defined requirements. A key benefit of Agile is to discover the true requirements. The assumption that people are stupid and will do work for free is not based in reality. Just use common sense.
The second issue around documentation is solved through the use of templates for high and low level design; and checklists on security and network details. You need 5 documents within Agile;
- Scope and Requirements;
- High Level Design to stat the work;
- Low Level Design built during the iterations of work;
- Code documentation;
- Confirmation on Security and Network Best Practices.
Keep the documentation relevant and practical.
Agile is not a silver bullet. Your people need to be trained. You need to understand Agile processes and components. You need to automate and use tools. Once you have this in place, Agile is without a doubt, so far superior to Waterfall that you will shoot on site anyone asking you for a Gantt chart.
