Are waterfalls agile?

People talk about agile but what do they really mean by agile?

David Matthewman CIO, OU

I read a very insightful and interesting interview with David Matthewman, the OU’s newly appointed CIO, in Computing Weekly and I’ve also had a number of discussions with him about development and programming. In the interview he says “As part of a more disciplined approach to market methodologies, Matthewman will be introducing a more prolific use of agile and scrum development, as well as service management standards such as ITIL.”

I think this is a move in the right direction for the OU and for other organisations who similarly have adopted up until recently very traditional waterfall methodologies for enterprise level system development, however agile development methodology on it’s own won’t solve the problem. I read a paper last week commissioned by Hays for example which was about agile development called “The Elephant in the Developers’ Room” – it’s kind of drawing conclusions it wants to make the case for agile, but the headline statistics alone are stunning:-

• 60% – 80% of project failures can be attributed directly to poor requirements gathering, analysis, and management, costing US businesses $30 billion per annum
• 50% of major projects (defined as costing >£10m) are cancelled when at least 40% of spend has been incurred
• 40% of system problems are found by end users
• 25% of all spending on projects is wasted as a result of re-work
• Up to 80% of budgets are consumed fixing self-inflicted problems
• Only 8% of large-scale applications projects (those that cost between £6 million and £10 million) succeed.
• Just 16% of software development projects close within acceptable constraints of cost, time and quality.
• Cost overruns of anywhere from 100% to 200% are common in software projects.
• IT workers spend more than 34% of their time fixing software bugs

Anyway I had a twitter discussions with some developer colleagues, which is by the way the best way to solve the worlds problems, and the conclusions were as follows:-

Agile methods alone wont fix the problem of very large developments failing.

Here are some of the practical reasons why very large projects fail from experience with projects of 60m+ which I been involved in with SUN and Microsoft and others who do this stuff well on the whole:-

1. Some issues are behavioural to do with the makeup and background of the team
2. Some are to to do with poor management, not specific project management but people management and lack of ability to think creatively, direct appropriately and react (important as scope changes)
3. Project scope changes, so some issues are to do with inflexibility, not reviewing scope regularly and adapting
4. Some are to do with lack of empowerment of developers, making them both understand and grow, giving them challenge and enabling cross working
5. Some are to do with siloing of activity, “only X knows about that” mentality
6. lack of ownership of issues “not my problem mate” – it’s everyones problem
7. Good (critical) people leaving during the project. Perhaps there’s a place for a ‘golden handshake’?
8. Some are to do with complexity. i.e.not breaking the big complex system build down into smaller manageable chunks.
9. Some are to do with people not understanding the vision. Everyone must understand it.
10. Finally by far the best projects I’ve worked on are ones where everyone contributes to the solution, feel tied to the success of it. The reporting, logging and reviewing processes serve a purpose that those in the project understand, i.e. it’s directly relevant to assisting them and their colleagues. The organisational structure is kept light and serves to help development, rather than for MI alone.

As we move into a more commoditized, off-the-shelf, ROI, SLA, cloud-based, shared solution, outsourced, yield enhanced and recession proof world it’s important to remember that the ‘uniquness’ of an organisation is created through the innovations that come from within rather than without. Developers can still add that uniqueness to an organisation by building bespoke services very well and at scale.

We’ve just started a venture to develop the OU Media Player for example which is going to create ‘the worlds’ most accessible media player’. It’s built using existing services but we’ll add the value to make it provide captioning and accessibility services and to link to all OU media materials on a variety of platforms including the VLE. This is a very small team working over the next five months in an agile way. I’ve got 100% confidence in it’s success because it’s a great team, everyone understands how important it is to the OU and they’re being given the freedom to build it iteratively, creatively and well, i.e. serving the OU’s mission in being “open and accessible”.