A short history of development methodologies

Currently in the news is an IT meltdown at TSB. A mega-project to migrate their banking software has ended in complete chaos, with clients unable to access their accounts, deductions from one person being made to another, pay cheques gone missing, etc. It's just about the worst outcome any software development project can have, and proof again that large development projects simply do not work.

Way back in 1975 Fred Brooks published The Mythical Man-Month, a book that every manager on earth should still read¹. One of the central theses is that "adding manpower to a late software project makes it later". This is primarily because of the characteristics of human communication networks. If you have three people on a team and you add a fourth, you've just increased the number of communication channels (paths between individuals) from three to six. Add another and you have ten. Every new person adds training burdens and other costs. Teams scale poorly.

Another take-away point is that the first designed system will be a discarded prototype, whether designed that way or not. The pilot system should never, as in the case of TSB, be delivered to the client. The problem, historically, is that software engineering inherited the monolithic methods of civil engineering. But building software is a fundamentally different problem than building a bridge. What works for one does not work for the other.

In the 1980s at IBM, James Martin developed a different approach, published in 1991 as the book Rapid Application Development. Rather than centralised systems planning, RAD focuses on iterative development and rapid prototyping. Each prototype is an incomplete test of part of the system, incrementally approaching what the customer/user needs and what the company/management desires.

RAD has four phases: Requirements Planning (defining goals, scope, constraints, and requirements), User Design (developing models of system processes with direct user input), Construction (application development) and Delivery² (testing, implementation, training). Well-defined business and data models drive the whole process.

Barry Boehm made another important contribution in 1988. His spiral model relied on the iteration of four stages:
1. determine objectives
2. identify and resolve risks
3. develop and test
4. plan the next iteration

I spent many years as a software developer and project manager. I first became aware of these development concepts in the 1990s. I recall in particular the methodology of "extreme programming" (XP) because it promoted an appealing flat management structure. But some of the other tenets were difficult to implement. Pair programming, unit testing, last-minute development, and client involvement in development were all beneficial techniques. But it was challenging for the team (myself included) to change our habits.

In 2001, XP and other methodologies (DSDM, UP) were included under the banner of Agile development. Supported by the doctrine of open source code and the collaborative possibilities of the web, Agile took the world by storm. Though not the whole world, as TSB makes evident.

Over the years many of the components of Agile have been formalised. But it is not so much which of the many acronyms you follow, only that you commit to the principles. Otherwise your clients will lose their data, be locked out of their bank accounts, and line up patiently to sue you into oblivion.

Notes

¹ One of the funniest things I ever read was a misprint of this title as "The Mythical Man-Moth". I still laugh!

² This phase was actually called "Cutover" in the original book.

RELATED POSTS