The New Methodology: Origins

(This piece — written by Bhushan Nigale — is the second in a series that explores the evolution of the software technology stack and software development methodologies in the last two decades. It examines the journey from the Waterfall model to Agile and LEAN, outlining the main factors that drove this change.)

A benefit of spending over two decades in an industry is that one develops a perspective to separate hype from substance. This viewpoint is especially useful in an industry like software, where minor feature increments are hailed as innovation, and press releases, blogs and Tweets tout routine upgrades as revolutionary. After having lived through several such hype cycles that have a high probability of going bust, one learns to exercise caution, and appreciate genuine path-breaking innovations (the first article in this series — written by Manohar Sreekanth — lists the technology changes that have stayed).

Innovation in software development methodologies is even harder to achieve and sustain. A paradigm shift is rare – at least in the original sense of the term (Thomas Kuhn used it to define  a fundamental change in basic experimental practices in a scientific discipline). Inertia is difficult to overcome, especially if established methodologies seem to be getting the job done.

I’ve been privileged to witness and experience firsthand such a paradigm shift in software development, namely the shift from Waterfall to Agile methodology. The shift has been so complete that new entrants to the industry have little – if at all – any familiarity with the older methodologies. Agile is their new default mode now.

Examining and reviewing this shift is both useful and important, because the promises of any established order need to be constantly reexamined as flaws and digressions inevitably creep in. Over time, unless tended carefully, practices tend to return to older routines — regression towards the mean is an iron-clad statistical law. Understanding the older practices and the change drivers that led to their evolution help us better appreciate the advances and detect costly deviations. An appreciation of the historical developments helps practitioners not only to address flaws, but also iterate over the methodology to adapt to the changing operational environments.

A variety of forces have led to this evolution from Waterfall to Agile: the increasing role software plays in all spheres of our life, the massive changes in software architecture and the mainstreaming of Open Source software, the increasing consumerization of IT and the changing demographics of the software industry. We examine these factors in this article, and treat the consequences of these changes in a subsequent one. 

From Waterfall to Agile

The previous article in this series traced the fundamental change in the technology stack used to build software applications. A parallel evolution, in the methodology of developing software, has accompanied these mammoth technological shifts.

When I entered the industry in the late 1990s, Waterfall had none of the negative labels one finds associated with it today. Terms such as ‘Software Requirements Document’ and ‘Handover to Maintenance’ were ubiquitous and carried a certain respect – passing a Quality Assurance Gate was a big milestone that invited celebration. The software development process flowed from a high perch (hence ‘Waterfall’) of analysis and design to the plains of testing and release, where software was then finally delivered to customers.

But cracks had already started to appear. Disenchantment was rising, both with long delivery cycles and the obsession with adherence to the strict development processes. The internet – which broke the traditional stack as we saw in the previous article – was triggering foundational changes in which software was consumed, and these consumption-driven pressures were now being transmitted to how software was being built. Consumers wanted their software delivered to them faster and better, even as it began to occupy an increasingly central part in their lives. 

Anxious to bring more predictability, agility and quality in the delivery of software, practitioners started to look at other industries that had an enviable track record of delivering value to customers in a rapid, sustainable way. In particular, the automotive industry’s production processes, inspired by a cluster of methodologies first used and honed at Toyota, stood out. LEAN thinking put customer value at its heart, and a cultural transformation – a mindset shift at its center. Various authors, such as Tom and Mary Poppendick, helped translate LEAN thinking and its core principles to the software development context.

Thus, when LEAN came in the aughts to the large enterprise software company where I worked, the excitement in development units was palpable. LEAN’s promise was simple: focus on working software, rather than working processes — customer value over process compliance. Its core principles were easy to understand — ‘flow’, waste elimination, smaller batches, ‘pull’, respect for people and culture, and relentless improvement. Putting this fundamental paradigm shift in practice proved far more difficult.

Fortunately, by then the Agile movement was gaining credibility. The Agile manifesto succinctly codified the principles that defined a new approach towards iterative software development and began to bring the benefits of these innovative methods to the industry. (Prominent software practitioners and leading thinkers in the software industry had drafted the manifesto in 2001 – a watershed moment in the history of software development.) In contrast to the Waterfall, Agile practices are iterative instead of rigid: what they seemingly sacrifice in the security of strict adherence to plans they more than make up in delivering customer value at scale and speed.

This combination of LEAN and Agile and its application to the software development process fundamentally changed the way we would build software. These methodology changes have been profound, and as I argue later, have helped software practitioners deliver immense value in these two decades. Similar to the evolution of the technology stack, these methodology changes have been large enough to discern a break between traditional and new aspects, as tabulated below along the phases of a product life cycle.

PhaseTraditional (Waterfall) PracticesNew (Agile & LEAN) PracticesDefinePlan-driven, fixed contracts between teams, timelines defined upfront with distinct phasesIterative development, flexible scope with immediate promise of value DevelopStrict adherence to development methodology and processes, well-defined handshake/quality gates between phasesRelative autonomy to development teams, loosely coupled interfaces between teams/departments, more openness and transparency to invite customer/internal stakeholders view based on the relationshipDeliverPrimacy of the process: rigid criteria for delivery, progress controlled via status reporting to management Primacy of working software over strict processes, customer value main decision driver to deliver (via demos, invited betas)Maintain/OperateRegimented separation between maintenance and development phases, organizationsUnified view of development and operations (also DataOps/SecOps), bug fixes prioritized over new features

With the benefit of hindsight, we now have a clearer appreciation of the variety of changes that continues to drive this shift: the ubiquity of software in almost every sphere of society, which in turn has led to a changing nature and expectations users have from software, the profound changes in software architecture and the accompanying changes in the consumption of software, and finally, the changing profile of software practitioners itself. In the following sections we explore each of these drivers.  

Primacy of Software

 A significant portion of humanity now spends a substantial portion of its daily time interacting with software, often without knowing it. Governments and enterprises were already at the forefront in exploiting software to reap benefits of efficiency, speed and productivity. Advances in hardware and software have not only accelerated this trend in enterprises, it has long permeated in our vanishingly small private spheres. We find software embedded in what were once purely mechanical processes enacted over physical means – driving a car, reading a newspaper, cooking a meal. ‘Cliff events’ such as COVID-19 have further accelerated the digital transformation of our lives, as the physical world cedes to the virtual.

With software becoming central in large parts of our professional and personal lives, this shift in agency to software has led to profound implications in what we expect from software. We are all consumers of software now. As consumers we expect a faster time to value, high satisfaction and place a premium on non-functional requirements such as availability, performance and security – even though we might not be expressing our expectations in exactly these terms.

Traditional development practices such as Waterfall simply could not cope with this evolutionary spiral. Much like software, Waterfall had its roots in government and defense, where stakes were considerably higher, with multi-million-dollar budgets and hundreds of specialized programmers. A scientific, rigid process smoothened information flow and brought predictability and reliability (at least, that was the idea). NASA adopted Waterfall with vigor, and its success blazed a trail: if Waterfall landed a human on the moon, it was good enough to manage inventories and invoices.

But when these inventory status checks and invoices started to appear on mobile devices, consumers started to demand the same level of usability for enterprise software as they were accustomed to in their private lives. Pleasing, functional design, speed and performance, wider functional range (“there’s an app for that”) became norms. Users also began to expect regular updates to their apps – without the long upgrade cycles. Decision makers began to demand the provisioning of functionality hitherto available only via desktop applications on their mobile phones: if they could respond to urgent requests via EMail on their device, why couldn’t they approve purchase requisitions and investment cases?

The parallel shift happening to on-Demand deployment of software meant that release cycles that were spread over months (if not years) suddenly could be compressed to weeks, days and even hours. Waterfall, with its inelastic development and release phases simply could not keep up with the needed swiftness and the speed of technology changes we’ve seen in the past two decades.(A case in point are the Microsoft Office releases. Office updates were once events, with large IT departments rolling out the upgrades in batches. Now hardly a month passes by without the Suite being updated.)

Increasing consumerization of IT

As users became consumers, buying power for IT also began to shift: Lines of Businesses now play an increasingly important role in buying and deploying software within enterprises, often supplanting the role of IT departments. IT departments reacted in turn sharply to avoid the rise of non-standard IT landscapes (‘shadow IT’), understanding the needs of agility and responsiveness. 

The ‘product manager’ role began to evolve, gradually eclipsing the project manager role in importance. This meant that business outcomes assumed higher importance than successful project execution or illusory progress – product demos gradually began to replace complex status slides filled with ‘traffic light reporting’. A single demo was worth hundreds of slides, and software development teams began to invite internal stakeholders and even customers (based on their relationship) to product demos before releasing/deploying the software. An improvement in developer tooling helped as well, with sophisticated wire-frames and mock-ups helped product managers to communicate product vision and ambition in more grippable/tangible terms. Customers welcomed the ensuing openness and transparency and being able to shape the software to fit their needs; developers enjoyed the early, direct feedback.

Software Architecture Changes

This agility grew lockstep with the accompanying changes in software architecture and deployment that continue to occur. Within a few years of the publication of the Agile Manifesto there emerged new architectural principles that both addressed and fuelled the trend of software increasingly being used by software itself [1]. These styles/principles encouraged the design, development and test of software applications in isolation from their eventual target deployment set-up.  Technology to develop independently and coherently was thus a key enabler for the rise of Agile practices.

This unfolding progression in software architecture found a mirror in software delivery. Plummeting communication and hardware costs meant that enterprise software could also be deployed – analogous to consumer software – on the Cloud and not delivered to the premises of customers. Software vendors managed and maintained software on their own premises and terms, rather than customer IT departments and their partners.  

Taken together, these twin-trends further helped the appeal of Agile. As the vendor manages and operates the software, customers expect to see progress quicker. But this shift of ownership of the deployment phase meant vendors had a bigger freedom in deciding when and how frequently new features could be deployed. Decoupled development between independent modules made possible by APIs, Microservices and the Open Source Software meant that development teams could not only implement their changes in isolation, they could also decide when to deploy these changes to ‘production’. The need to plan with extreme detail large scale release cycles — over hundreds of modules and spanning years — simply disappeared, and the DevOps movement stressed the need to think beyond the immediacies of development and release towards ensuring successful operations of the software.

Open Source becomes mainstream

The parallel rise of open source software also contributed to the adoption of Agile practices. The Open source movement promoted sharing of knowledge and code in a community, with the purpose of developing and improving software. As communities grew, informal roles within the community began to crystallize: product developer, customer, contributor, user and ultimately, the community itself became a product manager.  Tying these informal bonds together wouldn’t simply have been possible with the traditional ‘command and control’ approach commonly seen in Waterfall. 

Professionals moonlighting/contributing to open source witnessed its meteoric adoption and began to wonder whether the openness in knowledge sharing and informal hierarchies couldn’t  be replicated in the enterprise setting. As ‘Open Source took over the World’ with its commercial success and a demonstrably viable business model, its principles (and its products) became more mainstream. Organizations even began their tentative steps towards adopting the open source philosophy internally (“inner source”) with product teams opening up their code to other product teams for modification.

Developers and managers alike have enjoyed the robustness of open source software, tested thoroughly in the community. While the velocity and predictability often come in question with longer, infrequent release cycles, they appreciate the transparency of the backlog – again, hotly pondered and debated in the community, and the improved time-to-market.

Changing demographics

A lesser commented upon factor for this evolution has been the changing profile of the average software developer. The ensuing generational shift from ‘Gen-X’ to millennials, improved (but not-quite-there-yet) number of women choosing software development as a career, the increasing democratization of development tools are primary factors that are responsible for development groups becoming more diverse. Traditional command-and-control structures underpinning Waterfall offer less room for creativity for such diverse groups. For instance, Millennials are understood as being more attracted to organizations that offer an engaging, stimulating work environment. By providing more autonomy to teams and therefore individuals, Agile offers a more outcome-driven work environment to these groups. Moreover, the spectacular success enjoyed by hundreds of born-Agile start-ups like Netflix and Spotify have tempted large organizations to follow suit. (With the “fail fast, fail forward” mantra, even failure seems to have an allure, as long as agility is maintained.)

Summary

The foregoing overview outlined the main factors that led to the evolution and widespread adoption of development methodologies. Not a single factor can be singled out as a proximate cause: it is the interplay of these factors that have ultimately caused the ongoing evolution. For instance, while almost every post-war decade has furthered the role of software in society, and software architecture has constantly evolved, it is only in the past two decades that we’ve witnessed the step changes in consumer taste and the success of internet-based business models. 

The evolution is far from complete, however. These changes in turn have led to a changed environment for software and its practitioners to operate. In the fourth article of this series, I examine some of these consequences and answer the question: have Lean and Agile kept their promise?

[1] Domain Driven Design (2004), Hexagonal Architecture (2005) and Microservices (also 2005). Roy Fielding’s seminal 2000 dissertation on REST had already provided the architectural framework and formalization of interfaces for interoperability of software systems that freed developers from making assumptions about other modules without having to wait for them to function to be built first. 

Bhushan Nigale