(This piece — written by Bhushan Nigale — is the fourth in a series that explores the evolution of the software technology stack and software development methodologies in the last two decades. In this instalment Bhushan examines the consequences of the widespread adoption of Agile and Lean.)
In article two of the series I presented the various forces that have led to the evolution of software development practices from the Waterfall model to Lean and Agile. We saw a variety of proximate causes that caused this evolution: the increasing role software plays in all spheres of our life, the massive changes in software architecture and the mainstreaming of Open Source software, the increasing consumerization of IT and the changing demographics of the software industry.
This article examines the consequences of these changes. Mainly, it answers the question: did Agile and Lean hold on to their promise? When a species evolves to adapt to its new environment, manifest changes appear. Can we discern such changes in the industry, for instances in workplaces and the roles played by practitioners? If we live in a post-Waterfall world, what are the obvious signposts that the changes have ushered?
In what follows, I provide an overview of how other industries have begun to adopt Agile, to what extent the hierarchies still matter, the roles of teams over individuals, and the rising importance of roles such as the Product Manager.
Agile delivers
The agile movement that arose from the Agile Manifesto is now widespread to the extent that software development organizations consider it the de facto style for delivering innovation at scale. Software development and implementation projects are risky, failure-plagued endeavors: while statistics widely differ, reliable studies (such as the Standish Group’s Annual CHAOS report) report as high as two-thirds of technology projects ending in partial or total failure.
With its emphasis on involving end-users as early as possible and then collaborating with them, smaller release cycles and a clear articulation of user requirements, Agile addresses the most crucial reasons for these failures: users become stakeholders, vested in the success of the project, rather than just using the project ‘thrown over the fence’ to them (e.g. by IT departments). The transparency in progress improves trust and the health of interdepartmental relationships – truth is the best disinfectant.
Hierarchies matter less
A counterintuitive, but welcome, change has been the gradual flattening of organizational hierarchies. While Lean originated in manufacturing companies, traditionally hierarchical with a ‘command-and-control’ operational model, its fundamental principle of putting customer value first meant that employees need to be more empowered to ensure this principle lives in practice. Thus, a product owner several levels below the unit head, takes significant decisions and takes accountability in the success of the product: brand new announcements in products and cloud services are increasingly made by Product Managers and not development departmental heads.
Managers themselves stepped back from their hands-on, roll-up-the-sleeves style to a ‘coach’ role, imparting wisdom gleaned from experience and focussing on developing team members. Not all managers have managed this transition and the apportioning of their responsibilities successfully.
Adoption of Agile widens
This success of Agile – despite the inevitable failures that we address later – has led to its increasing acceptance and adoption in disciplines outside of software product development. While the results have been decidedly mixed, traditional functions (as opposed to newer product lines/new businesses with focus on innovation) are increasingly adopting Agile principles, as evinced by the widespread use of the terms ‘scrum’, ‘backlogs’ and ‘daily stand-ups’. Large IT departments are moving away from a purely project-based implementation mode to a hybrid mode, with a combination of Agile and traditional project management approaches.
A similar transformation has occurred in companies embracing Agile, as pressure has increased on traditional companies – the ‘not-digital-born’ companies – to deliver faster and in quality. Departments outside of IT enthusiastically begin to embrace Agile, as the recent story on Mozilla shows – reportedly, the Open Source company runs its marketing department using an approach inspired by Agile.
Learning and Development departments too have begun to include Lean and Agile in their approach, with Agile learning being promoted as an “approach to training and development that focuses on speed, flexibility and collaboration.” Even staid, process-intensive sectors such as Pharmaceuticals, are gradually embracing Lean as expectations mount to deliver pandemic-combating vaccines while reducing inventory levels quickly.
Winds of change are even blowing in NASA. NASA has its own Waterfall variant, with detailed policies and guidelines as the system progresses through the development life cycle. However, for the planned 2024 moon landing program, NASA uses Agile. The Orion Multi-Purpose Crew Vehicle team, responsible for checking critical functions of the spacecraft, is fashioned using the ‘self-organizing teams’ principle. The project lead reported faster decision making, cost savings and flexibility, resulting in a better product.
A fusion of Flows
Another central tenet of Lean found a wider resonance: Flow. Lean Flow stipulates how items in a process move from the first step to the last, as quickly as possible and in quality, to ensure customer satisfaction. With the direct and early involvement of users, teams began to firsthand observe and experience impediments to Flow: idling, unfinished work that assumed forms such as a highly detailed requirements document, in-depth design and architecture discussions, a long-drawn testing plan, etc. This introspection – triggered by the Retrospective session fundamental to Agile – led to a rethinking of the software delivery process to eliminate waste, both obvious and camouflaged. (The NASA team mentioned above discontinued sprints later, but continued their retrospective sessions, as it aided faster decision making.)
Thus, when a team truly delivers value in a smooth iteration, it experiences another type of Flow, first described by Mihály Csíkszentmihályi (albeit in another context): a team delivering value is so fully immersed in a feeling of energized focus, full involvement, working in full harmony with each other, and enjoyment in the process of the activity that it loses track of time.
Flowing to DevOps
When applied to the delivery of cloud-native software services, the Flow principle expands the horizon of a team/organization’s responsibility to ensure the successful operation of software: the value is realized only when the software is available. Agile has thus paved the way for DevOps, the movement that emphasizes tighter communication and cooperation between software developers and other information-technology (IT) professionals, most notably Operations.
Viewed with the lens of the Flow principle, teams not only create and deliver cloud software, but also ensure its smooth operation, and shorten the development cycle and improve the quality, using techniques such as Continuous Integration and Continuous Delivery. The success of largely autonomous, cross-disciplinary teams has paved the way for what was unthinkable when I began my career in the late 1990s: a unified view of development and operations (“DevOps”) rather than silos. Many operational tasks are unplanned and critical (“live site first”), and the Agile approach (one backlog, customer value first, extensive communication) provides similar benefits for these unplanned tasks as well as it does for planned tasks: Agile and DevOps work beautifully in tandem. DevOps aims at establishing a culture where the interdepartmental boundaries/silos between development and Operations are torn down.The resultant convergence of people, processes and an impressive variety of practices have promoted agility.
Teams over individuals
Software skills are increasingly specialized, but complex problems demand that specialists in various domains work together, especially if the solutions are unclear and requirements are fast-changing. This has led to the rise of multidisciplinary teams that are largely self-governing, as opposed to a ‘project task force’ or even a ‘rockstar developer’. This in turn has led to a partial-to-complete democratization of the software development process: less-to-no code ownership, an ability to challenge unrealistic timelines and a gradual shift of decision- making to teams. Teams even feel emboldened not to ship a release not meeting the ‘done criteria’.
A happy side-effect has been (at least until the time people worked in offices) the more communal feeling Agile lends to workplaces. Even the youngest/newest team member has a voice – the ritual of the daily stand-up ensures this. Colorful Kanban boards, the sometimes-festive celebration of results with a sprint review not only bring transparency, they create a sense of forward movement, of togetherness. (It is hard to feel enthusiastic in the same way about a project steering committee meeting.)
Thus, when the normal working mode was upended by the COVID-19 pandemic, development teams could easily switch to the changed realities, and (anecdotally) did not experience sharp drops in productivity as they already had considerable experience in self-organizing themselves. Impressive tools in the arsenal helped here: the practices of having a well-defined, prioritized list of items via the ranked backlog, visibility in the topics via a Kanban/Scrum board, regular updates via a daily standup, and various roles where the responsibility was distributed (as opposed to a hierarchical structure).
In summary: agile practices promote team effectiveness over individual brilliance. Complexity in enterprise and consumer software has exploded. No bunch of individuals, however gifted they might be, can steer the ship, and development teams are increasingly cross-functional rather than departmental. Members of self-organizing teams iterate over a plan-do-check-act cycle using effective communication means such as sprint planning, daily scrum, sprint reviews and retrospectives.
Tools ecosystem
The software industry is unique in its relationship with tools. We are fully vertically integrated: not only do we have software and tools for building and testing software (IDE, compilers, etc.), distribute software (online marketplaces), market software (websites, social media) but also tools for how to build software. A case in point has been the use of Kanban (Japanese term for a signboard). People have used post-it/sticky notes to remind themselves of important tasks. The Kanban system took the idea further for a team. Thus, using sticky notes on a whiteboard, teams create a Kanban system, bringing in visibility on the topics at hand and their priority. Team members can then pick up tasks, lessen duplicate work, and increase the agility, thereby achieving Flow.
Having sensed the increasing demand for Kanban, DevOps and related Lean-based tools for organizing Flow, both open source and commercial software vendors have moved quickly to offer a litany of tools. Once limited to version control systems at best, ‘software to build and operate software’ has now expanded to cover a range of activities, from backlog definition management (e.g. Jira) to Kanban (e.g. MS Planner), to CI/CD pipeline management (e.g. Jenkins) to Application Performance Management, culminating into DevOps toolchains.
Thus, a software practitioner – irrespective of their function – needs more than a passing familiarity with the tools, and based on the role can spend up to hours per day navigating the tools and the processes around them – an irony not lost on more seasoned practitioners. The temptation to throw a tool at a problem is often difficult to overcome, as scores of stale Jira items and ‘Planner’ Kanban boards testify.
Role of the Product Manager
Of all roles that arose from Agile, arguably the one that gained most prominence is the Product Manager. Product Managers play a pivotal role in the end-to-end lifecycle of digital products: they conceive, define, test, deliver, and measure (and occasionally recall) products in the market to maximize business results. They are expected to analyze the market, hunt for demand signals and distill them into precise requirements, enlisting practices such as customer research. Then they work with key stakeholders across business and technology to further their product vision. With all ‘go’s in place, they translate this vision into the broad themes and epics that the teams would develop, with the help of product owners. Together with product teams they deliver and grow the end-to-end customer experience, closely safeguarding customer value.
Their remit is therefore broader than the product owners. Product Managers sometimes have a business background, but as the technology stack articles in the series have shown, technical knowledge is increasingly a prerequisite for success. Above all, product managers quickly learn that cloud-native software operates on circularity principles as opposed to the traditional linear models of value delivery. Users generate a significant amount of ‘digital ballast’ while using digital products. Using telemetry, logs, feedback, etc., product managers have a much stronger insight into usage patterns in their software as opposed to on-premise software, and can use this data to improve their product.
Conclusion
In this article, we saw the major consequences of Agile and Lean. A common theme underlying these laudable consequences has been the emergence of a new culture: adaptive, responsive and flexible with demonstrably increased trust and transparency.Though all ideals of such a culture have not always been met in practice, the cultural transformation towards agility “putting customer value first” has been visible. However, as Agile becomes entrenched, it confronts an inevitable challenge – Agile is being touted as the perfect solution for all software projects, irrespective of size, flavor and operating conditions: in short, a ‘silver bullet’. How realistic are these expectations? Why doesn’t every Agile project succeed? Do Waterfall elements still have a role, and what are the conditions in which Agile can interplay with Waterfall? We turn to these questions in the following article of the series.
IT WORKS ON MY MACHINE 