Agile’s broad adoption and mediocrity – the fault lies…

I have to admit, I cringe whenever I say “agile” or “scrum” (™?) Even as I practice both every workday and care deeply for the values they represent.

Successful movements take on a cloying “fill me with your knowledge” cast. A perpetual newbie state where new adherents come on faster than existing practitioners have opportunity to develop experience and wisdom.

I really don’t need to have another conversation about how to phrase the first sentence of a user story.

And I definitely feel some of the same heat rising from the attention to lean. Buzz, buzz, kanban, buzz…

But can we blame the thought leaders, the coaches, the industry deriving wealth from a movement for the failings of that movement? Is it the corrupting influence of success or rather broad adoption itself?

I think the latter.

First, let me acknowledge that iterative improvement is a lengthy process and has to start somewhere. That your current state is entirely flawed is a given.

You don’t have to be excellent at what you do at this exact moment to begin improving your own practice and your workplace. A broad swath of not soul killing workplaces is at least as valuable as a small set of shining cities on the hill.

But whatever the starting point, taking on agile practice is dedicating yourself to a mission of fundamentally changing the nature of our work to something both disciplined and highly accountable but also collaborative, creative and sustainable.

And broad adoption means that a lot of people who call themselves “agile” just don’t rise up or even aspire to rise up to that mission.

If you claim you’re doing XP: have 24 hour builds, the developers all work solo and your test coverage is 10% then you’re either at the start of a very long journey (which I deeply admire you for) or you’re lying to yourself and you just plain suck.

If you claim you’re doing Scrum and the developers haven’t talked to a business person in months, can’t articulate what your team achieved in the last month, and you require “stabilization” sprints before you can deploy working code, you are either at the start of a very long journey (which, again, I deeply admire you for) or you’re lying to yourself and you just plain suck.

It’s not one of a thousand consultancies, professional coaches, certification tracks, associations or conferences fault if you suck.

It’s not Jeff Sutherland’s or Ken Schwaber’s fault if you suck. As a matter of fact, they and their peers have done a great deal to give a great many of us a chance at sucking less.

The mentor/mentee relationship is powerful but it’s up to each of us to do our work with courage, integrity and passion. It’s up to each of us to hold our peers to a standard of competence and care.

“The fault, dear Brutus, is not in our stars, But in ourselves, that we are underlings.”

Estimates and the desire to be lied to

A manager who does not accept estimates with range for uncertainty demonstrates a desire to be lied to.

A mundane corporate drama

We’re at a staff meeting of a company reliant upon but not about software.

Halfway through the hour long meeting, they start talking about a new software dependent product. The product has been a bullet point and a few sentences for months but now has some artwork and wireframes.

After a general conversation, the sponsor turns to the development team, “How much work is this?”

The team makes an educated guess – whether on the spot or after some closed room review. They express this as:

  • a rough sense of effort (easy, hard)
  • analogous to some other project or effort (at least as much work as)
  • a broad timeframe (month, quarter, week)
  • a range (it will take 2-4 days/months)
  • The team also describes some of the risks and uncertainties in general terms.

“Yes, but on what date will it be done? How much will it cost?”

***

What is an estimate?

A prediction made at a point in time with best available knowledge.

An estimate is fraught with more or less uncertainty. Early on an estimate has a very high degree of uncertainty — as high as an order of magnitude. As work proceeds depending on the risks involved, an estimate can become more accurate.

An estimate is subject to risk tied to the nature of the work, the organization, and/or external conditions.

Formal estimates (intended for consumption outside the team itself) should be expressed at an appropriate precision for the point in time they are made and with an appropriate range of error. Formal estimates should always include some analysis of the risks that might affect that estimate.

In our drama, the range of error is wildly large.

  • What is being built has never been built before — or — if it has, not by this particular team.
  • The build requires new learning and original problem solving.
  • The product launch is dependent upon third party tools, existing systems or data.
  • The ‘team’ doesn’t represent all the skills and access required to complete the software.

More fundamentally:

The sponsor doesn’t know what they want and to the degree they think they know what they want it will change during the course of the project.

And, even more fundamentally:

The sponsor isn’t the end user and doesn’t exactly know what those end users will find useful or desirable.

***

An estimate is a planning tool. Not a bid. Not a commitment.

A bid is a proposal to do a certain amount of work for a certain cost. It may or may not lock in a minimal required delivery. If so, it is managed as a revenue stream and losses against that stream are made up by other streams. A certain overhead for profit is planned in. A contract formalizes this as a business arrangement.

A commitment is a determination by a group of people to meet a certain goal. It is a real commitment if people make it themselves, there’s a clear and demonstrable definition of completion, if they believe it is achievable, if it is achievable, and if achieving it actually makes a difference for the individuals involved and for their company.

***

“Yes, but on what date will it be done? How much will it cost?”

“Give yourself a margin for safety – say 10% to 15%. I want a date I can trust.”

***

Why do estimates become a contract?

  • Companies lock in resources and budgets months before any work is under way.
  • Managers are under pressure to provide hard dates and costs to their bosses.
  • Managers may be used to working with outside vendors on a fixed cost basis.
  • Managers may be experienced in another kind of business with well-established, repeatable processes.

When people are under pressure they embrace certainty – even when they know none exists. Short of that certainty they want to know their staff cares, will work hard and is committed to delivering for them.

And they want the team to demonstrate their commitment by making a commitment.

But if the manager is capable of trust and new learning, has good intentions, and is focused on achieving their goals a team can turn this dysfunction around over time.

If not, seriously, look for another job because this one will cause you pain and stress. It will likely waste your hard work. It will rob you of joy and it may drive you from the software field altogether.

Rather than refuse to commit or make a bullshit commitment, commit to working toward your mutual success:

  • Communicate an understanding of what a sponsor is trying to acheive.
  • Demonstrate through your actions that you are working to help them achieve it.
  • Acknowledge their need to make schedule and budget commitments to the degree they really need to do this
  • Provide your estimates but be honest about the risks and uncertainties.
  • Attack the idea of “done”. The software may (should) be releasable before the full scope is built and it will require work after it is released.
  • Challenge and test assumptions about what features/implementations are needed.
  • Demand the sponsor’s participation in negotiating scope all along the process.
  • Make your progress visible both in tracking metrics against the goal and in demonstrating actual working code. Revise your projections as you go based on better information.

And don’t just build what your told. Build what people need. Waste as little as you can to get there.

Agile, Scrum and 4DX

Our company is adopting a practice called “The 4 Disciplines of Execution” (4DX). Superficially, it’s goals and techniques are very Scrum-like with a focus on rallying self-directed teams toward a single, important company goal.

Team Scoreboard

Layout of our combined Scrum/4DX team scoreboard

Like Agile, it champions practices but emphasizes values that give those practices reasons for being. 4DX is more focused on outcomes than specifics of performance – which is a nice puzzle piece to slot along side into a lean or XP discipline.

I’m fascinated to watch the course of a top down process change. I’m also wary of what happens when groups that don’t feel an urgent need for change or share the values of team, collaboration, and focus bang their heads into this particular demand.

Still, I am hopeful and excited. After a few years at my company once again championing a team-up Agile adoption, I feel tremors as the rest of the company shifts to meet us.

Part of being lucky is being ready.

Negative perceptions about software development. Do you have a solution?

Feedback on my proposed session at Agile 2012 on whether principled Agile practice is capable of creating workplaces and an industry more inviting of women software developers…

I could not agree more. There are many negative perceptions about software development these day in the US (off-shoring, hostile env, long hours, …). As a result, my friends at North Carolina State University tell me that overall CS enrollment is down. There was a similar event in Japan with the creation of the “Software Factory” in early 80s. I believe that they almost decimated their software industry. But how do we solve this? It has to start early as the career begins way before their first job. Leadership? If we want to maintain the industry, you bet.

Agile thought leaders came together in the first place to challenge the rest of us to empower individual contributors, elevate the role of craft and quality, cultivate collaborative ways of working, and create better, more valuable software products.

Following their lead, principled Agile practice is a determined process of honest observation and incremental improvement. It is dedicated, courageous advocacy for removing obstacles — an effort supported by analytics and a track record of improved performance.

The ambitions of this change don’t stop at a team or a set of engineering practices (though those are hard enough to accomplish). It is a change program within an organization.

We should see our mission is aligned with creating a software workplace and definition of the software developer inviting to articulate people, with diverse interests and points of views, who reflect our actual end users, and who want careers that have meaning and purpose.

I’ve never participated an agile adoption that didn’t ultimately set its sites on the larger company, the products that organization is building and why it is building them. I’ve never been part of a prolonged and dedicated agile adoption that didn’t bring developers closer to creative people outside the team, that didn’t make work a more rewarding place to show up each day.

Agile practitioners need to battle workplace cultures that discourage women and other talented people from entering and remaining in our field one dysfunction, one bully, one obstacle at a time, in one workplace at a time, because they are obstacles to collaboration and trust, disempower and burn out talented individual contributors, and distance us from our customers and end users.

I’m not saying this happens everywhere or that the changes are permanent. Widespread adoption brings with it mixed and often disappointing results. But enough of us need to drive for this change in enough of our shops, enough of the time that Agile remains a path to excellence for those of us capable of striving after it.

And by doing this we will create enough change to influence the rest of the industry. Agile adoption itself is an example of this kind of change.

Does this effort provide a clear path to success? Clearly not.

But is this approach capable of driving large and dramatic changes in companies and our industry? Yes.

How values create change from small networks to large

These are notes from my presentation at the Hawaii International Conference on System Sciences (HICSS) #45.

I’ll link to my full paper when it is available and to subsequent posts as I publish them.

Agile values, product innovation and the shortage of women software developers Part 7 of 7


(44) Agile values in an enterprise context

I’ve described two examples of how Agile principles call upon practitioners to battle hostile workplaces. My paper has several more. But let’s talk about how Agile teams instill Agile values into the enterprise. As a development team matures impediments become consistently rooted in the surrounding organization. Continuous improvement becomes an effort directed out into the larger company. Where an organization fails to support a team adopting an agile practice, the teams needs to drive for these changes in the organization by first building trust and influence by producing results in spite of their impediments and then using that success to win support for removing the obstacles that lay in their path.

(45) A principled Agile enterprise

In response the larger organization will begin removing impediments to team performance by, for example, adopting a retrospective type review process, rewarding collective over individual performance, compensating for span of influence over span of control.

(46) How values create change from small networks to large

But how can small change within companies produce large order changes across an industry or society?

(46) Ba

To model this, I’ll use Nonaka’s concept of Ba, or “a shared context in motion, in which knowledge is shared, created and utilized[65].” Sectors that thrive off innovation do so by sharing knowledge across direct and extended-relationships among people. Each set of relationships exists within a physical or virtual space. Each of these spaces at any given moment in time is Ba.

(47) Ba in knowledge work

Knowledge workers interact within their local communities, interest groups. They graduate from school and change jobs. Companies are distributed across locales. Consultants travel among companies and conferences bring individuals together from across the industry. In sharing, creating and synthesizing knowledge one Ba influences the other, fostering change on the small scale to the large and back. The broad adoption of Agile practices is itself an example of knowledge occurring first within individuals and teams and then spreading across an industry.

(48) The challenge

But widespread Agile adoption has been a mixed blessing for principled agilists. Agile values are not permeating as well as the practices themselves. To invert Alistair Cockburn’s dictum, the industry is valuing agile practices over agile principles.

(49) Snowbird

This threat is on the minds of prominent Agile thought leaders. Enough so that the notes from the 10 year reunion of the initials signers of the Agile Manifesto contains “four things the community needs to do in the next 10 years”: demand technical excellence, promote individual change and lead organizational change, organize knowledge and improve education, and maximize value across the entire process[66].

(50) Conclusion

Agile is not about doing “Agile” things. It is about continually improving ourselves, our teams and our organizations to create better software for our customers and our end users. If we embrace that on a wide scale, we will recognize we are driving away an incredibly valuable source of talent and an incredibly valuable contribution in our effort to create products relevant to over half of our end users. We can use the principles underlying Agile practice to guide our efforts to remove this impediment.. Successful embrace of agile principles within teams will instill a more social and engaged view of the software developer role that can shift companies and the larger industry, driving beneficial change into academic institutions and the perceptions of the greater public. This change in our workplaces, in the common perception of our work, and in the institutions that educate software developers would encourage more girls to pursue computer science and help the industry recruit and retain larger numbers of talented women.

Thank you.


All slides.

Previous: Antidote the diving catch culture of heroics and privileged roles

There is abundant research on the problems women face in our field. I would love researchers to jump in on whether Agile principles and Agile practioners can really make a difference.

I’d also love any suggestions of organizations, institutions and individuals I might reach out to for more information, collaboration, or to take up the cause.

Please comment on my proposal to Agile 2012.

The full citation list for my paper.