Continuous Process Improvement — Deliver Less Code at Better Quality to Achieve More

In my last post I claimed we rebuilt our website properties with with half to one third the developers and with “about half 65%” the resulting code. All to justify my claim that “You accomplish more by doing less.”

Now, I’ll back that claim with some metrics.

Lines of code

There are a number of tools for measuring ruby code size and complexity. The metric_fu gem does a good job a aggregating them.

But because our retired rails application is six years old, I resorted to the perl based cloc to count lines of code. I excluded standard libraries and counted code in the app and lib directories supporting websites common to both codebases.

Old Application Lines of code (LOC) New Application Lines of code (LOC)
Websites 39,981 Websites 27,071
Admin websites 15,011 Data ingest/processing app 6,176
Shared library 2,716
Total Old 54,992 Total New 35,963

Our current codebase is 65% the size of our original*.

Code size is a predictor of defect count and ongoing maintenance. That our current codebase is 35% more efficient at delivering roughly the same perceived value implies it is materially less expensive to maintain.

Take from this that lines of code is a terrible measure of productivity.

Less code is not our only indicator of the health

As measured by rake stat, we now have 3.6 lines of test code for every line of production code. Our old codebase measures 0.4. Our new applications have 9 lines of test for every one in our old.

Test lines of code is not a measure of test coverage or quality but remember I also work in this code. I know from direct experience that the new tests are more thoughtful — less fragile and opaque — better at testing real business logic with real data. We have invested in improving our test practice.

Better tests and better test coverage allows developers to take more initiative in changing and optimizing code. We can turn around features faster and we can improve our code with less fear of breakage. It’s a virtuous circle.

Again, we rebuilt our corporate websites with half to one third the development resource, 35% less code and better test coverage. We did this over a similar time frame in one week increments without schedule pressure or extraordinary efforts.

For the product and executive managers out there

Spiral Staircase

Managers want predictable delivery in short cycles. But that is an outcome. You achieve that outcome by reducing waste and building with quality.

If you don’t make quality your priority, your developers — out of loyalty and best intentions — will not build with quality. And that leads to larger, less maintainable code, longer and less predictable delivery cycles. The opposite of what you want.

Allow developers their due as craftspeople and professionals. Instead of banging the drum on delivery and delegating responsibility for quality, commit time and attention to building with quality and your team will deliver.

Next time, I’ll dive into how and why our practices have changed over time…

When I first measured this it was 50%. In the meantime, we’ve added features, services and a new source data feed. Still 5% of our current codebase is localized terms of use and privacy policies that didn’t exist in our old website. So, growth of the codebase is something to watch but I’m not sweating it yet.

ProgressVisualizer – visual reporting for your Trello boards

ProgressVisualizerI’m working on a pet project I’m calling, for now, ProgressVisualizer. It is intended to be a quick and easy way to create visual reporting for Trello boards.

Trello is the first tracking tool my team finds so easy to use that we’ve largely stopped using our physical cork board. It offers simplicity, openness and doesn’t impose unnecessary assumptions. It offers an API that lets you interact with your data and webhooks which allow you to do so in near real time.

I’ve tried to follow Trello’s lead and make ProgressVisualizer easy to use. It is a tool intended for leads and managers who want burn ups, reports and other charts, don’t want to spend alot of time crafting reports for themselves, and believe they should impose as little overhead on their teams as possible.

ProgressVisualizer sample charts and reports

ProgressVisualizer sample charts and reports

ProgressVisualizer is a nights and weekends project. I don’t know what it will become. That said, it’s more than a prototype. I respect your privacy, user profile information is encrypted, and I have provided the ability to export and destroy any data collected by the site.

Right now, I’m gathering feedback and suggestions, so if you use Trello, please try ProgressVisualizer and let me know what you think.

Dear agile consultant – hiring criteria for coaches and mentors

play at your own riskDear consultant who uses “agile” as part of your brand,

I am a possible customer who embraces agile principles, practices agile techniques and has done so for years.

I admire the tenacity and talent it takes to build a business in consulting.

The best of you offer your clients techniques, language and a way of thinking that can better lives and turn around companies: advising them in ways to change their own organizations, bolstering their courage to experiment, helping them learn to iteratively learn, inspiring them to foster teams, empower individual contributors, incrementally improve, and helping them grow past dependency on your services.

Here’s how I judge whether a thought leader is a worthy mentor:

  • Your reputation among those of your peers I know and respect.
  • Do you provide context/share credit: “This is where it originated, this is who popularized or researched it, this is who’s written well on it. This is my version of it.”
  • Do you espouse principles: “This is about defining work that builds equity because it offers real benefit to end users and a way of working that entrusts individual contributors of diverse talents to collaborate within teams to deliver that benefit.”
  • Are you frank: “Most companies who try this fail. Most managers who try this will not change their own behavior enough to allow their teams to succeed for them.”
  • Do you embody the principles: “It is about your team(s). Not me. I do not have right answers only experience, a willingness to listen, and techniques to help us figure out what you need to do to help yourselves.”

Conversely, these are the bad smells:

  • Celebrating the widespread adoption of “Agile” without acknowledging most “agile” adoptions are crap.
  • Celebrating scale not individual team excellence.
  • Focusing on techniques not principles as if “stories” and “iterations” were magic.
  • Talking about software tools before disciplined engineering practice.
  • Talking about “value” and “productivity” as if a leader’s understanding of these terms were not a/the major obstacle to their workers ability to perform.
  • Jamming agile practices into a contradictory way of thinking: “Agile process manager” anyone?
  • Coining new jargon for a slight spin on existing practices: It looks like a timebox, smells like a timebox, tastes like a timebox. “I call it creato-inno-rations™.”
  • Putting yourself above the problem: Just because you were really good when you practiced doesn’t mean you are a brilliant coach. Just because you’re a brilliant coach doesn’t mean you can do my job better than me. Just because you can do my job better than me doesn’t mean paying you to not do my job provides my company value.
  • Overheated claims of personal invention/Not giving credit to others. Sorry guys (and I mean guys), Mary Poppendieck was talking kanban and software development fifteen years ago. You’ve advanced the craft, you’re changing minds, and you may be very good at what you do but you are not Archimedes.

As a potential customer, I need your honest criticism, I am impressed by your ability to learn from others. I respect determination and humility more than bravado.

Give credit where credit is due and do exceptionally well.

Estimates and the desire to be lied to

A manager who does not accept estimates with range for uncertainty demonstrates a desire to be lied to.

A mundane corporate drama

We’re at a staff meeting of a company reliant upon but not about software.

Halfway through the hour long meeting, they start talking about a new software dependent product. The product has been a bullet point and a few sentences for months but now has some artwork and wireframes.

After a general conversation, the sponsor turns to the development team, “How much work is this?”

The team makes an educated guess – whether on the spot or after some closed room review. They express this as:

  • a rough sense of effort (easy, hard)
  • analogous to some other project or effort (at least as much work as)
  • a broad timeframe (month, quarter, week)
  • a range (it will take 2-4 days/months)
  • The team also describes some of the risks and uncertainties in general terms.

“Yes, but on what date will it be done? How much will it cost?”

***

What is an estimate?

A prediction made at a point in time with best available knowledge.

An estimate is fraught with more or less uncertainty. Early on an estimate has a very high degree of uncertainty — as high as an order of magnitude. As work proceeds depending on the risks involved, an estimate can become more accurate.

An estimate is subject to risk tied to the nature of the work, the organization, and/or external conditions.

Formal estimates (intended for consumption outside the team itself) should be expressed at an appropriate precision for the point in time they are made and with an appropriate range of error. Formal estimates should always include some analysis of the risks that might affect that estimate.

In our drama, the range of error is wildly large.

  • What is being built has never been built before — or — if it has, not by this particular team.
  • The build requires new learning and original problem solving.
  • The product launch is dependent upon third party tools, existing systems or data.
  • The ‘team’ doesn’t represent all the skills and access required to complete the software.

More fundamentally:

The sponsor doesn’t know what they want and to the degree they think they know what they want it will change during the course of the project.

And, even more fundamentally:

The sponsor isn’t the end user and doesn’t exactly know what those end users will find useful or desirable.

***

An estimate is a planning tool. Not a bid. Not a commitment.

A bid is a proposal to do a certain amount of work for a certain cost. It may or may not lock in a minimal required delivery. If so, it is managed as a revenue stream and losses against that stream are made up by other streams. A certain overhead for profit is planned in. A contract formalizes this as a business arrangement.

A commitment is a determination by a group of people to meet a certain goal. It is a real commitment if people make it themselves, there’s a clear and demonstrable definition of completion, if they believe it is achievable, if it is achievable, and if achieving it actually makes a difference for the individuals involved and for their company.

***

“Yes, but on what date will it be done? How much will it cost?”

“Give yourself a margin for safety – say 10% to 15%. I want a date I can trust.”

***

Why do estimates become a contract?

  • Companies lock in resources and budgets months before any work is under way.
  • Managers are under pressure to provide hard dates and costs to their bosses.
  • Managers may be used to working with outside vendors on a fixed cost basis.
  • Managers may be experienced in another kind of business with well-established, repeatable processes.

When people are under pressure they embrace certainty – even when they know none exists. Short of that certainty they want to know their staff cares, will work hard and is committed to delivering for them.

And they want the team to demonstrate their commitment by making a commitment.

But if the manager is capable of trust and new learning, has good intentions, and is focused on achieving their goals a team can turn this dysfunction around over time.

If not, seriously, look for another job because this one will cause you pain and stress. It will likely waste your hard work. It will rob you of joy and it may drive you from the software field altogether.

Rather than refuse to commit or make a bullshit commitment, commit to working toward your mutual success:

  • Communicate an understanding of what a sponsor is trying to acheive.
  • Demonstrate through your actions that you are working to help them achieve it.
  • Acknowledge their need to make schedule and budget commitments to the degree they really need to do this
  • Provide your estimates but be honest about the risks and uncertainties.
  • Attack the idea of “done”. The software may (should) be releasable before the full scope is built and it will require work after it is released.
  • Challenge and test assumptions about what features/implementations are needed.
  • Demand the sponsor’s participation in negotiating scope all along the process.
  • Make your progress visible both in tracking metrics against the goal and in demonstrating actual working code. Revise your projections as you go based on better information.

And don’t just build what your told. Build what people need. Waste as little as you can to get there.

The Functional Manager in Agile

Home Farm by Hellsgeriatric, on flickrTeam managers should till the soil with their teams.

Anything else is waste and waste must be rooted out.

Still it is hard.

Luke Melia wrote about how he performed as functional manager and dedicated 75% of his time pairing.

There are two tremendous challenges with this.

The first is limiting distractions in order to remain a reliable contributor.

Luke has tremendous reserves of focus and enthusiasm. As his manager, I did everything I could with our scrum master, Salim Divakaran, to support him, remove distractions and share workload.

The second challenge is being both the boss and a peer.

Luke recruited most of the team, he held weekly one on ones with each person, he insisted on unvarnished feedback, and is worthy of respect as both a peer and a manager.

So, here is the pattern: An experienced coach with people skills and authority over development practices pairing in with the developers. An experienced scrum master. Functional management residing in one or the other or divided up in some sensible and easily described way among the two of them.

This enables direct participation in the work, management attention to the team, and strategic contribution to the rest of the company.