Beyond the Numbers: Five Questions to Ask About Metrics


Have you ever gotten that request, fly-by your desk, stopped in the hallway, email, or chat, that looked something like this?

"I need to know how many points that one team is getting done, by the end of the day.”

or

"Can you tell me how many story points each team member is signing up for each sprint?"


All too often the focus in Scrum turns to METRICS. Everyone seemingly loves to talk about metrics. I always find it fascinating to understand exactly what someone is really looking to do with this information. During my classes and workshops, I offer up a set of questions that you can ask someone who comes to you with those requests for metrics.

Whether it’s a misunderstanding of Story Points, number of PBI, or another metric, there is often a longer discussion to be had about what the request is, and more importantly who it’s coming from and why. This discussion is valuable to get to the core of not just the metrics numbers, but perhaps larger organizational concerns that an agile leader should address.

Instead of dropping everything and getting that individual EXACTLY what they need, start the conversation by using these five questions.

I am NOT advocating using these questions in a belligerent, argumentative, or troublesome nature. Rather, use them from a curious place. Ask, inquire, be curious, and then try to offer solutions and helpful support to get to the root of things and an agile-minded way.

 

Question #1 - What question are you trying to answer?

Too often it starts here... The request begins with "I need this... Because I need to know...." – Use this to to identify what they are really trying to learn about, since it likely will go deeper than what they are asking for on the surface. If that doesn’t give you any insight, keep going….

 

Question #2 - What problem are you trying to solve?

This could range ALL over the place, from identifying where to spend time, energy, money, etc. - Is there actually a problem that's out there? "I just need to know" isn't a real strong reason to whip everyone up over story points or some urgent request for information. Maybe this is a dead end too, then try the next question….

 

Question #3 - What decision are you trying to make?

People are often faced with a choice between differing alternatives, strategies, tactics, etc. - explore the decision that is on the table that the metrics request is intended to help solve. If there’s a decision to be made, keep the focus there and how you can help get them additional information that can help, but you need more clarity to do that.

At this point, after you have tried all three of these questions, there’s a chance that whomever has this time sensitive request will drop “Oh, well it’s not really me who needs to know” and that’s your window to ask the NEXT question…

 

Question #4 - Who is asking?

I end up here a good bit of time because the person saying "I need this" very likely can come back with "Well it's not for me... It's for (insert person's name)" - Find out who that person is, get a better idea of where the request is coming from, and then go for the follow up …

 

Question #5 - What does success look like for that person?

How does this person know they are having a good day / week / quarter / year, and how does this request for data impact that success? - HUGE insights once you can chip away at this!

By getting to this point, this is where you can uncover, for example, that someone gets a big bonus if the aggregate velocity numbers for the enterprise fall within a certain compliance range triggering a clause in someone’s compensation package. Think that might cause some urgency, and impact behavior in a potentially negative way?

Metrics often cause people to alter their behavior, depending on what success marks they are trying to achieve. This reminds me of two quotes from a famous documentary with respect to metrics in the field of military battle:

 

"The problem….as it often is, are the metrics. It is a situation where if you CAN’T count what’s important, you make what you CAN count important."

– James Willbanks (Army Advisor)

 

"…you don’t get details...you get numbers, and the numbers are lies, most of them. If (this) is your success mark, then you are pushing otherwise honorable men….to become liars."

– Joe Galloway (Journalist)

 

Imagine an organization that comes up with some potentially misleading success mark and consider the possibility that behaviors may change in an unintended way to meet that metric. We see these metrics, and resulting behaviors, all over the software world, often pitting team members against each other, or sacrificing value for “credit.”

If you can't measure what's important, what you CAN measure becomes important.

Think about that next time everyone gets all spun up over metrics. Rather than focusing on just the number, think about the behaviors and dynamics that lead to it, and determine if they follow the spirit of what you are trying to achieve. Thanks for reading!