Kelly Weyrauch Agile Quality Systems LLC 2015 Opinion-Based Metrics Easy to Collect & Powerful Drivers of Process Improvement

Kelly WeyrauchAgile Quality Systems LLC2015

Opinion-Based Metrics Easy to Collect &

Powerful Drivers of Process Improvement

? Metrics are an important element to a Process Improvement Initiative• Agree• Somewhat Agree• Somewhat Disagree• Disagree

What went through you mind when I asked you to state your opinion?

What did we learn about this crowd from the answers?

Reason for Metrics

Agile Quality Systems 2015

? Metrics are an important element to a Process Improvement Initiative

Why did I ask? Why was a subjective question like this sufficient? Wanted to know if we all agreed

If so, I don’t need any more information If not, I would need to ask further questions

Sometimes, an opinion is all you need

Reason for Metrics


? My organization has good Metrics to help us in our Process Improvement Initiatives• Agree• Somewhat Agree• Somewhat Disagree• Disagree

What did you consider to form your opinion? What did we learn about this crowd from the

answers?

Metrics Programs


? My organization has good Metrics to help us in our Process Improvement Initiatives

Why did I ask? I want you to think about Metrics from your perspective Wanted to know if we all agreed or differed

Perspective is key to Subjective Questions

Metrics Programs


Benefits of Process Improvement


Productivity Predictability Team Health

Quality

? What do you use to measure Quality?

? What’s good about those?? What’s not so good?

Quality Metrics


? What do you use to measure Productivity?


Productivity Metrics


? What do you use to measure Predictability?


Predictability Metrics


? What do you use to measure Team Health?


Team Health Metrics


Quality Productivity Predictability Team Health

? We are good at ___• Agree• Somewhat Agree• Somewhat Disagree• Disagree




? We are good at ___

What would it mean if: We all strongly agreed we were good at it? We all strongly disagreed?



“WE” = The entire team, not individualsNeed to scope the survey – define who the “we” is

“ARE” = Where we are right nowSo we can decide whether a change is warranted

“GOOD” = What we want to be

Team owns the metricTeam owns the action to improve it

Power of the question:“We are good at ___”



? My organization is good at ___

Good categories, but too broad to be actionable



Benefits


Productivity Predictability Team Health

Quality

Benefits


Quality Productivity

Predictability

Team Health

• Customer facing• Meets customer expectations of

“good”• Does what they want / need• Bug free• …

• Internal facing• Reliable / stable / maintainable• Find bugs early• Confident in assessing quality• …

• …

Benefits


Quality

Productivity Predictability

Team Health

Productivity = Value / Cost

• Value• Deliver high value to customer

(features / functionality)• Deliver high value to business

(deliverables meeting requirements of process/phases/business)

• Prioritize well• …

• Cost• Efficient (lean)• Fast• Right amount of rework• …

• …

Benefits


Quality

Productivity

Predictability Team Health

Predictability = Making good commitments, meeting them with reasonable adjustments

• Awareness• Visibility• Feedback

• Control• Reality-based estimation and

planning• Manage all levers of project

management• Reasonable responses to change

(new needs / problems)

• …

Benefits


Quality

Productivity

Predictability

Team Health

Team Health = The soft benefits that lead to more tangible ones

• See my impact on the project• Accountable & Empowered• Aligning skills & interests• Enjoyment• Teamwork (intra-team, across

teams)• …

Defining the topics for a survey forces a definition of, or even a recognition of, what matters to you

Asking about those topics tells everyone what mattersProviding focus to process-improvement effortsProvides motivation - “Want to move the dial”

Choosing the Topics for a Subjective Survey


Establishes what matters to us, what we care about being good at

Provides a baseline for improvement Motivates and empowers us to improve Shows whether we improved

Benefits Survey


For each of the Topics, two questions “We are good at ___” “Our practices are helping with ___”Are we set up for success, or are we

hindered by our processes / practices / behaviors / …

Adds information on another factor for determining the path for improvement if we aren’t where we want to be

Survey Structure


Six-point scale for answer Strongly Disagree Disagree Slightly Disagree Slightly Agree Agree Strongly Agree

“Agreement” – your opinionEncourages thought to explain/defend the opinion

Forces a choice– no middle groundEnough choices to differentiate, not too many to

overwhelm or quibble

Survey Structure


An Example - Quality


The product quality is high.

Our practices help us get better quality.

We satisfy our customers.

Our practices help us satisfy our customers.

We deliver bug-free software.

Our practices help us deliver bug-free software.

We deliver safe and effective software.

Our practices help us deliver safe and effective software.

We deliver software that is reliable, stable, maintainable.

Our practices help us deliver software that is reliable, stable, maintainable.

We find bugs early enough.

Our practices help us find bugs earlier.

We are confident in assessing quality.

Our practices increase our confidence in assessing quality.

Strongly Disagree Disagree Slightly Disagree Slightly Agree Agree Strongly Agree

4.5

4.2

4.4

3.7

3.5

3.8

4.4

4.2

4.4

3.8

3.5

3.5

3.9

3.7

4.0

Productivity – Team


Our productivity is high.

Our practices help us increase productivity.

We deliver high value to our customers.

Our practices help us deliver high value to our customers.

We deliver high value to the business.

Our practices help us deliver high value to the business.

We prioritize well.

Our practices help us prioritize our work.

We are efficient.

Our practices help us be efficient.

We are fast.

Our practices help us be faster.

We do the right amount of rework.

Our practices help us optimize the amount of rework.


3.9

3.8

4.2

3.8

4.7

4.1

3.2

3.4

3.2

3.5

3.2

3.3

3.2

3.4

3.6

Productivity – Management


Our productivity is high.

Our practices help us increase productivity.

We deliver high value to our customers.

Our practices help us deliver high value to our customers.

We deliver high value to the business.

Our practices help us deliver high value to the business.

We prioritize well.

Our practices help us prioritize our work.

We are efficient.

Our practices help us be efficient.

We are fast.

Our practices help us be faster.

We do the right amount of rework.

Our practices help us optimize the amount of rework.


2.9

2.3

4.6

3.7

4.0

3.4

3.2

2.8

2.6

2.4

2.2

2.1

2.6

2.1

2.9

An Example - Predictability


Our projects are predictable.

Our practices help us be predictable.

Project progress is visible.

Our practices help us make project progress visible.

We gather and make use of feedback.

Our practices help us gather and make use of feedback.

We are good at estimating.

Our practices help us make better estimates.

We manage all levers of project management.

Our practices help us manage all levers of project management.

We respond reasonably to changes.

Our practices help us respond reasonably to changes.


3.1

3.2

3.2

3.5

3.6

3.3

2.2

2.8

2.2

2.4

4.2

3.8

3.1

An Example – Team Health


Our team is healthy.

Our practices help us improve the health of the team.

I see my impact on the project.

Our practices help me see my impact on the project.

I am accountable and empowered to take action.

Our practices help me be accountable and empowered to take action.

I can align my skills and interests with the needs of the team.

Our practices help me align my skills and interests with the needs of the team.

I enjoy what I work on.

Our practices help me enjoy what I work on.

My team works well together.

Our practices help my team work well together.

My team works well with other teams.

Our practices help my team work well with other teams.


4.6

4.6

4.4

4.1

4.6

3.9

4.6

4.3

4.8

3.8

4.9

4.4

4.1

3.5

4.3

Summarization:Radar Graph


PRODUCT QUALITY

PRODUCTIVITY

PREDICTABILITY

TEAM HEALTH

Benefits Survey

Initial

Predictability Comparison


1.0 2.0 3.0 4.0 5.0 6.0 1.0

2.0

3.0

4.0

5.0

6.0 Agile Benefits - PREDICTABILITY

Our projects are predictable.

Project progress is visible.

We gather and make use of feedback.

We are good at estimating.

We manage all levers of project management.

We respond reasonably to changes.

We are good at ___

Pra

cti

ce

s a

re h

elp

ing

wit

h _

__

_

? “Subjective questions are useful”• Agree• Somewhat Agree• Somewhat Disagree• Disagree

Subjective, and that’s a good thingThe opinion of the team mattersThe opinion of stakeholders mattersBetter if supported by objective evidence

Subjective Questions


The “How Do You KNOW…” story Do you trust the opinion of the team? Are they qualified to have such an opinion? If the team is to own process improvement, they

must also own the opinion on the need for it Is the team’s opinion consistent with:

Other metrics? Reality? (Who defines “reality”?) Stakeholders (yours, management, customers, …)?

How Do You “KNOW”?


? “Surveys must be anonymous”• Agree• Somewhat Agree• Somewhat Disagree• Disagree

? Why? Why Not? (columns on whiteboard)Surveys are made to be anonymous when:

There is a perceived “right or wrong” answerUndesired consequence comes from answering wrong

To the survey taker, or someone they care aboutUndesired action after the survey

Need to be anonymous suggests mistrust

Anonymous Surveys


Surveys described here must not be anonymousTeam should be comfortable sharing honest opinions

within the team and with stakeholdersTeam must own their opinions

And the desire to make improvementsWant to identify & explore different perspectivesWant to evaluate changes over time

Anonymous Surveys


How much do you care about this topic? 5 = My reason for living, the real #1 on a priority list 4 = A top priority, but accepting there are some ahead

of it 3 = This matters, but it’s not a huge thing 2 = “Would be nice to be good at this, but….” 1 = This doesn’t matter to me, but it’s okay if it matters

to someone else 0 = This doesn’t matter to me, and it shouldn’t matter

to anyone else either

Additional Survey Questions:Interest Level, Importance


A variation of “how much do you care” could be “how good do you want to be” 5 = I want us to be excellent at this (A Student) 4 = I want us to be good at this (B Student) 3 = I want to be good at this, but not right now 2 = There is some good and some bad in this 1 = I think this will hurt us, so if it matters to someone

else, keep it away from us 0 = I don’t want us to care about this at all, it would

hurt us to be good at this

Additional Survey Questions:Interest Level, Importance


How strongly do you believe your opinion is right, reasonable, defendable?

5 = Adamant. Can make a strong/complete case that would convince most. Either have strong supporting

evidence, or such an easy case that you don’t need evidence. Can easily explain why, and would not be swayed by differing opinions or perspectives.

4 = Certain, but… Strong from what you can see, but possibly incomplete. As strong as #5 from your perspective, but

you accept that there could be other perspectives with differing opinions.

3 = Convinced, but hard time convincing others. You have a strong belief, but would have a hard time explaining it to others, and would change your

opinion if faced with a strong counter argument.

2 = Lots of holes in the case, but a reasonable “gut feel” about it Would readily accept a disagreeing opinion, would be easily swayed

1 = Nothing to back it up, it’s just what I think. 0 = Don’t count my vote, my opinion is meaningless.

Additional Survey Questions:Strength of Opinion


Data-Driven Metrics can be hard, but Subjective Surveys are easy to do

Opinions may be all you need Discussions of the opinions has huge value Combine Subjective Surveys with other “hard

data” to tell a more complete story

Summary


Kelly Weyrauch [email protected] 763-688-0980

www.AgileQualitySystems.com

Connect with me on LinkedIn

The End


mailto:[email protected]

http://www.AgileQualitySystems.com/

Documents

Kelly Weyrauch Agile Quality Systems LLC 2015 Opinion-Based Metrics Easy to Collect & Powerful Drivers of Process Improvement