Transcript

How to measure and improve developer productivity | Nicole Forsgren (Microsoft Research, GitHub, Google)

Notes

• So here we go. Draw four boxes on a piece of paper,
  • two on the top, two on the bottom.
  • So they'll be kind of aligned.
  • The first two to the left of them write the word words.
  • And below them, write the word data
  • then between the two on the top, draw an arrow between them. So it'll say words, box, arrow, box, right. Is that making sense?
• Okay. So on the top half, this is where, if you want to think about measuring something or testing something, you have to start with words.
  • So as an example, let's just say, I think that customer satisfaction gets us more money or customer satisfaction gets us return customers.
  • Let's do customer satisfaction. So the first box, you'll put customer satisfaction inside the box, and you'll put return customers in the second box.
  • Now always start with words. You do not start with data. You always start with words.
• And then you'll go around to a couple of people, stakeholders, managers, others, and you'll say, "Do you agree with this? Is this actually what we're doing?" And it can turn into a sentence. And then, in the boxes below it, this is your data.
  • How are we going to measure customer satisfaction? It could be a survey.
• And so this is where you'll go, and you'll say, "What data points do we have that could proxy for? What could be our data points for customer satisfaction?" And this is where it gets tricky. You could say, "Well, customer satisfaction could be return customers. But we think it leads to return customers, so we can't use that here." But return customers could be... So that's where you kind of roll this out. So how else would we measure customer satisfaction? I made this hard on myself.
• Okay. Now, return customers. Let's go to the next box. How are we going to measure return customers? Depending on our context, let's say that this is an online business. We could say that it's return customers as measured through the website. We could say that it's returned customers. We could just ask them, right. Maybe we have a follow-up survey. Return customers. Maybe we're going to do a stretch here. Maybe we say it's a referral link. This helps us get super clear on what it is we're going to measure. Now, the reason I like this is because if some of our... Now this data analysis. We'll just do correlations here, right.
• If we have longitudinal overtime, that's fine. You can hand this to a data scientist. You can hand this to someone and you can say, "What data do we have? Let's go run this." If something here falls apart, now you can point to the data boxes, and we can get mad about the things in the data boxes, and we can say, "What's wrong? Is the data poor quality? Are we missing data? Was this a bad proxy?" Proxy stands in for something else. "Was this ridiculous?" One of the things I made up. It was just a bad idea.