Track Success Using Evidence-Based Management

00:00:07: Welcome to Episode 17 of transform together, the Holisticon

00:00:11: podcast meetup. I'm Johannes Schartau and today I'm talking

00:00:15: to Simon Flossmann. Simon is a Professional Scrum Trainer

00:00:18: at Scrum.org and coach for product management.

00:00:21: A central topic in Simon's work is Evidence-Based Management,

00:00:25: Scrum.org's framework for helping teams and organizations make better

00:00:29: decisions based on goals, measures, experiments and actual value

00:00:33: delivered. Simon has described EBM as one of his biggest aha moments.

00:00:37: Especially because it connects goals, metrics, and real value

00:00:41: in the Scrum context. In his training, coaching, writing

00:00:44: and webinars, he uses EBM to help organizations move

00:00:48: beyond simply doing Scrum and toward more meaningful conversations

00:00:52: about customer outcomes, product value and whether their way

00:00:55: of working is actually helping them achieve their goals.

00:00:58: Simon, welcome to the podcast.

00:01:00: Hello Johannes, thanks for having me.

00:01:02: I'm looking forward to this conversation.

00:01:05: Great, I already said in the introduction that you

00:01:08: had a big aha moment with Evidence-Based Management, and

00:01:12: I'm always curious when people come here and they

00:01:15: have a specific technique or a method that they

00:01:19: use, why you know why they're so invested in

00:01:22: this and what did EBM for example clarify for

00:01:25: you that Scrum alone did not fully clarify?

00:01:28: Yeah, maybe aha moment is phrasing it a little

00:01:32: bit too nicely. It was more an embarrassing moment.

00:01:36: So when I started out after university, I basically

00:01:40: worked as a Product Owner for some time.

00:01:43: And then I later moved into product coaching.

00:01:46: I had one of the product managers working with

00:01:50: Scrum. And finally, or after some time working together,

00:01:54: I could convince him that the notion of finishing all

00:01:58: user stories or tasks underneath an epic is not

00:02:01: a sign for success. So his idea of success

00:02:04: was Simon, our goal is: if we have this

00:02:07: huge epic and if we finish all the stories

00:02:09: then we are done. And basically I could convince

00:02:13: him yeah maybe, but I think that's not the

00:02:15: definition of success. And finally he agreed on and

00:02:19: said okay Simon I understand, so finishing all the

00:02:22: work underneath an epic it's not a measure of

00:02:25: success, but what is a measure of success How do I

00:02:28: know that we should continue and what should we

00:02:32: continue. And at that moment I thought damn, I

00:02:35: have no clue myself because before it was always

00:02:38: okay we finished that epic then there was another

00:02:42: one. So basically when I was a Product Owner

00:02:45: the most of the work was driven let's say

00:02:48: by stakeholder demand or business and ideas.

00:02:51: So it was a huge pile of epics and

00:02:53: work, a huge backlog. And then with this new

00:02:56: team it was more like in the

00:02:58: up environment there was not so much stakeholders telling

00:03:02: what to do is it was more about figuring

00:03:04: out what to do. And that was a tough

00:03:06: situation for me because at that moment I had

00:03:09: no answer. I said yeah, I don't know basically.

00:03:12: And that was the starting point where I start

00:03:15: to try to understand okay, what the notion of

00:03:17: unrealized value. So what is there out there?

00:03:20: Does it make sense to continue?

00:03:23: And along that journey I discovered Evidence-Based Management.

00:03:26: I think it was maybe two thousand seventeen or something.

00:03:30: So this is what closed the gap for you,

00:03:32: something that really helped you. I imagine a lot

00:03:35: of people listening might not be familiar with Evidence-Based

00:03:39: Management itself. Could you quickly go into what this

00:03:42: is and what problem it is trying to solve?

00:03:45: Yeah, so we can tackle it from two sides.

00:03:48: So one major problem or difficult situation for Scrum

00:03:52: practitioners is this Scrum Guide mentions something like value,

00:03:56: the Product Owner or the team needs to create

00:03:59: value, but it does not really define what value

00:04:02: is. And there are certain definitions and things you

00:04:06: can do out there. And I think Evidence-Based Management

00:04:09: helps you to get a different perspectives on value.

00:04:13: So it's not just value, so there are

00:04:15: different perspectives on value. So we have current value,

00:04:19: the value we are creating at the moment, what

00:04:22: the potential is, how fast we can create value

00:04:25: and how good we are at creating value.

00:04:27: These are at least four more perspectives which product

00:04:31: people can use to understand the notion of value.

00:04:34: So I think that's the problem EBM tries to

00:04:37: solve from one perspective. And the other one is

00:04:40: I think it's a little bit from the historical

00:04:43: point of view I Ken Schwaber one of

00:04:45: Ken Schwaber, one of the co-creators of Scrum, came

00:04:49: up with the notion of Evidence-Based Management and in

00:04:52: the early days prior to Scrum when he also

00:04:55: introduced Scrum or what it was called in those

00:04:58: days to other companies, he worked a lot in

00:05:01: medtech and he was always very impressed how medicines

00:05:04: are developed and what scientific rigor and he said

00:05:08: okay we need to develop software like that because

00:05:11: at that time software well yeah just build it and then oh

00:05:15: Oh, finally we build something awesome. It's not the

00:05:18: way and it's more like the experimentation thing which

00:05:21: you also see in Scrum basically, it's the Sprint

00:05:24: Review. So we build something and then we validate

00:05:26: something. And that's the notion of Evidence-Based Management.

00:05:30: If you combine it with the notion of value

00:05:32: and with this okay, we build something and then

00:05:35: we try to ev evaluate if it's actually creates

00:05:37: value, that's Evidence-Based Management in a nutshell.

00:05:40: So we're looking at goals, which direction we want

00:05:43: to go, we try to describe what

00:05:45: value actually means. So what does progress mean?

00:05:48: And then instead of having a fixed plan or

00:05:50: a Gantt chart or an epic which we need

00:05:53: to complete, we do some kind of let's say

00:05:55: experimentation step by step and learn along the way

00:05:58: and fail along the way.

00:06:01: And it's a little bit broader than just Scrum.

00:06:04: It's applicable to all sort of areas.

00:06:07: So basically I used it for not just for

00:06:10: software development, but also for HR for instance to

00:06:14: help them create more value and achieve their goals.

00:06:18: Now a lot of organizations they already

00:06:21: use something like KPIs, OKRs, some kind

00:06:24: of dashboard, reports, controlling processes. What does

00:06:27: EBM add that those things often miss?

00:06:30: So all of those things I would say are

00:06:33: very helpful. I'm a big advocate in outcome-focused OKRs

00:06:37: for instance. But at the end of the day

00:06:39: most of them are more like yeah, don't get

00:06:42: me wrong, like step by step approaches.

00:06:45: So if you look at OKR, so have a

00:06:47: three month cycle and it has to be written

00:06:50: like that and so on. Or if you look

00:06:53: at SMART goals, it has to satisfy five conditions

00:06:56: and all these kind of things and there is

00:06:59: a place for that. EBM

00:07:00: steps a little bit more out and, thinks about

00:07:04: how goals, measures, and behaviors actually linked together.

00:07:10: you think about project, product, companies or whatever, at

00:07:14: the end of the day the job of management

00:07:17: is to create goals, measure, whether you achieve those

00:07:21: goals and try to influence the behavior of your

00:07:24: employees or your customers in a way.

00:07:26: So if you have this holistic view, you

00:07:29: see okay, one goal does not cut it.

00:07:30: So you need maybe you need different types

00:07:33: of goals. So maybe you have a more

00:07:35: strategic direction, a long-term goal, something you want

00:07:38: to change in the world, how what makes

00:07:40: different types of goals. So maybe you have a

00:07:43: more strategic direction, a long-term goal, something you want

00:07:46: to change in the world, how what makes you

00:07:49: different, what's your playing field, these kind of stuff.

00:07:52: And on the other side, maybe you have a

00:07:54: little bit more tactical stuff. So what is the

00:07:57: team actually doing this week or the teams in

00:07:59: this month? So we have different notion of goals

00:08:02: and EBM does not describe so how to define

00:08:04: those goals, but helps you to think okay, there

00:08:07: are different hierarchies of goals, let's say.

00:08:10: And which you need to take into consideration, the

00:08:14: last thing I want to mention is it's a

00:08:17: framework. And from the no it's more like a

00:08:20: minimalistic framework. So if you remove one thing, then

00:08:24: it breaks. You can add stuff to it.

00:08:27: So for goals I think you can use OKRs.

00:08:30: For team goals you can use SMART goals.

00:08:33: or anything else you like.

00:08:35: There's one thing that I just noticed because you

00:08:38: talked about evidence-based medicine that this approach seems to

00:08:42: be inspired by and I imagine that I'm not

00:08:45: an expert on that but I imagine that it's

00:08:48: kind of more clear cut maybe where you can

00:08:50: actually directly measure the effect for example of any

00:08:54: kind of medicine that you develop. Now in product

00:08:57: development for example my experience is that it's a

00:09:01: lot more messy and things are m more difficult

00:09:04: to measure so in

00:09:05: context, what does evidence actually mean? How

00:09:07: does that how does it apply How?

00:09:09: What does it look like in reality?

00:09:11: So if we talk about evidence, so

00:09:14: I don't want to go into the

00:09:16: notion of scientific evidence. Oftentimes

00:09:19: it's a too much overloaded term Nevertheless.

00:09:22: we have scientific evidence that goes basically in the

00:09:26: same direction. So what I like to call it is like

00:09:31: signals, and then we need to think about how

00:09:33: strong your signals are, or if you call it

00:09:36: evidence, how much evidence do you have.

00:09:39: So we have a spectrum. So on the one

00:09:41: side we have actually scientific evidence, but you need

00:09:44: to be very careful. the most of the scientific

00:09:47: evidence done at university is wrong. And there was

00:09:51: a study like m like I don't know if

00:09:53: it's really like 89 percent of all the experiments

00:09:56: are basically are disproven after a year.

00:09:59: So we need to be very careful

00:10:01: that side. But that's one side. So the

00:10:03: scientific evidence from universities or in medicine.

00:10:06: And on the other hand of the spectrum,

00:10:08: on the other side we have your gut feeling.

00:10:12: And there is something in between. Okay?

00:10:14: And we tr with EBM, we try to ask,

00:10:16: there is something in between, so how can we

00:10:19: think about those what's in between. I can make

00:10:22: it very practical. So if you move a little

00:10:25: bit away from your gut feeling, so we go

00:10:28: out of the building and talk to people.

00:10:30: The easiest thing what we can do to increase

00:10:33: evidence would be instead of ranking the feature based

00:10:37: on your gut feeling, we could rank the features

00:10:40: based on the stakeholder

00:10:42: it's a little bit more evidence, it's more opinions

00:10:45: that could be one dimension. The other dimension we

00:10:48: could rank it on the how much effort it

00:10:50: actually takes for the developers to build those stuff.

00:10:54: So we have another dimension. Now the critical part

00:10:57: that is often missing is the question how confident

00:11:00: are you in your estimates? It's often missing so

00:11:03: and like a product managers or sales persons say

00:11:07: I have a customer who expresses he wants to

00:11:09: have these feature or I have a paying

00:11:12: or he already gave us money for us to

00:11:14: build it. or if you go on the effort

00:11:16: side, yeah, I never did it but, I have

00:11:18: a gut feeling it looks like a lot of

00:11:21: work. Or a basically for another company we actually

00:11:24: build the same things. I'm pretty confident about how

00:11:27: to build it. It's a lotta it's a different

00:11:30: a level of evidence. So you see there is

00:11:32: a different notion of how much evidence we have.

00:11:35: And to carry it out that understanding of I

00:11:38: think that's EBM and we can make it a

00:11:40: little bit more formal later

00:11:42: that I think that's a very helpful way to

00:11:44: think in terms of product. So we have effort,

00:11:47: we have value, and we have the confidence.

00:11:50: It's another dimension. And in confidence we need to

00:11:53: understand what confidence actually means for us.

00:11:56: So in software development to get the highest confidence

00:12:00: basically if you ship stuff or if you do

00:12:03: A/B testing I think that's the highest confidence you

00:12:06: get is the most accurate evidence in terms of

00:12:09: whether something is working. And on the

00:12:12: decide, we have your gut feeling,

00:12:13: and there are steps in between.

00:12:14: And to make this explicit, I think that's

00:12:17: the job of a Product Owner and also

00:12:20: here Evidence-Based Management helps a lot to understand

00:12:24: the different kind of confidence.

00:12:26: Do we usually see this applied on a product

00:12:29: level or also other levels in the organization?

00:12:33: Yeah, that's so interesting because a lot of

00:12:35: people are distinguishing between product and organizational change

00:12:39: to be honest, it's the same thing at

00:12:41: least in my opinion. The only thing that

00:12:43: is changing is the notion of a customer.

00:12:46: So let's let's start with product. So the what's

00:12:49: the goal of product management? Basically it's to change

00:12:53: the behavior of people to buy it, to use

00:12:56: it, to recommend it to your friends, whatever.

00:12:59: One thing. Now if you look at organizational change,

00:13:02: so what's the goal of organizational change?

00:13:05: Change the behavior of your employees, not your customers.

00:13:09: So what should they do differently? It's the same

00:13:12: question as we look about product development. So what do

00:13:16: should our potential customer should do differently?

00:13:20: What should our employees do differently? At the moment

00:13:24: we have this huge, I don't know, virus going

00:13:27: on about AI strategic transformations. And the question is

00:13:32: fundamentally the same, what should your employees do differently

00:13:36: now? Should they use more AI? I don't know,

00:13:39: I think. As we need to influence those behaviors

00:13:43: in order to create value for the organization

00:13:46: or if you think about product with our products.

00:13:49: It's so for me it's basically the same thing,

00:13:52: but for organizational development the methods are not so

00:13:56: Yeah, don't get me wrong. well developed

00:14:03: or known Like if you think about Evidence-Based

00:14:06: Management in an organizational context, and we have

00:14:10: goals for instance, we have policies, we have

00:14:13: also some kind of nudges which are evidence-based.

00:14:16: You know it from the supermarket.

00:14:18: So organ supermarkets try to nudge you in a

00:14:21: certain direction. And we can use the same ideas

00:14:25: which are very evidence-based in organizational development.

00:14:29: So we want to change the behavior of our people. We

00:14:33: have rewards or these kind of things.

00:14:35: So it's I think it's the same idea.

00:14:37: The behavior change, the method might be a little

00:14:40: bit different. We don't just build feature, we also build other stuff.

00:14:45: what I would like to look at next is

00:14:47: what does that look like in practice for example

00:14:51: for a Scrum team. as you know I co-wrote

00:14:54: a book about how a lot of teams struggle

00:14:57: applying Scrum and I always have like a very

00:15:00: different understanding of what for example a Sprint Review

00:15:04: could look like or a colleague of mine and

00:15:07: me we often when we t speak at conferences

00:15:10: we ask people if they use Sprint Goals and

00:15:13: how they arrive at their sprint

00:15:15: goals and how many people actually use Product Goals.

00:15:18: And I could imagine that this is something where

00:15:20: EBM really comes in because in my experience a

00:15:23: lot of teams struggle to define what success is

00:15:25: or they kind of maybe read that there is

00:15:28: some kind of Product Goal that is that they're

00:15:30: supposed to have, but they don't really know what

00:15:33: that looks like. Is this an area where EBM can help?

00:15:36: Yes. so EBM we call it intermediate goal I.

00:15:39: think that's best describes Product Goals. It's a Product

00:15:44: Goal typically these are most co-organizations which are a

00:15:48: little bit larger are not strategic themes.

00:15:52: Okay. And so they're underneath the company strategy I would say.

00:15:58: And so what can be learned from EBM then?

00:16:01: in terms of setting Product Goals, first we need

00:16:04: distinguish what success looks like. So at the end

00:16:07: of the day now for organization success looks basically

00:16:11: increased profit to make it oversimplified simplified.

00:16:14: So what goes into profit so, we have revenue,

00:16:17: we have costs and these kinds of things.

00:16:20: So if you would this would create the most

00:16:23: evidence if you're successful. Assume you build a product

00:16:26: and you can create more

00:16:28: it. Awesome. You have one. You have some evidence

00:16:31: that you do the right thing. It's working.

00:16:34: Now this is hard and not easy to influence

00:16:37: and very lagging. So what we tend to do

00:16:40: is now let's focus on what we actually can

00:16:43: do. Okay, our Product Goal could be let's build

00:16:46: this feature, ship this product or whatever.

00:16:49: And now we have a gap between the impact

00:16:52: create revenue and actually what we are doing shipping

00:16:56: stuff. And I think the a good

00:16:58: to increase the likelihood that we actually c create

00:17:01: impact is to focus on customer outcomes.

00:17:04: So the change of behavior it provides a little

00:17:07: bit more evidence that it's actually working, but it's

00:17:10: it's in the realm that teams actually can influence

00:17:13: it. So the new feature so if we ship

00:17:16: the new feature we expect that people are using

00:17:18: it fifty percent more of the time, our application

00:17:22: for instance, because everybody's both waiting for the new

00:17:25: AI chat-board functions on the website.

00:17:28: That's what we believe. And so as before we

00:17:31: see again so different types of evidence.

00:17:33: So the most evidence that something is working would

00:17:36: be increased revenue for instance if you could hit that target.

00:17:41: Much less evidence would be we just ship

00:17:43: a thing, then we just learn that we

00:17:45: can actually build the stuff, but it does

00:17:48: not move the customer. So I bet a

00:17:50: little bit more evidence would be on focusing

00:17:52: on customer outcomes. And that's again Evidence-Based Management

00:17:56: thinking in terms of product management. Yeah.

00:17:59: Great and in my work when working with teams

00:18:02: for example I noticed that I come to categorise

00:18:06: the type of Sprint Review that people have in

00:18:10: the three broad categories and one is essentially they

00:18:14: talk about are we sticking to the plan and

00:18:17: the iteration is kind of like are we still

00:18:20: in the big project plan we finished something what

00:18:24: does that mean. Then another type would be a st of stakeholder

00:18:29: feedback. So we show something to someone often inside

00:18:33: the company and we want their feedback and they

00:18:36: kind of give a thumbs up or they say

00:18:38: maybe change this. And then the third type is

00:18:41: much rarer which is we actually talk about outcomes.

00:18:45: so we talk about what effect have we created,

00:18:48: what can we measure. So it sounds like exactly

00:18:51: wh what you just mentioned that this is kind

00:18:54: of something that you would be driving for using

00:18:57: EBM and then in this way a

00:18:59: review would maybe transform more in that

00:19:01: direction, is that is that correct?

00:19:03: Yes, so if you as a Product Owner start

00:19:05: thinking about what actual value is, then you exactly

00:19:09: you make those steps what, you said.

00:19:11: So I would I think that the best notion

00:19:14: in terms of value is if you think about

00:19:17: customer behavior change, that's the best indication for value,

00:19:21: which is also influenceable and you're you're totally right.

00:19:25: And I think for a lot of organization I

00:19:28: that's not a bad thing, that's how things involve,

00:19:31: but what you just

00:19:33: the first level or the first type of Sprint

00:19:35: Review, it's more like for some times we call

00:19:38: it project thinking, but I think that's not a

00:19:41: it's not a bad thing. Projects are not a

00:19:44: bad thing. The thing is it's what we assume.

00:19:47: Like we assume if we can build this stuff,

00:19:49: then it actually will work and customer will be

00:19:52: satisfied. And the biggest fear or problem or risk

00:19:55: we see can we actually build this thing, which

00:19:58: for a long time was really difficult for software teams.

00:20:02: Okay, that was the major problem. Can we build

00:20:05: these type of stuff? If we can, then people

00:20:08: will buy it. So they're supposed to build them.

00:20:11: But I think nowadays it changed a little bit

00:20:13: that actually building stuff is not so much the

00:20:16: problem anymore. It's more about okay, we can build

00:20:20: stuff, but it doesn't actually improve the lives of

00:20:23: our customer. do they actually find the product, are

00:20:26: they actually using it, do they find it helpful

00:20:29: and so on. That's the main risk nowadays. So we

00:20:32: moved a little bit away from actually building stuff

00:20:35: on time and budget. It's more about okay, the

00:20:38: risk is there's so much competition from the other,

00:20:40: I don't know, companies and products. Are they actually

00:20:44: using our stuff? I mean that's the basic problem

00:20:46: with the German automotive industry at the moment.

00:20:49: So you can't build cars, but are there actually

00:20:52: people who buy the cars? Or do they buy

00:20:54: Chinese cars or whatever? That's the major problem.

00:20:57: But don't go into automotive stuff. No, let's just

00:21:00: pretend nothing is happening.

00:21:02: Awesome. Do you have any kind of favourite example

00:21:05: of working with an organization or a team and

00:21:08: they created some form of evidence and then they

00:21:11: changed their approach or their behavior as a result?

00:21:16: So to be honest. Now that do you see

00:21:18: the same thing with being busy? It's great.

00:21:21: Everybody sees oh Johannes is so busy he must

00:21:24: be so helpful for the organization and so on.

00:21:28: Okay that's the easy part. The hard part is

00:21:31: to commit to a goal. Because committing and focusing

00:21:34: to a goal actually means it ma it might

00:21:37: show you very early that you're on the wrong track.

00:21:45: ma it might show you very early

00:21:46: that you're on the wrong track.

00:21:50: And that's difficult. So we decide to do something

00:21:53: and now we learned after one week that it's

00:21:55: actually not working. It's a failure, which is really

00:21:58: hard to admit. Okay, we focused on something and

00:22:01: learned early that it doesn't work. So everybody knows

00:22:04: we are failing, which is an unpleasant situation, at

00:22:07: least for most teams and most organizations.

00:22:10: So we need to okay, that's a given.

00:22:12: So even if you frame it as an experiment,

00:22:15: it's still you suck. It didn't work, so you

00:22:17: failed. So that's the fear we need to overcome.

00:22:20: And now the moment we start now measuring with

00:22:23: more data and more evidence, it gets even harder,

00:22:26: because now you actually know you're not just busy,

00:22:29: but you do not create any meaningful impact, which

00:22:32: is really hard, which is a for a lot

00:22:34: of teams it's really difficult. So we need to

00:22:36: be step by step. So let's say we think

00:22:38: about okay, we want to introduce OKRs, outcome-centric OKRs,

00:22:42: but I also say at the beginning maybe you

00:22:44: have some output in the OKRs as well Okay.

00:22:47: in order to okay, we build something we, don't

00:22:49: know if it actually

00:22:50: the needle, but actually we build something.

00:22:53: That's the first step. And over time we add

00:22:56: more evidence to it. So not just building stuff,

00:22:59: but do we change the customer behavior?

00:23:01: Do we actually create impact? And I have an

00:23:04: example for that which I can talk a little

00:23:07: bit about. they call it a complex product, but

00:23:10: actually it's a combination of hardware and software parts.

00:23:13: And which needs to be installed in their for

00:23:16: their customer, into their organization's very complicated system.

00:23:21: And in order to sell these kind of complicated

00:23:24: systems you need to convince your own sales people

00:23:28: to actually advocate for the new features.

00:23:30: Okay, otherwise if their customer do not buy the

00:23:34: new add-on features the, stuff will never be used.

00:23:37: Now we can now think okay, this feature is

00:23:40: so great. Our sales rep will actually show it

00:23:43: to our actual customers. Yeah, that's wishful thinking.

00:23:47: That's a gut feeling. So then we set a goal.

00:23:50: Okay, but what did we have? I'm not sure

00:23:54: if it's one in the correct end.

00:23:56: sales rep attend our Sprint Review at least three

00:24:00: out of four Sprint Reviews and ask for a

00:24:03: new feature idea or stuff. Or want to try

00:24:07: out new features. So that was for a three

00:24:10: month period. At the end of the day no

00:24:13: sales rep did ev ever show up.

00:24:17: it's hard. So you actually failed. So we built

00:24:20: something but we c not even convince our own

00:24:23: sales person to show it to our customers.

00:24:25: So actually we created a lot of costs.

00:24:28: It's short. It's a failing. Now if we have

00:24:31: it very early it, would be easier to hide

00:24:33: that in fifteen years. Everything will be okay.

00:24:37: No, it won't. And evidence-based management and goal setting

00:24:40: is a r it's really a struggle at the beginning. So

00:24:45: there's something I noticed because I it feels like

00:24:48: we have a similar approach. I'm a big fan

00:24:51: of making things measurable and in creating confidence or

00:24:54: evidence. And something that I noticed is that at

00:24:58: some point it kind of plays into a question

00:25:00: of leadership. So I it often sounds very appealing

00:25:04: to organizational leaders to be more evidence-based and then

00:25:08: I also noticed that once you create an environment

00:25:11: in which people actually work that way, some fear that

00:25:15: there's a bit of lack of control because in

00:25:17: the past they used to be able to just

00:25:20: go in and like this is my opinion, this

00:25:23: is what we have to build and now maybe

00:25:25: as a leader for example I have to justify

00:25:28: my or I have to my show my assumptions

00:25:31: and then if I there's a topic that I

00:25:33: carry deeply about then suddenly it's position twelve in

00:25:37: the Product Backlog instead of me wanting it at

00:25:40: number one. this podcast is also kind of therapy

00:25:43: for myself but also

00:25:45: I'm I'm kind of outlining my struggles and

00:25:47: just hoping that someone can help me.

00:25:49: So how do you handle these situations?

00:25:51: I'm sh you're nodding so I'm assuming you

00:25:53: have seen this before, but do you have

00:25:55: the perfect solution to dealing with moments like that?

00:25:58: have the perfect solution for how to get fired

00:26:01: as a product coach. It's really simple.

00:26:04: Now, yeah it's like was a couple of years

00:26:07: back. And there a lot of stakeholder wishes from

00:26:11: all different departments of the company. What we did

00:26:15: is okay, we build this stuff for you totally

00:26:18: fine, but actually how do you know that it's

00:26:21: actually successful. So if we build this widget on

00:26:25: the website, so what should change. And we

00:26:28: tracked those answers. So we made an additional column.

00:26:31: So in our sprint backlog we had the traditional

00:26:34: typical to-do, done and so on. And behind it

00:26:37: we had I'm not sure what its actual name

00:26:40: was maybe validated or something like that.

00:26:43: And then we wrote down what the person actually

00:26:46: said. Or we looked at some it was a

00:26:48: website so we looked at some analytic stuff.

00:26:51: Are people actually using it to a certain extent

00:26:54: and so on. And then we made a marker.

00:26:57: I think it was eight

00:26:58: So and after three sprints or so and the

00:27:01: review we came back to that feature we said

00:27:04: we shipped it a quarter ago and now let's

00:27:07: look at the data what we actually get out

00:27:10: of it, if people are using it and it

00:27:12: actually move the needle as you said boy, that's

00:27:16: the that's the easier way to get fired.

00:27:19: It never did. And right and that's not just

00:27:22: us or different companies I think there were a

00:27:25: lot of study done by I think it was

00:27:28: Microsoft that published a paper on that, that only

00:27:31: eight percent of the stuff they're actually shipping has

00:27:35: a meaningful impact on their key metrics.

00:27:37: And it's so yeah it's crazy. So that's the

00:27:40: game. Yeah, that's the game. So you can look

00:27:43: it up. It's it's a it's a paper about

00:27:45: A/B testing, I guess. It's really it's really interesting.

00:27:49: And it and that's the reality. And the evidence

00:27:52: part here is you make it transparent.

00:27:55: That's the Scrum thinking also here. You don't make

00:28:01: transparent. That's basically the retrospective to learn how to

00:28:04: we actually improve our development process. A lot of

00:28:07: teams are sitting together and talk about their feelingss.

00:28:10: I'm not a big fan of that.

00:28:12: It's more about okay, we build something, we learn

00:28:14: something, how to incorporate this kind of learning in

00:28:17: our in the development process. So how can we

00:28:20: make sure that earlier that we can somehow validate

00:28:23: if it's a good idea. Even if it's hard

00:28:25: to say to our stakeholders, yeah, I don't think

00:28:27: that's a good idea, we don't we won't build

00:28:30: it. So how can we

00:28:31: work with them together to actually build something that

00:28:34: at the end of the day they move the

00:28:36: needle better than eight percent, which would be remarkable.

00:28:40: You just hit on a couple of key things.

00:28:42: One is that I always come back to the

00:28:44: fact that essentially what we want to install in

00:28:47: organization is kind of this learning loop.

00:28:50: and some people for some reason take offence to

00:28:53: this or I've heard people kind of criticise this

00:28:56: idea, but I'm I'm still it always comes down

00:28:59: to this. And for me there are three steps

00:29:01: in that and one is kind of create the

00:29:03: evidence or create some kind of form of visibility

00:29:07: transparency. The other one is that you then have

00:29:10: to start interpreting what is happening, and the third

00:29:13: one is that you actually have to act on

00:29:15: what you have learned, and then you kind of

00:29:18: create, like you complete that learning loop.

00:29:21: You just mentioned a couple of things in there,

00:29:24: but how do you help teams and organizations go

00:29:26: through this process so that it's not just for

00:29:29: example a heap of useless evidence that no one

00:29:32: looks at, or a lot of 'maybe we shoulds'

00:29:34: you know like we talk about maybe oh this

00:29:37: isn't working, maybe we should do something

00:29:40: else, but then nothing happens. and how like

00:29:43: do how do you g guide people through

00:29:45: the entire thing where in the end act

00:29:48: we have meaningful action and there's this constant

00:29:52: learning and acting on evidence?

00:29:57: there's this constant learning and acting on evidence.

00:30:02: Yeah, so if you have an answer to that,

00:30:05: next month I have a workshop where actually I

00:30:09: need to figure it out or not.

00:30:11: So we can we can track it from we

00:30:14: can think of it from cultural organizational perspectives.

00:30:18: So in order to learn, I think you are

00:30:21: familiar with that, we need to create, I don't

00:30:25: know how to call it, some kind of learning

00:30:28: culture which actually starts with appreciation, open to

00:30:32: back, receiving feedback. It's not for fair and micro

00:30:37: stuff. And it was oftentimes really hard for developers

00:30:41: to admit they're wrong or provide helpful feedback, take

00:30:46: feedback. It's that's on a micro level.

00:30:50: On the other side I'm so really tactical I,

00:30:53: always hope that if I do a good job

00:30:56: to present facts which are given and do not blame somebody,

00:31:02: people are smart enough to draw their own conclusions

00:31:06: and make the next steps. And I think that's

00:31:09: the hard part. So how can we provide data

00:31:12: where people can in look at it and figure

00:31:15: out what to do next. I think that's hard

00:31:19: because often times we provide data which is hard

00:31:22: for people to admit. So okay, the last sprint

00:31:26: you were not here for instance which is it's

00:31:29: it's very aggressive. So that's the

00:31:32: thing. And then how to follow up on change?

00:31:35: I think I will quote something from your book.

00:31:38: It's the snowball metaphor where change starts with a

00:31:41: small snowball and then if you start rolling it

00:31:45: get bigger and bigger. I think that's always my

00:31:48: strategy. Okay, so what is the thing we can

00:31:51: do until tomorrow or next week and how can

00:31:53: I follow up with you and help you.

00:31:56: It's not about controlling you, it's about maybe I

00:31:59: can assist you a little bit.

00:32:02: And for a coaching perspective, this is always my

00:32:05: stance, which is actually not coaching at all.

00:32:08: It's how can I help you? Let's make it

00:32:10: together. So it's going together through the struggle.

00:32:14: That's what I tried, which is basically not coaching,

00:32:17: but it's like working together. So how can we,

00:32:20: I don't know, try to change the code base

00:32:23: or go to a manager and talk about this.

00:32:25: Let's do it together. So it's more like the

00:32:28: community aspect of change. It's just not the basic

00:32:32: coaching aspect. So that was would be one

00:32:35: thing what, I tried to do is do

00:32:37: it together with people, which is actually really

00:32:41: hard as an outside consultant. Yeah.

00:32:44: Oh yeah. I'm a big fan at looking at

00:32:46: the boundaries of any approach that we have.

00:32:49: I'm noticing that any time there's something new, we

00:32:52: run the risk of people going oh, this is

00:32:55: going to solve all of our problems and we'll

00:32:57: ju we'll just use it for everything.

00:32:59: Is there anything specific that people should not expect from EBM?

00:33:03: Yeah. Now, as EBM is based on the idea like Scrum,

00:33:06: it's your mother-in-law. It doesn't tell you how to do stuff,

00:33:10: it just tells you you are wrong. Fix it.

00:33:14: And that's the same thing with Evidence-Based Management. So we have these four key value areas. We have current value and the thing is

00:33:27: I mean I can ask you, do you

00:33:28: know whether your product if you're working on

00:33:31: products actually delivers value right now?

00:33:34: And to be honest, not so many people can

00:33:36: answer those questions. And so do you actually know

00:33:40: the user base or how much money do you

00:33:42: make in these kind of things. Most Product Owners,

00:33:45: I don't know. So how do we actually decide

00:33:48: if we should fix this bug or build this

00:33:51: feature? How do we do it? And that's what

00:33:53: Evidence-Based Management shows you and what's the downside of

00:33:58: it, it does not provide you an approach, method,

00:34:01: and how to solve this problem.

00:34:04: I'm looking for that. Yeah.

00:34:07: Once we find it, we'll let everybody know, okay?

00:34:10: Yeah. And for some areas and products also

00:34:13: it's developed let's say, like current value and

00:34:17: unrealized value in terms of conversion funnels for,

00:34:21: I don't know, let's say software products on

00:34:24: websites. It's pretty there are steps you need

00:34:27: to follow in order to understand the situation.

00:34:31: It I'm sold on EBM I'm fully bought in.

00:34:34: why is this not already the default way that

00:34:38: organizations manage products? What are you noticing that kind

00:34:43: of keeps people from using this? Or what have

00:34:46: people used in the past and why?

00:34:50: Data is not very inspirational. And I believe

00:34:55: people do not follow logic or data.

00:34:58: They follow people. And it we always tend

00:35:03: towards the leader and not towards the data.

00:35:09: everybody talks about yeah, we should be equal and

00:35:12: there shouldn't be any hierarchy and it's bullshit because

00:35:16: we need hierarchy in order to feel safe.

00:35:19: So there is a leader oh, I follow him

00:35:22: and then I'm safe. We are scared like from

00:35:25: a nature perspective and we are not really good

00:35:28: in surviving. So we need a person who tells

00:35:31: us what to do which we can follow and

00:35:33: this is really hard and organization have these hierarchies

00:35:37: and leaders which are in

00:35:39: operational, and Evidence-Based Management is a little bit the

00:35:43: opposite, so it's conflicting. By the end of the

00:35:47: day I think we need both. We need good

00:35:49: leaders who ha show us the vision, inspire us,

00:35:52: but their decision-making is grounded in evidence.

00:35:56: It's not purely evidence-based, but they're actually using evidence.

00:36:01: But we need to admit that we need leadership.

00:36:04: So I think there will never be a company

00:36:06: which is also only run by scientific

00:36:09: Yeah, maybe with the future of AI

00:36:12: Yeah. hopefully not. But so that's a

00:36:15: given b we need leadership.

00:36:20: which makes it hard. Because the if there is

00:36:23: no evidence in an organization they are purely driven

00:36:26: by leadership. So I have one customer who drives

00:36:29: me crazy. There's everything is done via attendance of

00:36:33: meetings. Like that's the default decision-making process.

00:36:37: So the more status you have in an organization,

00:36:40: the more meetings you run basically. And if you

00:36:43: have lower status, you need to attend.

00:36:45: So you or CC on e email, you'll just

00:36:48: attend the meetings, don't say

00:36:50: It's really these tribe-driven leadership stuff. It's

00:36:53: totally the opposite o like with evidence

00:36:56: or something. It's about you need to

00:36:59: be there in order to be seen.

00:37:01: It's crazy. So and I think a lot

00:37:04: of product decisions can be made without attending

00:37:09: meetings and just look at the data.

00:37:13: Now let's imagine someone's listening and wants to

00:37:15: try out Evidence-Based Management. How do they get

00:37:18: started instead of like installing the entire thing

00:37:21: at once? Is there something is there one

00:37:23: small step that they can take or one

00:37:24: small experiment that can run to get started?

00:37:27: Yeah. It's the fight club. Never tell that you

00:37:31: are in the fight club. don'ts I mean that's

00:37:34: the number one mistake. A lot of people I

00:37:37: see the mistake so they learn something and then

00:37:40: they immediately try to apply it to their team

00:37:43: and organization and then use the new language and

00:37:47: everybody says whoa, wait a minute, hold on a

00:37:50: minute, what's that. So don't use the language, just

00:37:53: use the ideas. So the first idea is do

00:37:56: we actually have a

00:37:57: and you have a goal, the

00:37:59: question is, is it explicit?

00:38:01: Oftentimes there are implicit goals. Maybe that's just some

00:38:05: measurements people were measured by, which are the goals.

00:38:09: Like you need to be here from eight, from

00:38:12: nine to five, and that's your goal.

00:38:15: So make the goals explicit. And it doesn't need

00:38:18: to be perfect, but you need to look or

00:38:20: w what the goals are. That's the first thing.

00:38:24: And then the next thing is what you can

00:38:26: do. So how do we actually achieve those goals?

00:38:29: Is there a plan step by step

00:38:31: Are there some check-in milestones

00:38:34: and these kind of things?

00:38:37: Maybe that's a starting point. So how can we

00:38:40: not think of it as a five-step process and

00:38:42: if we follow then we are successful.

00:38:45: So let's change the language a little bit.

00:38:48: We believe that if we do X_ by Z_

00:38:50: then we will be successful Success. for us means

00:38:53: mm-hmm. Now you start a little bit with an

00:38:56: experiment. It's not too much. But so you can

00:38:59: think of Product Backlog items. We believe we should

00:39:02: build this feature. If we are successful then we would drive more

00:39:07: version for instance. Alright, now you start a little

00:39:10: bit more experimentation and the last step is to

00:39:14: add data to it. And don't call it Evidence-Based

00:39:17: Management. Simon, thank you so much for your time.

00:39:20: i if people want to kno learn more about

00:39:23: you, you have a Substack for example where, can people find you?

00:39:28: Yeah, I have a Substack that is German and

00:39:31: called Scrum with AI, but it's basically Evidence-Based Management.

00:39:35: So I'm trying to answer the question which I

00:39:38: find very interesting. So we had Scrum, then we

00:39:41: used SAFe, now we have AI, how do we

00:39:43: actually know that it's working. And that's the answer

00:39:47: is how do we measure success and that's Evidence-Based

00:39:50: Management. So I circle around the topic a lot

00:39:53: Maybe that's a starting point or my LinkedIn.

00:39:56: I also share some posts

00:39:58: Great. We'll put all the links in the show

00:40:00: notes. Thank you so much. I hope this session

00:40:03: has given you some concrete ideas for your own

00:40:05: transformation challenges. At Holisticon, we provide companies with holistic

00:40:10: support, from strategy and technology to organizational change.

00:40:13: If this sounds interesting, please feel free to contact

00:40:16: us. You can find all the information you need

00:40:19: at holisticon.de and don't miss our next transform together

00:40:22: session. See you then.