In this episode, experts from across our organization assembled to illustrate how organizations can execute impactful culture change at speed and scale.
[INTRODUCTION]
[00:00:02] SW: Welcome back to season 7, episode 7 of Your Brain at Work podcast. If you're not measuring success, then it's a hobby. How do organizations move forward with big ideas to successful execution? How do we spot problems and build smart solutions? How can we know that our habits are steering us toward the right outcomes? Leaders and teams wrestle with these questions every day, whether they know it or not. This episode will bring experts from across our organization to outline how to build, execute and measure a successful change management strategy from the ground up.
I’m Shelby Wilburn, and you're listening to Your Brain at Work from the NeuroLeadership Institute. We continue to draw episodes from our weekly Friday webinar series. This week, our show is a conversation between Marshall Bergmann, Senior Vice President of Sales and Corporate Solutions at the NeuroLeadership Institute; Katherine Milan, Senior Vice President of Client Experience and Product at the NeuroLeadership Institute; Dr. Emma Sarro, researcher at the NeuroLeadership Institute; and Drew Sumter, Director of Client Experience at the NeuroLeadership Institute. Enjoy.
[EPISODE]
[00:01:09] SW: Hello to all of our viewers across the world, and welcome back to another week of Your Brain at Work Live. I’m your host, Shelby Wilburn. In this week's episode, we'll have a conversation to reveal the origins of our measurement platform.
For our regulars, it's great to have you back. But for those of you that are new to Your Brain at Work Live, welcome to the party. For some context, it is the title of one of the best-selling books by our CEO and co-founder, Dr. David Rock. And it's also the name of our blog and podcast.
Now, as we get ready to start, grab some coffee, grab some water, and settle in as we dive deeper on the topic of measurement.
We have a full house today. Let's introduce our speakers. Our first guest is a researcher at the NeuroLeadership Institute where she focuses on translating cognitive and social neuroscience into actionable solutions for organizations, as well as helps to communicate relevant research in an accessible manner for the public. Previously, she served as a professor at Dominican College and New York University, and as a researcher at The Nathan Kline Institute. She holds a PhD in neuroscience from New York University and a bachelor's degree from Brown University. A warm welcome to NLI researcher, Dr. Emma Sarro. Emma, thanks for being here today.
[00:02:23] ES: Thanks for having me.
[00:02:25] SW: Our next guest partners with large businesses to implement key initiatives for leadership transformation, culture change and inclusiveness. Over the past 10 years, he has led teams that identify and overcome challenges to promote innovation at scale. Consulting across many industries, including finance, technology, healthcare, biotech and pharmaceutical, he and his team are able to help drive real sustainable change while allowing smooth business procedures. Please join me in welcoming our Director of Client Experience at the NeuroLeadership Institute, Drew Sumter. Drew, great to have you here today.
[00:02:58] DS: Great. Thank you, Shelby. Excited to be here.
[00:03:00] SW: As the heartbeat of our client experience team, our next guest is not only in charge of delivering products and solutions NLI offers to our clients, but continues that world-class relationship by leading the charge on sustainment support and measurement data. She lives and breathes the impact of our work on organizations and the people who are a part of them. When you partner with NLI, you are placed in the hands of a best-in-class team led by our Senior Vice President of Client Experience and Product at the NeuroLeadership Institute, Katherine Milan. Katherine, it's so great to have you back.
[00:03:30] KM: Great to be here. Thanks, Shelby.
[00:03:33] SW: And for our final guest, as the captain of our sales content and marketing teams, he is no stranger to the business world. He's a numbers guy, but also has an abundance of experience developing and implementing solutions and initiatives that transform the way leaders and their organizations think and behave. He's also presented at TEDxGramercy about how the power of games can drive behavioral change in business. A warm welcome as I pass the mic to Senior Vice President of Corporate Solutions at the NeuroLeadership Institute, Marshall Bergmann. Great to have you here, Marshall. And I’m passing it over to you.
[00:04:04] MB: Thanks, Shelby. And thank you for the great introductions. Thank you, audience, for joining us today. We have a great crew. I’m just going to spend a little bit of time kind of setting the stage. And then I’m going to let our experts do most of the talking today.
But one of the things that's great about working at NeuroLeadership Institute is not only do I get to work with amazing organizations like the people who have joined us today, but I also get to work with really brilliant people. Dr. Emma Sarro is a researcher and someone that I am lucky to work with every day. And I’m really excited to have her share with you some of the underlying signs of what do you need to understand as a leader of an organization or as a leader in talent development in order to drive behavior change in your organization?
And so, she's going to really provide that kind of science lens to why we do the things that we do. And you'll be able to pick up some great insights that you can apply immediately back to your day-to-day work.
Now, on the other side, we don't just have great scientists, but we have great actual folks who do work on client sites. Drew and Katherine represent some of the best of what we do with our clients. They work hand-in-hand with clients to drive behavior change in order to help organizations achieve their business goals. And that is not easy work. And so, I’m really excited for them to build on the science that Emma will be sharing with and talk to you not only about the results that we're able to get when we work with clients, but also share with you some best practices that we found that will really help you and your organizations.
That's an overview of what the different folks are going to be talking about today. And today's session, just to get real specific, is really about summarizing some work that we've been doing over the last six years, which frankly sometimes I think we should be more proud of it. But I think what we've really done is, over the last six years, working with our researchers and our client experience team have really cracked the code on how to drive sustainable behavior change at organizations.
As someone who spent 20 years in the human capital space, developed countless training programs with clients and different organizations, actually being able to – Behavior change and measure behavior change over time is kind of that holy grail of what learning and development organizations are looking for.
Today is going to be all about the underlying science that drives that. The best practices that can help you drive behavior change. We're going to share some amazing results that we've been able to achieve. We're really proud of the work that we've done today, and we're really excited to share it with you.
Without further ado, let me give you a quick overview of what we're going to do. I’m going to do a quick agenda setting. Just introduce folks who might be new to NeuroLeadership Institute or new to Your Brain at Work Live. Share a little bit about the work that we've done so far. And then I’ll be handing it over to Emma, and then Katherine, and then Drew.
For those who are new to the NeuroLeadership Institute, we've been around for about 24 years. At the core of our business is research. We have the NeuroLeadership Journal. We've been publishing research since our inception. And we continue to publish research. We have a very deep research team that's working not only internally and with our clients, but also with external organizations. And we've published 50 research papers since 1998. Some on the science of behavior change, which I’ll share with you in just a bit.
Our work reaches some of the greatest in the world. Right now, we work with 63 of the Fortune 100. We work with global corporations large and small. And we work with them to help them achieve their goals by driving behavior change at scale. And we have operations in 24 countries. And our work appeals and works in all sorts of settings.
And just getting into a little bit of the research, I mentioned the NeuroLeadership Journal before. These are some actual papers from our journal that actually talk about some of the underlying science that we're going to be covering with you today. We've got the science of making learning stick. Getting to a culture of feedback. Leading change with the brain and mind. There's a lot of insights from both social science, neuroscience and cognitive neuroscience, that if you apply can really make a difference in how you drive change in your organization.
But what I do want to share with you, it is definitely very possible to get better results than what many of you are experiencing in your programs right now. That's achievable goal. And we're going to share with you how you can get to that.
We've spent a lot of time in the virtual world over the last couple years, and a lot of organizations have shifted a lot of their training to virtual. The question here today is, if you consider three different virtual training solutions, we've got what we call the HIVE, the high impact virtual experience. Think of like a half day – A three-hour workshop broken up into one-hour sessions. One session per week. So, spread out over time. Or a virtual digital learning solution. Again, think one digital interaction once per week over time. Or an in-person workshop. Let's think of a three-hour half-day workshop. Which of these do you think delivers the greatest behavior change?
We're going to share with you what we're getting in terms of results here at NLI. NLI, we're looking at these sorts of results. 88% of people changing behavior once per after a virtual HIVE. 79% through our digital solutions. And in-person, only 54%.
One of the things that we've been talking to clients a lot about lately is, "Yes, we know you all want to get back together in person. And we all want to see each other." And there's great value in bringing people together. But if you're looking to drive behavior change, you may want to continue to focus on virtual or digital learning solutions delivered in the right way in order to get the results you're looking for.
Again, we'll talk a little bit about how you can achieve that, because there's some really interesting design decisions and some signs that Emma, Drew and Katherine will be sharing as well.
And one last question, and then we'll get into it. Here's a question about a client we had. One of our clients had 130,000 employees, and they wanted to do a non-mandatory training program. And so, think about your own organization. Imagine you have ten thousand, or a hundred thousand, or five hundred thousand people in your organization, or fifty. If you did a non-mandatory program, what sort of enrollment do you think you would get for completion in those first 30 days? For this particular client, we actually got 96% enrollment. There are techniques you can use not only to drive more behavior change, but also drive more people into enrollment.
And one of the things that we found is making mandatory training, or making training mandatory, sometimes leads to less behavior change. Because people are sort of frustrated that they're even there because they're being forced to. And Katherine's got some great data on that as well that she can share.
That is my little introduction right now. Kind of excited about what's possible. Now, it's up to Emma, Drew, Katherine to tell you how to really do it. Emma, can I hand it over to you?
[00:10:41] E: Yep, absolutely. Thank you, Marshall, for the introduction. Yeah, I think I’m going to kind of start by taking a little bit of a step back in terms of how we think about designing programs to be measurable and to be successful at the same time. It seems that when we're thinking about instilling organizational change, what we're really looking to do is to design a program that can kind of equip employees with the ability to fundamentally change their behavior.
And what we've found is that the best way to ensure success of any change initiative is to build behaviors that are reliable, that are sustainable and easy to recall, especially under pressure, in all different kinds of pressures. And so, in other words, what we're really designing programs to do is to build a strong set of habits.
What I’m going to do is just talk a little bit about the science of habit formation. Why it happens? Why we naturally build habits? And how to best do it given the literature. What we do know is that habits are the key to real sustainable strong change. But the way they're formed is through consistency and through practice.
What I’m going to start with is really kind of the science and neuroscience of habits in relation to the rest of our behavior. If you think about all of the behaviors that you engage in every day, you're constantly making decisions on what to do, how to respond, what to respond to, how you react to the environment, and how it links up with the rest of your motivational drivers.
However, over time some behaviors become habitual when they're frequently and consistently performed in a similar context and along similar goals. And this naturally happens over time. But what you end up with is a set of behaviors that are automatic, they're less effortful, seamless and somewhat mastered acts of what you can do.
If we were to kind of dive into the brain science of this in terms of how behaviors are executed, we're going to start by talking a little bit about the prefrontal cortex. And if you're a follower of NLI, you know that we talk about the prefrontal cortex a lot, because it's incredibly important for everything that we need to do at work. It's important in all of our executive function, activities, especially in terms of planning and executing novel behaviors.
And if you know us, we also talk a lot about respecting its cognitive capacity, because it isn't an unlimited resource. And so, while this area does set us apart from all other animals and that we can do all these complex behaviors, in light of this limited capacity, in terms of attention and working memory and all of this, the prefrontal cortex has also built up the ability to learn how to automate certain behaviors to the point where they're no longer taking up this precious working space.
What actually happens as particular behaviors are repeated and rewarded along similar expectations, and consistent environments, and along similar goals is that the systems that actually performs these behaviors shift in a way that kind of takes the weight off of the prefrontal cortex and places the behavior, even the networks that perform the behavior, in a place that's much more efficient that's closer to your motor output so you can respond faster. You can respond in an energy-efficient way. And you can do it in a way that frees up all of this mental capacity from your prefrontal cortex.
Essentially, what your prefrontal cortex is doing is it's kind of offloading all of this effortful mental activity from the cortex to be in an area that is much more efficient and can be performed without hijacking an area that you might need. If you were to ask why are we designing programs to build habits, it's because habits are formed. They're able to be energy savvy, fast, and efficient, and they can be sustainable over multiple environments in different pressures.
In terms of how habits are formed, we've all tried to build different kinds of habits. I’m sure, we've all tried to build healthy habits, healthy eating habits. Maybe drinking more water, whatever. They're not the easiest thing to form. It does take a certain amount of attention and repetition especially on the individual level. And this is because new behaviors do take a while to emerge because you are kind of rewriting all of the neural code that was prior reinforced to other habits. So, other habits that you're kind of overriding, habits that you want to override with new better habits. It can take attention and it can take repetition.
And we know that with enough attention, and repetition, and, actually, consistency, these are the most important factors for developing strong habits. But given all of this and given this research, we've developed a set of strategies to best facilitate strong habits in the way that really works with how our brains work. And so, this is on the individual level. We know that we can form new strong habits this way. But if you wanted to develop habits at scale, in addition to these kinds of strategies, there are other kinds of strategies that we kind of put in place into our programs to build the best strategies on a larger scale.
What we do know is that if you are to enable strong insight in the environment of building these habits, you're going to get the best motivation for driving new behavior. We know – And we talked about strong insights in the past, and we know that in situations where insights happen, especially if they're strong, you're going to be more engaged, you're going to have a better emotional experience, and you're also going to tap into your motivational drivers. So, that will continue to push an individual to keep performing this behavior.
We also know that we, as humans, are highly attuned to our social situation. We are very sensitive to social norms and we are very sensitive to behaviors that we want to kind of link up with our own social identity. For instance, there's plenty of research showing that if individuals see a group of individuals doing an activity, they're more likely to do that activity. So, "Hey, these individuals are hanging up their towel in the hotel room." You're more likely to do that. And so, that's something that we use to our advantage in social learning solutions. And that will kind of engage more people to perform that behavior.
And finally, given that we know how habits form in the brain, we also know that focusing on one habit at a time is the best way to instill strongest habits. This is the way that we design our programs to build habits that are built in the presence of insight in social situations and one at a time to build the strongest habits.
But finally, as I get to the end of my little spiel and habits and hand the mic over to Katherine, is the reason that we build these habits, as opposed to other complex behaviors, is because habits are the behaviors that we go to when situations change in pressure.
In social pressure situations, or performance pressure, which happens all the time at work, that the behaviors that we default to are those of habits. You end up with a set of behaviors that are consistent in differing levels of pressure, different environments, and you can measure them. This is the way that you're able to actually measure the success of your program because you do have a set of behaviors that you can go to and call to and they're going to be consistent.
But then, of course, the next step is what assays do we use to measure? And I think that's where I’m going to throw the mic over to Katherine.
[00:18:13] MB: I’ll hand it back to Katherine in just a moment, Emma. Thank you so much. And thank you for the insights. I’ll just highlight one last thing here. This part about what you want people to remember under pressure is incredibly powerful when having a conversation with business leaders about how they want to transform their culture. What sorts of behaviors they want?
One of the questions that we always ask. I’ve heard David Rock asked this to CEOs many times, "So, if things just got really crazy and all the pressure was on, what are the three things you want your team to remember? What are the three things you want your team to do?"
When the pressure hits, we default back to that unconscious behavior. And you want that unconscious behavior to be the things that you know are going to help your organization achieve its goals. The more clarity a leader can get on the behaviors they want in their organization, then we can go ahead and build programs and measurement strategies to make sure that that's embedded. When the you know what hits the fan, they default to that right behavior, the right habits, not the wrong habits.
I’m going to hand it over now to Katherine to talk a little bit about how do you actually build these programs and measure them? Katherine, over to you.
[00:19:16] KM: Yeah. Thanks, Marshall. One of the most common measures that we see clients using when they're looking to measure their learning programs, and it is a measure that we ask as well, is something called net promoter score, or NPS. This was developed by Bain. And it was built to measure brand awareness and market penetration. It's the question that you get, a lot of end of surveys after you purchase something from an organization, "How likely are you to recommend X company to a friend?"
NPS, the calculation of the score. Then you take the percentage of promoters. Those are the folks who say nine or ten. 100% I’d recommend. Anytime someone wants to buy a TV, I’m sending them to your store. And you subtract the detractor. Anyone who said, "There's no way I’m recommending." All the way up through. If, maybe, 50-50 kind of toss up. And you leave the passives, the people who answered that a seven or eight, out of the calculation. And that gets you the NPS score.
As I said, we do ask an NPS question, because we do like to know. How are people enjoying the program you? If they're going to recommend it, it means that they like it, right?
These are broken down by modality, by format of our behavior change program. The HIVE, the DLS virtual workshop and in-person workshop. Because you see that people tend to say, "Yes, I like the HIVE. The best. That's my favorite format," right?
The problem with NPS and with only using NPS is that it wasn't built to measure the effectiveness of training or of behavior change programs, right? At the end of the day, the reason that we spend money on training programs that we put resources to or that we spin up whole talent and learning teams is because we want people to do something differently, right?
And we tend to think that if a learner walks away from an experience saying, "That was great. I loved it. I had a fabulous time. I would definitely recommend it to someone." That that means the experience was successful. And that's what we call the conventional wisdom. But we're NLI, and we're not about conventional wisdom. We're about research. We're about the science.
What we've found in terms of the research is that positive behavior change is a successful learning experience. Again, as I said, positive behavior change. So, getting people to leave the program and do something differently is the most effective way to measure your return on investment. Are you getting what you want out of the program? Right? You wouldn't send people through this program if you didn't want them to behave in a new or a different way.
Okay. How do you assess then positive behavior change if it's not NPS? If we can't just rely on what people say they like? Well, we look at a few different factors. We look at preparation. Did the program equip them, equip the learner, to actually engage in new habits? Do they know what you want them to do? Do they have the tools that they need to do those things?
Where are they applying those learnings? Are they doing it in one-on-ones? In team meetings? Private? Public? All of that? Which helps you understand, again, have they learned what you want them to learn? And are they practicing enough in the situation in which they need to use the habit to have that build so that it's an automatic trigger when they encounter the situation that you're equipping them for?
What are the obstacles? Right? We all know the more difficult that you make something, the less likely people are to do it, right? So, we want to know what obstacles they're encountering so that we can then remove them or mitigate those obstacles.
And then finally and most importantly, frequency-based behavior questions. This is really where we like to focus. This is where all those great numbers that Marshall was showing you earlier are coming from are these frequency-based behavior questions.
An example of a frequency-based behavior question, from our decide program, how many times in the last week have you deliberately used a strategy to mitigate a bias? Decide is all about breaking bias, mitigating the effects of unconscious bias. The SEEDS model, which is based on, is NLI's proprietary model. The only one to mitigate bias that we know of out there in the world.
We asked how many times in the last week. And the reason that we focus on the last week is that's what people can most accurately and easily remember. The example that I like to use is, if I say, "How many times in the last week have you had chicken for dinner?" You're going to think back and you're going to say, "Okay, I had chicken on Monday. And I’m having it today. And, oh, yeah, and I had it on Wednesday. So, three times this week."
If I were to say, "How many times in the last month have you had chicken for dinner?" You're going to think about what happened in the last week and you're going to multiply it by four. Because people can't really accurately remember, "What was I doing three or four weeks ago?" Right? So, we want to get an accurate measure. So, we ask about a period of time in which people can accurately remember what were they doing? Where were they practicing things?
Then we look at anyone who says that they engaged in this behavior, deliberately using a strategy to mitigate a bias. Anyone who says I’ve done that one to three times a week or four to six times per week or more, that gets moved into what we call behavior change percentage. The percentage of people who are changing their behavior as a result of the program.
For a decide, we've got a sample size of a little over 16 000. We've got a BCP there of 79%. 79% of all of those 16,000 participants are doing something differently at least once a week or more as a result of this program. That's what we really like to focus on there. And that's what BCP is.
What we're able to do then is look at the BCP scores and compare it to NPS. What you'll see here is that, again, we're able to design programs that people like as events by the NPS scores there on the right, and that are effective. We've broken down NPS by format here as well. You see that the HIVES have the greatest level of effectiveness when it comes to changing behavior. In-person workshops, both least recommended and lowest level of behavior change. Still good. You're still getting over half of your people to do something differently. But it's not 88%. Right?
As I mentioned earlier, research really shows kind of people's willingness to recommend a program. How much people liked a program doesn't accurately predict their change in behavior. How often are they doing something differently?
And at NLI, we recently went through and made our own contribution to this body of research that proves that looking at, again, NPS scores and BCP scores. We've looked at data for the past four years. This goes over 259 clients. And what you're seeing here on the screen, this kind of scatter plot, is showing the correlation between BCP and NPS. How many BCP scores could be explained by the NPS scores? Right? And what we see is a bar of only about 0.3.
What that means is that only about 30% of the BCP scores could be explained by what the NPS scores were predicting there. Right? That's a fairly low correlation. We're really happy to see this just adding in there more, that when we want to look at effectiveness of training programs, we need to look at behaviors changed. Not whether or not people enjoyed the program. Or not simply whether or not people enjoyed the program.
Emma, was there anything that you would add to this?
[00:27:17] ES: Oh, yeah. I think that you're right on, that NPS shouldn't ever really be used alone. And I think that's great that we use multiple measures. But you can get some value from this just in terms of general attitude, especially in reference to the detractors.
If you were to look at the detractors, that can tell you a lot about whether or not you want to keep using the program the way you're using it. And research does show that attitudes or poor attitudes, if you ask somebody whether or not you will do something, if they are negatively responding, they're really very much likely not going to do that behavior. It's pretty predictable. If you're not going to do this, if you say you're not going to, you won't do it.
But whether or not you say you promote something isn't a great predictor of actually doing the action. But I think it's a great way to just get a general feeling for the experience going forward and just not the only measure. I think combining both of these is a great way to do it.
[00:28:11] MB: Thanks, Emma. And thanks, Katherine. Who defines what the right habit is, which is a very meta question? What I can tell you is I’ll give you by way of an example. It depends on what you're trying to accomplish. Right? For example, Katherine talked about a habit around mitigating bias. Right? What you need to do is you need to define what are the results you're trying to achieve? We want our organization people to be more inclusive. We want people to be more innovative. Whatever that goal is. And then you actually need to find a researcher like Emma, or Katherine, or our team, and actually go look at the research to find out what are the most important habits that you could practice that are going to get you the results you want.
The analogy that I like to use is, if you decide to yourself I want to be more healthy in my life. You could Google on the Internet be more healthy, and you'd come up with 18 million different habits that you could practice. Any one of those might be good. Some of them might be less good. What you need to find out is what the most impactful habits are for the largest group of people. And so, that's when we come up with the three habits of a program or the three behaviors that we want people to practice. That's based on research. Because those are going to be the habits that are going to have the biggest impact on the results that you're trying to achieve.
There's no right answer on what the habits are. But there are more effective habits than other habits. And I’ll pause. See if Katherine – Do you have any comments on that? Or emma as well?
[00:29:32] KM: Yeah. The only thing I would add to that, Marshall, is when we're looking at building our products, we're looking at what's the most effective habit to solve the problem that we're trying to solve? And also, how do we make that as memorable as possible? Which kind of links back to what Emma was saying earlier about habits need to be really easy to recall under pressure. Right?
And the example that I like to give about that is a story that I remember my mother told me when I was growing up, that when they launched the 911 program, that it was originally advertised as 9-11. As simple as 9-11, right? What they found was that people, in a panic, when they were trying to use the service were looking at their phones to try and find the 11 key. They weren't thinking. They were under such tremendous pressure they forgot there's not an eleven button. They had to change it to 911 because that was the simple, easy to remember and effective under pressure. When we're looking at that with program, we're looking at both effectiveness and ease of recollection under pressure.
[00:30:35] MB: Brilliant. Thank you so much, Katherine. Emma, anything to add?
[00:30:38] ES: Yeah. I mean, that's right on. And I would just say, in addition to that, just thinking about what the outcome you want to have, right? You're looking for some kind of outcome that's measurable. What habits get you there the easiest? I would say thinking about the outcome too is another piece of this.
[00:30:55] MB: Wonderful. Thank you. And then, Katherine, do you need to have some kind of baseline measure beforehand in order to see this behavior change? Maybe you or even Drew can talk about that.
[00:31:05] KM: The answer to that is generally no. When we're asking habit-based behavior questions, no, we don't do pre-measurement. And the reason for that is pretty simple, which is before you tell someone what you want them to be doing, before you define what you want the habit to be, how are they going to know to engage in it? You're going to get a low baseline because they don't know what the habit is that you're looking for. Right?
What we're looking to do is really instill new habits. We teach people what those habits are. We help them embed the behaviors that are going to like form into those habits and then we ask about the rate of practice of those habits over time. For things where you're just trying to do more of a temperature check of the organization, which is going to be something I talk about in a minute, we do a pre and post there. But for the BCP questions, it's really just a post.
[00:31:57] MB: Fantastic. Why don't we go ahead then? Because you're going to address a little bit about that now.
[00:32:01] KM: Yes. Okay. How do we ask? When do we ask this behavior change question? When do we ask NPS? What does this look like? At NLI, we have four core measurement offerings. It's our measurement portfolio. The first is the participant impact questionnaire or PIQ. This is what most clients would call something like a smile sheet. It's a survey that we deploy immediately at the end of training. Our facilitators actually ask this and give folks time to complete this five-question survey right when they finish a training.
We ask NPS. We ask about the value of the content that they've just been through. We ask about the facilitation and other things like that that we just are getting a sense. A snapshot in time of what was the participant's initial response? The value there is just getting confirmation, "Yes. Okay. People are seeing value. The facilitation went well." And we have a free-form field. W we'll use that if we need to make any little minor course corrections before we run the next session for the next audience.
The behavior change assessment, this is where we ask the BCP question. The how many times in the last week have you done X or Y thing? Right? That is deployed two to four weeks post completion of a training program. We have a self-report version that is 15 to 17 questions long depending on the program. We also have a 10-question version for each of our programs for people who like something shorter. And then we do have a direct report or colleague version of that survey.
We ask participants to self-report. And then we send a survey out to their direct reports or colleagues where we ask them, "How often are you seeing your manager engage in this behavior? Where are you seeing them practice this? Are you seeing your co-worker do X, Y, Z thing?" It gives us a more rounded view than just a self-report.
The behavior change pulse, this is a newer offer for us. I’m very excited about it. Drew's going to be talking us through some of the fabulous initial results that we're seeing from this. It's a five question, kind of truncated version of the behavior change assessment that we ask 3, 6, 9 and 12 months post completion of the program.
While the behavior change assessment gives us value by showing our habits being formed. Where are they being formed? Where are people practicing? The pulse helps show that those habits are being sustained over time. Right? That you're continuing to see value because you're seeing sustained change.
Finally, the organizational climate survey, or the OCS. This is one where we do have a pre or post. It could be used just pre, just post, or ideally both. This is to give you a sense of the sentiment within an organization about a given topic. We have an OCS for psychological safety, for bias, for growth mindset. It's a short, generally 10-ish, 15-ish question survey where we ask about employees lived experience of a given topic.
That helps you get a sense, for example, of what is the employee's lived experience around growth mindset in your organization right now? Then you run a training intervention. And then what's their sense? How are they feeling about your organizational growth mindset culture post?
Ideally, these offers are combined, right? What you'll see here is what we call you know a minimum or a minimal measurement approach. This is what we would want clients, bare bones, to do.
At the end of the program, at the end of a DLS, for example, we would deploy the PIQ where we ask them to give their sense of how the facilitation was, the content, all of that. Ask the NPS question for the first time. And then, two to four weeks following that program, we give to participants the behavior change assessment. And then two to four weeks after participants get the behavior change assessment, we send it to their direct reports and colleagues. And the reason that we wait there is just to give those direct reports or colleagues more time to actually interact with the participant, right? To make sure that they've had enough interactions that they can give an accurate gauge there. This is the minimum measurement approach.
What we're really excited that we're starting to see more clients do is this robust or kind of platinum approach to measurement, right? They are doing an OCS pre-training intervention. They're getting a sense of how people are feeling. Are they feeling psychologically safe, for example, in the workplace? They deploy their behavior change program. We get the PIQ. We get that snapshot kind of moment in time. Then they deploy the behavior change assessment. Participants report how are they engaging with habits. Colleagues are saying what they're seeing. Then we do the pulse, 3, 6, 9, 12 months out. Are those habits being sustained? What are we seeing?
And then we do the post-OCS. Okay. It's been a year. You've run this training program. You've got people engaging in new habits. What are you seeing now in your organization? What are employees reporting now?
This is kind of, overall, how we like to talk to our clients about measurement, is working through kind of those four different types of offers. Talking to them about what are they really looking to measure engage and then put together an offer specific for them that's going to give them the results that they need to get an accurate view of what's actually happening within your organization around a given behavior change program?
Thanks, Marshall.
[00:37:38] MB: Wonderful. We're going to go over to Drew in just a moment. I do want to say there are a couple really good questions around neurodivergence, neurodiverse, different brains. How do we measure? I think those are really complicated topics that I don't know that we'll have time to do.
I do want to call a couple things. Number one is what are the actual questions? Katherine, I think you shared. But maybe, Drew, when you do your sec, you could talk a little bit about the specific questions that get asked. And then last but not least, Katherine did mention something I think is really important, which is our organizational climate survey around psychological safety. I know that lots of organizations right now are really interested in understanding. If they're creating psychological safety for their people, how safe the people feel?
If you're interested in an organizational climate survey on psychological safety, just type in the chat right now. Say psych safety, or psychological safety. We'll be happy to follow up with you. Because I just know that that's something that many organizations are really striving to achieve. We think it's really important. And we'd love to help you on that journey as well.
[00:38:38] KM: Just in the interest of time, I’d love to kick it over to Drew so that we can see kind of how these offers and what this kind of like manifests as for clients.
[00:38:48] DS: That's great. Thank you, Katherine. And really, this portion is used to pull together where the rubber meets the road so to speak. Based on what Emma brought to in terms of habits. Based on our approach outlined by Marshall and Katherine.
We want to tell some client stories. Capital One, we've been in partnership since 2016. A lot of you might know them. If you've watched the TV in the last five years, you know what's in your wallet. Right? A large financial services firm, 50,000 employees. And we've been partnering with them since 2016 in terms of their diversity inclusion and belonging initiative.
And how they've done this to create a culture of belonging is through this framework you're seeing right here, create, attract and ensure. Creating that culture belonging, developing the talent through diverse means. And then, of course, creating the systems and ensuring fairness and equity throughout.
And what we've done is we've paired each one of these with our program offerings, include, which is about optimizing inclusion through use of the NLI's SCARF model. Decide, Katherine mentioned this, it's the SEEDS model, to label mitigate and engage the teams to really normalize the conversation of bias. And then differentiate, which essentially blends SCARF as well as SEEDS with a focus on performance and fair performance evaluations.
We've rolled out three different programs. And when we look at the data here, we did this through the terms of our DLS. It's bite-sized, chunked-out, video, practice tool, research summary type learnings set to take place over a four-week span. And we leveraged the participant impact questionnaire for this example. Being that the participants went through the program. They come out of a 60-minute capstone session that basically consolidates what they've learned, as well as how they've been applying. And then we measure it out, level-one type feedback, to understand the participants program sentiment. Immediate coming out.
You have the content, the facilitation. Whether it resonated. The actual value they felt from that program initially. And what we see here is it's really hit home within the 4.6, 4.8 range for content, facilitation, value. And again, this is a sliding scale from one to five. And I also want to note, very positive NPSs right here. Net promoter scores,77, 81. These are very, very strong.
But as Katherine mentioned, as well as Emma alluded to, we don't always hang our hat on the net promoter score. Right? Our goal is to look at long-term behavior change. Through the behavior change assessment, which took place two to four weeks upon that capstone session, we're still seeing that 67 of the reports use the strategy to mitigate bias.97%applied positive SCARF signals to be more inclusive. This was through include. And 97% feel more effective as a result of the decide and include program. That's very, very strong data.
And we're very happy that we've had a partnership, and we set this up for success, and we really drove it forward. And you can really see here how both the PIQ is used from the direct outset, all the way to the behavior change assessment.
That's Capital One. I would like to take us forward to an oil and gas industry client, energy conglomerate, so to speak. About 40,000 global workforce. And as we look here, we could start out with the clients measures of success. They had some very clear objectives on what they wanted our grow program, which is around growth mindset, to achieve from an enterprise standpoint.
They want the culture to shift to exhibit more of a growth mindset. They wanted participants to leverage a growth mindset in the face of change, which is resiliency. And they wanted people to feel motivated to actually employ growth mindset, right? Maybe there's a psychological safety element here.
We worked with them. We fine-crafted this. We do this with all of our clients. Right? We want to understand what goals they want to meet, so then we can use the behavior change data to really paint a really strong narrative, a strong story, around where we're actually seeing positive change in terms of organizational push. Right?
And what we did in their case was we actually used the organizational climate survey. And this is the pre and post type survey. That's creating the baseline. Similar to one of the questions here. This is a series of five simple questions. And what we have here are these are the results around the agree. Right? So, we put forth a specific statement. In this case, we're seeing that respondents agreed with the fact that my manager is opening to meeting with me to discuss my progress and performance.
Pre-program, 84% are agreeing that their manager is open to meeting with them. However, post-program, we're seeing that 95% are now reporting that their manager is more open to meeting with them to discuss progress and performance. And we're seeing the same boost. There's a delta of maybe about nine to ten percent in each of these specific areas that my manager is now more regularly allowing me to pursue innovation, or professional development, or develop my skill sets. Right? That's a very strong indicator of positive behavior change.
Conversely, we also looked at what a lot of the respondents were disagreeing with. Right? Pre-program, 11% disagreed that their manager was open to meeting with them to discuss progress and performance. Now, we went back, they go through the program itself. And then what we see when we survey them again is that dropped all the way down to two percent. Meaning, that two percent now disagree that their manager is open to meeting with them to discuss progress and performance.
And again, it's the same delta. Roughly between 10%, 12%, in which we see this change. We want to see the lower numbers on the disagree on this right-hand column. And that's what we're, again, driving towards in terms of positive behavior change.
And if I could go ahead and take us forward to the –
[00:45:03] MB: Yeah, Drew. I was just going to say, what would be the home run before we let people start to wrap up?
[00:45:08] DS: Well, yeah, I know. I’m running out of time. I’d love to just push forward to the Lockheed Martin story, and really talk about the pulse surveying. And this was mentioned previously. Lockheed Martin, very similar to growth mindset. They're looking to institute a growth mindset culture. A growth mindset common language.
What we did, we sent them through the DLS as well. This is enterprise-wide. We're seeing great behavior change two weeks post-program, two, four weeks post-program. 85% saying they've shifted from fixed growth. 92% sharing mistakes. 95% discussing growth mindset with their colleagues. Right? That stacks up against our goal for the common language.
Now, what we did with them was we looked at these same respondents six months down the road. And we found that through a very strong sustainment plan, through a series of nudges, through a series of coffee house, keeping the initiative alive outside of those four walls of the DLS, we're seeing that six months down the line 79% are still reporting that they're shifting from a fixed to a growth mindset. 86% continue to discuss growth mindset with others. And 91% share mistakes and learnings with their colleagues. I think that's very positive that, hey, it's great to fill out the behavior change assessment two to four weeks out. But when we look down the line and when you pair it with a very strong sustainment plan, we can see sustained organizational change over time.
[00:46:36] ES: And the other thing that I just want to chime in there about response rates and survey fatigue, and those are very like hot topics as always in measurement, we actually had higher end size. We had more responses to the six-month pulse for Lockheed Martin than we had to the initial behavior change assessment. We were able to get more people to actually complete the survey six months out, which is fabulous. And really, I mean, stellar results. Like, you couldn't ask for better results from this six months after wrapping a program.
[00:47:11] MB: And Katherine, or Drew, is it essential to have sustainment? Or is it just an add-on? How does that help? Or is it really important?
[00:47:19] DS: Yeah, sustainment is critical. At the end of the day, if you want to see lasting organizational change, you want to spend the time. You spend so much time up front implementing and getting the whiff in from your stakeholders and announcing and communicating. Right?
Like, a lot of times we see the momentum dies out post-survey. Right? That's why the sustainment plan is super critical. We partner with clients to really build that out, what that looks like, how long, as we continue to drive towards the measurement success.
[00:47:49] MB: Wonderful. All right. Well, thank you, Katherine, Drew and Emma. Just real quick, if you're interested in any of these programs, I’ve actually got those behavior change data numbers for each of our programs. Please tell us if you'd like to speak with one of our representatives here around how you could bring some of these behavior change programs to your organization. We'd love to just have a chat. Just tell you more about BCP, more about these programs, etc.
Other things I do want to share with you, someone actually said, "Are we going to be down at ATD in Orlando next month?" Yes, we will have a booth there. We'll be there as. We look forward to meeting with you and having further discussions and letting you get a chance to actually see some of our tools, some of our programs, some of our measurement tools, etc. We'd love to have that conversation.
Also, we have a special group called NLI Insiders. And we have an Insider Exchange, which is basically a chance for us to hear from you all what problems are you trying to solve in your organizations that you can't find satisfactory solutions for? So, that we can go out and put folks like Emma, Katherine and Drew on the task of building new solutions for you.
Also, we are definitely hiring. NLI a fantastic place to work. If you want a chance to be a lucky guy like me and work with really smart people every day and learn new things, please view our careers website at neuroleadership.com. We would love to have you send an application. And we're actively hiring in all sorts of different roles.
And last but not least, we do turn all these Friday events into a podcast. There are over 60 episodes online that you can go back. If you'd like today, you may find even more interesting podcasts from the past that would be interesting to you as well. Go ahead and subscribe. It's one of the top 10 business podcasts on Apple's Podcast app. So, please join us there and become part of the NLI community.
And that really wraps things up for me. Thank you all for coming. Thank you to my panelists. Thank you for all the great chat and insights. And we look forward to seeing you at a next event. Shelby, over to you.
[00:49:46] SW: Thanks, everyone, again for joining us. And as always, we'll be back here next week. Have a wonderful weekend. And we look forward to seeing you again soon.
[OUTRO]
[00:49:55] SW: Your Brain at Work is produced by the NeuroLeadership Institute. You can help us make organizations more human by rating, reviewing and subscribing wherever you listen to your podcast. Our producers are Matt Holidack, Mary Kelly, and me, Shelby Wilburn. Original music is by Grant Zubritsky. And logo design is by Catch Wear. Thanks for joining us. And we’ll see you next week.
[END]