That time we made it possible for Employees at Novo Nordisk to practice their leadership skills with AI
Novo Nordisk is a leading global healthcare company
Key challenges
- Expensive leadership programmes
- Incomplete leadership upskilling
Desired outcomes
- Learning results that can be applied
- A next generation of skilled leaders
Novo Nordisk's goal was to: Improve business performance by improving leadership skills and training in the organisation.
Improving leadership in a large organisation through technology
In a large organisation, there are initiatives and products to go to for most given situations. For leadership, there were already in place several programmes and learning modules on different levels. For the most part, a primary pain point was the specificity of these modules and the accessibility: Most modules were generic and they would take place at certain times. Our product positioned itself where no other tools could. It is a tailored (to your context) tool, available at all times.
In the product, we wanted to target those that were most likely to get something out of role play practice using AI personas. As part of our research, we interacted with emerging leaders, leaders, and senior leaders. Our findings were quite clear: senior leaders provided guidance for us and how they would approach such a challenge. Emerging leaders and leaders overwhelmingly saw this type of role play tool for practical use.
Hypothesised value by user group
Based on several interviews.
Validating our hypotheses using ChatGPT
Early on we had several ideas on how to approach role play using AI. Ideas ranged from a conversation with an AI employee, where a separate coach would hop in and guide you constantly, to a tool where you would be on your own until the very end. Providing feedback to leaders, storing that feedback, and using it was always a core element.
We had no intention to wait for polished interfaces to test our ideas. Therefore we designed a system prompt within ChatGPT that would emulate these experiences. Role play, coaching, and feedback were all tested within the confines of a normal ChatGPT chat with real leaders. Quickly we discovered that the real-time coaching was seen as useful by only a minority. With the added complexity, we scrapped it confidently.
Although the ChatGPT interface was limiting, the content of conversations held there raised a host of new questions. Such as: When does a conversation end? Who ends it? Conversations turn generic and general quickly - where do we get context from?
And indeed, we had no idea what quality level we’d be able to achieve once we could separate persona, scenario, user information and feedback criteria in a deployed application. More on this later.
Core design and interaction
The design itself takes shape around concrete work place scenarios, with personas that have varying traits. This positions the product for tailored practice where leaders can test themselves against a vast array of personalities. Users pick a scenario, and a persona. Each scenario has a concrete goal and description, that the leader has to solve. It’s slightly different every time - especially if you pick a new persona.
Hello goodbye
The “say goodbye” button is our solution to the “who ends this conversation” problem. We had several iterations of this. It was in fact one of the bigger interaction challenges we faced in the product.
We discovered early on that users with no prior knowledge of the product would not be aware that the conversation had an end at all. This is understandable, as no other chat products (e.g., ChatGPT) have an end. Conversations just go on forever, or until they compact. In our product, the conversation ending was a deliberate choice. The end would trigger a separate large language model (LLM) that would create highly detailed feedback based on different weights and Novo Nordisk skills. Also, the leader being able to end at the right time is a skill in itself. The leader has to ask themselves if they solved the problem, if they addressed the right items, and if they approached it in the humane way that they should.
Did we create a better environment for leadership development?
Every year, Novo Nordisk gathers results from employees in an engagement survey. One part of this survey has to do with leadership. Unfortunately, there are not many ways to measure the effects of such a product that we just made. Could we wait for the engagement survey results? How would we know that our product had anything to do with those results?
In the end, we measured the effects by gathering survey results from users of the product.
What users said in the survey
The conversations felt realistic
87 answersStrongly agree 25% · Agree 60% · Neutral 10% · Disagree 3% · Strongly disagree 1%
The feedback was actionable
87 answersCan implement immediately 25% · Can use in my work 61% · Might be able to use 9% · Cannot use it now 5%
It helps my leadership skills and confidence
87 answersSignificantly helps 11% · Helps me develop 46% · Somewhat helps 33% · Minimal help 3% · Does not help 6%
I will keep using it
87 answersVery likely 41% · Likely 36% · Somewhat likely 20% · Unlikely 3%