Today, most organizations feel unready for the challenges that Generative AI brings to risk management. Only 23% of leaders surveyed by Deloitte last quarter said their organization was highly prepared in risk and governance for rolling out AI.

Yet many of the best AI applications will have to navigate the line between value, user privacy, and risk. This was true for the non-profit Royal Foundation, which worked with business consultant Brian Kolodny to prototype an AI agent designed to detect suicidal tendencies in individuals online.

This week, we sat down with Brian to hear tactics the Royal Foundation used to navigate privacy.

The business challenge

Royal Foundation is a nonprofit founded by the Prince and Princess of Wales that focuses on addressing five significant societal challenges, including mental health.

In 2017, Brian’s team was hired to prototype a chatbot that could detect and engage with individuals showing signs of suicidal tendencies based on their online behavior. The goal was to identify concerning language or media shared in online conversations that indicated suicidal tendencies, and place subtle “interceptions” in their path to encourage them to seek help.

To do this, Brian’s team trained an LLM on real patient data and the analysis of that data from more than 140 PhDs and mental health professionals. This training data included text and visual documents – patient notes, pictures that patients drew, and other artistic impressions that depicted psychological distress. All of this was anonymized to protect patient identity.

Using this training data, the AI tool was designed to scan digital conversations on social media to detect discussions related to self harm. The chatbot was designed to work in the background.

“The best chatbots are invisible—they solve problems before the user even realizes,” Brian said.

Once the AI agent detected concerning language or activity, it was also trained to intervene.

If users didn’t seem to be urgently in distress, they were prompted to engage with mental health resources such as telehealth consultations, helplines, or licensed therapists. If the bot deemed that someone was in immediate danger, it would loop in a human and flag the interaction to a moderator who would call emergency services.

“When it’s a matter of life and death, you can’t afford to get it wrong.”

The ethical considerations

If you’re thinking this all sounds a bit intrusive, you’re not wrong.

“We were building something that could scan the digital void, looking for signs of distress and then route people to real help,” Brian explained. “It was very ‘Big Brother’-like, and we had to tread carefully with how data was handled.”

Brian’s team faced multiple ethical and risk challenges.

1. Protecting patients whose data trained the model. To do this, all training data was stripped of personally identifiable information (PII). However, the EU’s General Data Protection Regulation (GDPR) has additional strict guidelines for how even anonymized data can be used and handled. To manage this, the data was stored in a private cloud instance.

2. A sense of overreach. Beyond legal concerns, the team had to battle the perception of overreach. Therefore, the team decided this agent should primarily operate in public spaces (e.g. public social media posts, open forums) and avoid private messages to minimize concerns around government surveillance.

3. Bias in the bot. The team also introduced robust testing during training to ensure it wasn’t biased in how it chose the people to flag or inappropriately escalated normal conversations. For example, the bot was explicitly trained to understand the nuance of offhanded comments vs. genuine cries for help – e.g. “I’m so done with this day”.

Measuring success against ethics

In the end, the prototype was able to successfully identify high risk individuals based on online conversations and present them with options for help.

The agent could correctly interpret context – such as slang, idioms, or sarcasm – and filter out false positives. The engagement rate of targeted individuals with mental health resources was strong enough to indicate the agent was effective at reducing stigma and targeting users with the right kind of language.

For anyone developing AI experiences with a similarly sensitive nature, here are Brian’s tips:

1. Get the basics right – ensure compliance with data privacy laws. Depending on your training data, your AI agent might be subject to regulations that lead to big fines if not followed. Given the global reach of digital tools, design your solution to comply with the most conservative data privacy laws.
   
   
2. Consider training data beyond text documents. By including images and artwork created by people in the tool’s target segment, the agent was able to capture the full context of human expression in its training data – which is crucial for building ethically-sound AI tools that can truly understand human intent.
   
   
3. Binary decision-making won’t work for AI tools like this. When it comes to sensitive AI applications, false positives are not just an inconvenience, they can be actively harmful. Your testing should be focused on making sure the agent can handle a spectrum of graduated response levels and that it’s correctly classifying scenarios.