In August, we published our first AI Proficiency Report, a data-driven assessment of AI proficiency in the workforce. This immediately begged the question: How AI proficient are we at Section?
We’ve been talking the talk, but have we been walking the walk? For the last 18 months, we’ve been working to make Section more efficient and productive using AI. We’ve given ChatGPT for Teams and Claude for Teams accounts to our 25-person team. And we’ve embedded AI in our business cadence with things like Lunch & Learns, shout outs to AI, and AI audits within our functional teams.
As COO, I want to know whether these investments are paying off. Enter the AI Proficiency Benchmark. Full disclosure: I spent four years before Section at Scott Galloway’s previous company L2, a research and benchmarking company. So I personally developed the Section AI Proficiency Benchmark at Section, based on my experience creating benchmarking products.
So I had two goals with this test – understand the ROI of Section’s AI investments and test this benchmarking product against a workforce I’m familiar with at Section. Here’s what I learned. (And if you’re interested in benchmarking your team, let’s talk).
Section is (unsurprisingly) ahead of the workforce
According to our last AI Proficiency Report, 7% of the workforce is an AI expert, and 25% is an AI practitioner (the two most sophisticated segments of the workforce). Section over indexes – 57% of our team are AI Experts and 43% are AI Practitioners. We don’t have any AI Experimenters or Skeptics. This was reassuring, given our investments in AI. It also makes sense – as a content company, everyone’s role is, to some degree, AI-powered.
Unsurprisingly, our Education team is the most AI proficient – their average score was 88% (the max score is 100%). These are our subject matter experts that develop our courses and, right now, most of that development is focused on AI courses. They’re researching AI and using AI to help them do their work. Marketing and Creative (also content-heavy departments) were also very strong – the average team member is an AI Expert.
These results were reassuring, but unsurprising – I expected us to be ahead of the workforce index. So I was also interested in benchmarking us against the AI Expert and AI Practitioner index – how do our AI Experts stack up against the average AI Expert in the workforce?
To do this, we looked at three sub-scores that we calculate: usage (self-reported AI usage data), knowledge (test results for questions on AI’s capabilities and responsible use), and prompting (performance in prompting simulations, scored by AI).
Looking at this data, our AI Experts and AI Practitioners actually use AI less than the average Expert or Practitioner in the workforce. But they over index on knowledge of AI concepts and responsible use, and our Practitioners also over index on prompting – they’re better at prompting than the average AI Practitioner.
Are we getting ROI?
When evaluating payoff from our investments, I was interested in two things:
- Are the use cases and tools that we promote internally being applied by our team?
- Are we getting productivity gains from these investments?
On the first, our internal dialogue and promotion of certain tools and use cases seems to pay off. For example, the majority of our AI shout outs usually focus on using Claude, and Greg and I both personally talk a lot about using Claude.
We can see this in our team’s usage. While we use almost every tool (except Pi and Copilot) more than the broader workforce, we really over index on Claude. 82% of our team uses it consistently – and it’s nearly 7x more prevalent at Section than in the broader workforce.
Similarly, Greg and I constantly promote using AI as a thought partner – we both teach classes on it, have written about it, and push our employees to use it. And it’s working – again, as a company we use all four AI use cases more consistently than the broader workforce, but our team is 2x more than the rest of the workforce to use AI as a thought partner.
This is especially true among our AI Practitioners – they use AI less frequently than our AI Experts (weekly vs. daily), and when they do, it’s usually as a thought partner.
On my second question, the AI Proficiency Benchmark isn’t a perfect measure of productivity gains. But we do ask employees to self-report their perceived productivity savings from AI. And again, we over index here. 50% of the Section team reports at least 10% productivity gains from AI, versus 42% who reported the same across the broader workforce.
If I assume 50% of the team is getting 10% productivity savings, and the rest of the org is getting ~5%, I can start to quantify ROI. I’m going to use placeholder numbers here, but let’s assume we’re spending $50/month/employee on LLMs – across 25 employees, that’s $15,000 per year. If we assume an annual employee cost basis of $3 million, and apply our 5-10% productivity savings, we’re saving about $225K in time costs per year. That’s decent ROI.
This is the case in theory – in practice, we think about AI platforms like ChatGPT and Claude like other project management and communications tools: (Asana, Slack, or email). At this point, these platforms are central to the day-to-day operations of our business – and we don’t currently crunch the numbers on Gmail’s ROI.
But the AI Proficiency Benchmark makes me more confident that we’re actually seeing real gains from AI, and using it effectively – a good return on our initial investment.
What to do next
18 months after beginning our AI journey, I’m happy with these results. But next year, I expect to see an increase in the portion of AI experts on the team – we’ll run the benchmark again in six months.
In addition, I want our AI Experts teaching the rest of the organization. So we’re integrating this into our next offsite, with two sessions focused on getting hands-on with AI. One will focus on how to build a custom GPT (since this is how so many of our AI Experts have found value). The other will be focused on auditing our workflows inside each team and finding additional places we can augment our work with AI.
I’d love to hear your thoughts and see if this work can be valuable for you. If you’re interested in running the benchmark with your own team, get in touch. And look out for an update to our AI Proficiency Report and broader workforce benchmark in the coming weeks – we just finished collecting data from 5,000 additional knowledge workers across the US, UK, and Canada.
