Experiment boldly, measure closely: A CIO's GenAI playbook

As AI technology evolves rapidly, constant experimentation can help IT leaders stay ahead of the pace of change.

At Samsara, a technology company that offers an IoT platform for industries managing physical assets such as vehicle fleets, CIO Stephen Franchetti has adopted a venture capitalist-like approach to internal generative AI (GenAI) implementation. He maintains a continuous funnel of AI experiments, knowing most will offer little return, but that a few initiatives can offer outsized business value. Examples of the company's successful AI projects include Samsara GPT, an internal AI assistant that taps company knowledge to increase employee productivity, and AI-powered help desks that automate routine IT requests.

In the following interview, Franchetti discusses key strategies and challenges around the company's internal GenAI deployments.

Editor's note: The following transcript was edited for length and clarity.

What is your internal AI strategy at Samsara, and how does it shape which projects you choose to pursue?

Stephen Franchetti: We have a two-pronged approach. One is like a venture capital mindset to how we think about experimentation. A typical venture capitalist is experimenting all the time and looking at potential investments. Ninety percent of their investments come to nothing but the other 10% take hold and provide the 100x return on their investment. I think about technology investments inside the organization in much the same way. We need to have a constant portfolio or funnel of AI experiments coming in -- so that's one aspect.

The other piece is a bit more traditional. In the last year or so, we've been applying more value metrics to AI. We're trying to get more specific about AI's ROI or value. As AI projects move through the experimentation funnel, we get to a point where we ask, 'What is the business problem we're solving? What is the proposed value associated with that? And can we quantify that?'

What is Samsara GPT and how does it help your employees?

Franchetti: It's one of the most successful and adopted things that we have done since I've been CIO. CIOs lament about the fact that no news is good news -- if you're not hearing anything around the technology, then you're doing a good job. Rarely do you hear a lot of praise, but we've seen a tidal wave of good tidings around Samsara GPT.

It's almost like a pre-trained, fine-tuned version of ChatGPT that knows about Samsara context. The interface is Slack. On the back end, we're using OpenAI as our large language model (LLM). Then we're connecting that LLM to multiple data sources, including our product knowledge base, sales playbooks and sales analytics tool, Gong.

Sales enablement teams identify the top 1% of sellers, codify the techniques they use and then train the rest of the organization on those techniques. We've been able to track those techniques, load them into an LLM and instantly make them available across the organization. It provides our sales representatives with answers at their fingertips. We saw a 16% increase in sales quota attainment for expert users who were using Samsara GPT.

Can you tell me about the automated help desk you implemented?

Franchetti: On the internal side, we've implemented a technology that is essentially a support agent that offers 24/7 support for IT-related questions. We automate and divert a significant portion of the support requests that come into our IT support desk from employees. The level of positive sentiment from our users has increased simply because they're receiving support whenever they need it.

On the customer support side, we've implemented technology that can deflect almost 60% of chat tickets. We're freeing up support agents to focus on higher-value work and spend more time on the voice channel. Typically, when a customer calls in, they have a much more technical challenge than they could address in a chat.

We're also running experiments on AI voice capabilities. We're getting close to the point where these AI voice bots can interact with humans -- not quite as well as a human would -- but it's getting close in terms of response times and local colloquialisms.

How did you structure your teams and roles to best execute these AI projects?

Franchetti: For me, AI is everyone's job. It's not just the CIO's responsibility or the CIO organization's responsibility. The people best positioned to apply AI to their work are those working in the functions.

From a company standpoint, we take both a top-down and bottom-up approach. For instance, AI is very important to our CEO. People recognize this and understand that they should figure out how to use AI within the context of their work. That top-down approach has been a big advantage to us.

We also take a bottom-up approach with our AI champions network, which my organization facilitates. We've selected individuals from across the company -- including those in supply chain, finance and sales -- who are passionate about technology and truly understand the business. Those individuals are responsible for the AI roadmap for each of their functions.

What KPIs do you focus on to measure the success of internal AI initiatives?

Franchetti: It depends on the area of the business. For example, quota attainment is an important KPI for sales. We are applying AI software to drive that KPI, and the same logic applies to other areas of the business. Code generation is a very mature AI use case. Our developers accept approximately 40% of the code that our AI tools suggest, which is a significant percentage of the code in our codebase.

Supply chain KPIs are a little different. They're about how we compress time frames, such as delivery times, and how AI can help achieve those improvements.

What were the biggest challenges you faced during the two GenAI deployments that we touched on?

Franchetti: One is finding that sweet spot between deep understanding of the business and what's possible with AI. The processes we built for a non-AI world were suitable for that period. However, we're rethinking those processes completely, as AI gives us the ability to solve problems that we couldn't solve before.

This is why we were very conscious about selecting our AI champions. They require the ability to think differently in this new AI world -- and that takes education. I think this problem will be solved over time as more people become educated about AI.

The second is the readiness of AI software in the agentic space. AI excels in providing knowledge assistance. Samsara GPT is a good example of that. However, when it comes to taking complex actions in an enterprise setting, AI is not yet there.

What advice do you have for other CIOs thinking about deploying GenAI?

Franchetti: It goes back to the venture capital mindset. You must jump in, experiment and be bold -- but also be conscious about it. The easiest way to do it is with a top-down approach. Persuade your CEO to talk about AI all the time. That will grease the skids to drive AI into your organization.

However, CIOs must also lean in and create muscle around constant experimentation. The only way to keep up with the rate of change is to experiment and have an eye on what's going on in the industry.

Tim Murphy is site editor for Informa TechTarget's IT Strategy group.