Salesloft targets token effectiveness to measure AI's value

Identifying AI adoption rates and prompt use

To get a closer look at AI's value, Salesloft first measured internal AI adoption rates. Salesloft splits adoption rates into daily, weekly and monthly active users, Liebert said. The metric only looks at workdays, which differs slightly from other adoption metrics. Looking at user activity and dividing it this way is the company's first step in measuring AI value, he explained.

"Value comes when tools are utilized, and in order for them to be utilized, we need to see adoption," said Jesse E., vice president of product at Salesloft. "Tracking adoption [and] having a sense of who is touching some of these tools is a first gate to understanding [how] we are providing value … with the tools we've rolled out internally."

At Salesloft, adoption rates are up. Toil work -- traditionally manual, repetitive work -- is significantly done now with automation and AI, said Punit Sheth, principal lead engineer of the digital transformation team at Salesloft. "[This] really opens up folks to do the higher leverage, higher impact work for the organization," he said.

Many of Salesloft's AI enablement goals at an internal level focus on business process improvement, Liebert said. For example, one of the company's AI tools has an orchestration layer that connects multiple models and software, so employees can use the tool to address a sequence of related needs within a single business process.

With adoption rates up, the next set of metrics pinpoints what employees are doing with AI, down to the prompt level. Instead of only measuring overall AI use, Salesloft categorizes prompts by use case, Liebert said. For example, was this prompt a coding task? And in more detail, how many steps did it take the prompter to reach their result or to complete the business process? By grouping prompt use by task, Salesloft aims to get a better understanding of how employees are using AI in any given instance.

"[There] are all sort[s] of discrete steps along the way that we can actually track and understand not just who is using the tools that we're providing, but what are some of the differences in either time or the number of tasks that can be done by an individual utilizing these tools in those different steps of the workflow," E. said.

Token effectiveness is the new North Star

Examining prompts stemmed from the need for a new perspective on token use. One of the most pressing pain points in enterprise AI is token costs, which can quickly proliferate with no direct tie to whether that cost results in an ROI for the business.

Looking simply at token spend or adoption numbers doesn't indicate much, Liebert said. "It means that your people are using a tool, but it doesn't mean they're using it effectively."

To get closer to that effective use, Salesloft closely inspects the prompt -- or, token -- data to see how the AI tool was used, what for and whether that specific outcome was successful. By comparing outcomes to token input per use case, it is starting to better understand the ROI of its AI tools for internal employee use across a variety of tasks. So far, it has been a successful metric to measure value, Liebert said.

Tokens are the new energy, the new oil. You want to make sure you're burning just the amount to get that token spend to the best possible output.

Measurement of value involves an inference engine that evaluates conversation context to determine whether the prompter will use the AI's output, meaning it was a reasonable success and met the request criteria, Liebert explained. From that data, they score the prompts against a framework to assess how successful each prompt was relative to its cost, from both time and resource perspectives.

"A higher score in our rubric would be if you had fewer conversations to get a good outcome, versus longer conversations that might have burned more tokens and raised more resources," Liebert said. "Better output, faster is really what we're trying to get to. Faster in terms of fewer tokens spent … From a business perspective, that's resource savings, and that's really important."

The company is building out this token effectiveness framework, has the telemetry in place, is scoring and collecting data, and is rolling it out company-wide, Liebert said.

"Tokens are the new energy, the new oil," he said. "You want to make sure you're burning just the amount to get that token spend to the best possible output. That is the true North Star. And I feel like we're on the cusp, if we haven't already solved it, of how we identify that formula and get those ROI metrics finally in place."

Using metrics for AI upskilling

Salesloft has found that not all AI adopters are prompting efficiently. Looking at AI so in depth goes hand in hand with another important aspect of its AI use: embracing employee success and failure with the tool.

Inefficient use of AI is not met with repercussions, Liebert said. That is crucial. Salesloft's AI initiatives focus on growing from within, and that is what it aims to do with these token effectiveness metrics, he added.

Token effectiveness metrics are used as the foundation for Salesloft's AI upskilling portfolio, including hackathons, third-party training, education resources and an AI champions program. By identifying areas where tokens are used ineffectively, Salesloft has created targeted training and support networks to provide employees with feedback and help them improve their prompting skills, Liebert said.

AI champions program

As part of his role, Sheth leads the AI champions program. The goal of the program is to enable AI across the Salesloft organization by developing AI champions within each business unit, Sheth said. The AI champions work to identify areas that can benefit from AI and help their teams use AI for those use cases. The program currently has two AI champions for each of the 10 Salesloft business units, one primary and one secondary. There's also an opportunity to add more to each team, he said.

We provide the training to make sure that as an AI champion, you have the right training and tools available to help with your organization's business unit requirements.

Scott Kosciuk, senior vice president of revenue marketing at Salesloft, is an AI champion for his business unit. While he was initially interested in the role after it was announced, it also fell into his purview organically because he had a good understanding of his group's end-to-end workflows and use cases, he explained.

Employees who embrace technical change and understand their business unit well are prime candidates for the AI champions program, Sheth said. And, while it can be an added benefit, it's not necessary to have any technical expertise with AI to be part of the program.

"We're looking for people who have very strong indications and inclinations of wanting to be a part of that change management process," Sheth said. "We provide the training to make sure that as an AI champion, you have the right tools available to help with your organization's business unit requirements."

As part of that change management, AI champions spend time helping their teams improve their AI skills. For Kosciuk, this is a big part of the role: three-fourths of it involves enablement and sharing best practices with the team, he said. This involves helping his team with tool access, mapping data sources and instilling confidence when working with AI. "I'm also getting deputies working with me just because the movement is growing so fast," he added.

Sheth described the AI champions program as a collective think tank where employees are given the space to uplevel. While he isn't a champion himself, his work leading the program has helped him learn how leaders use AI effectively and translate it to others within the organization.

"The personal growth is huge," Kosciuk said when reflecting on his time so far in the program.

Getting the workforce on board

Getting 10 to 20 employees, many of whom are already leaders in the business, to buy into AI is one thing. But getting the rest of the workforce to engage with AI is another entirely. Part of what has made Salesloft successful so far is role-based training, targeted use cases and hands-on experimentation.

"One thing that's been really effective is role-based training," E. said. Not everybody uses the same [AI] tools in the same ways. Different roles and functions will utilize those in different ways or even different sets of tools."

Salesloft also tailors its AI use cases, so teams have a hand in developing how AI will be used for their specific workflows. The digital transformation team works with AI champions to identify the workflows that will have the most impact when automated with AI, Sheth said. They conduct ideation sessions to understand the tools and how best to accomplish those use cases, and then host hackathons for the business units, where employees build workflows and test different scenarios and edge cases.

Allow for that productivity to happen, have visibility of it [and] make sure you can pump the brakes if needed.

"I haven't seen too much pushback," Sheth said. "I've actually seen a lot of excitement around the use of AI. That partnership [between the digital transformation team and the AI champions] has really helped speed up the adoption of agentic use in our environment, because we are working with each team individually to identify what problem they are looking to solve."

There's going to be a level of discomfort at first when users build with AI and are from a traditionally nontechnical role, he added. "Having those sessions where you're hands-on-keyboard and actually working on it really helped rip that Band-Aid off."

Experimentation with AI has been part of the Salesloft zeitgeist since the beginning, so it quickly became a feature of business-unit hackathons. After selecting an AI vendor and LLM for general use, Salesloft made the conscious decision to give everyone a license to use the tool for experimentation, Liebert said.

"Allow for that productivity to happen, have visibility of it [and] make sure you can pump the brakes if needed," he added. "Get people to be productive, not afraid, and then help them along the way."

Olivia Wisbey is a site editor for Informa TechTarget's AI & Emerging Tech group. She has experience covering AI, machine learning and other emerging technologies.