The move from established dashboards to conversational analytics gives employees more freedom to explore business data, but it also puts them in charge of evaluating answers they may lack the expertise to check.
For example, a question about quarterly sales might produce a convincing result, but leaves out which transactions were counted, how revenue was defined, or if the employee was entitled to see all the information used.
Data managers must make those choices visible as they help business teams adopt the tools, as errors that once could be caught while building and reviewing a dashboard may now surface only after someone acts on an answer.
Meanwhile, connecting the analysis to automated workflows further raises the stakes-- an incorrect calculation or overlooked access restriction can influence a business process before an analyst reviews the result.
Teresa Tung, global advanced data lead at Accenture, said conversational tools must first interpret terms that carry different meanings across departments.
“If it gets that interpretation wrong, everything that follows can be wrong too,” she said.
A reasonable answer can still be wrong
Sales and finance can ask for last quarter’s customer count and get different answers without either number being miscalculated. Sales may count everyone who made a purchase; finance may count only accounts with active contracts, Tung explained.
Unless the tool makes its interpretation clear, employees could mistake a disagreement over definitions for a discrepancy in business performance. An established dashboard, however, allows data teams to resolve those differences before publishing the figures, and conversational tools can leave that choice to the AI each time someone asks.
Megha Kumar, research vice president for analytics and AI at IDC, said errors in how datasets are combined can distort results without making them look obviously implausible.
“A bad join can count orders twice and inflate revenue or drop records that don’t match and leave out part of the business,” she said.
Kumar also cautioned a tool might use information collected for another purpose or filter results using a sensitive attribute. Checking whether the final number seems reasonable would not necessarily reveal either problem.
Users unfamiliar with the underlying data are particularly poorly placed to recognize those errors, explained Nick Maynard, vice president of research at Juniper Research.
“Without full familiarity on the data sources, an answer that meets a user’s preconceptions can be accepted at face value,” he said.
Permissions must follow the question
Conversational access can also expose inconsistencies between systems that previously served different audiences. An employee permitted to view purchasing trends should not gain access to individual customer records simply because a tool draws on both sources.
Sonal Ramrakhiani, CEO of LatentView Analytics, said organizations need to preserve existing restrictions as they introduce conversational interfaces.
“Companies need to make sure employees' access to sensitive information through a conversational tool is identical to the more well-established tools in the business,” she said.
Tung cautioned conversational analytics does not inherently bypass controls. The risk increases when identity and permissions are enforced inconsistently across the systems supplying an answer. Data managers working with security teams therefore need to examine access throughout the analysis, including the sources consulted and the information returned.
Retain enough evidence to challenge an answer
Before an analysis informs a consequential decision, users need to understand how their question was interpreted, which definitions applied and how current the data was. Tung said the evidence should also allow teams to reconstruct sources, transformations and lineage.
For automated actions, Kumar recommended rules governing which sources are acceptable and records connecting the action to the answer that triggered it. Investigators need the original question, requester, generated query, model version, permissions and resulting output.
Those records support different responsibilities. Data and analytics teams must investigate the analysis, while business owners review decisions affected by a mistake, Ramrakhiani said. Keeping only the final answer leaves both groups without enough evidence to establish what happened.
“That requires keeping a history of the analysis so the team can reproduce the result, even if the underlying data has since changed,” she said.