Humans and AI: The role of people in the new AI world

Under-involving humans adds risk

AI makes mistakes. It can exhibit bias, fall victim to security vulnerabilities, such as data leaks and injection attacks, and hallucinate output. As AI independence increases, such as with AI agents or reduced human oversight, these risks become more concerning.

Generative AI is also non-deterministic, said Jesse McCrosky, principal architect in generative AI at consulting firm Egen. It can give different answers to the same question, making it harder for humans to understand patterns and identify when it's wrong.

"You really have to think carefully about risk when you're deploying AI, and it's going to be interacting with real people or social systems," he said.

Sumit Johar, CIO at Blackline, a cloud-based software platform for financial services automation, also cited the possibility of third-party audits as a consideration when introducing AI into business processes -- and a key reason why humans need to be involved in AI workflows.

"In finance, accuracy isn't negotiable, so human involvement scales with risk and impact," Johar said. "The higher the regulatory audit or financial exposure, the more critical human validation becomes, even when AI is doing the heavy analytical lifting."

There are also broader societal risks to consider, such as the ethical implications of AI use ramping up without human oversight, McCrosky said. AI automation has the potential to change many aspects of human society. If unchecked, many fear what the future of an AI-enabled world would look like.

The evolving relationship between humans and AI

To support AI's growing autonomy, many business initiatives include requisites for human oversight. This combination, often called a human-in-the-loop approach, ensures that humans oversee and participate in AI workflows.

"Keeping that human in the loop is risk mitigation," McCrosky said. This approach ideally improves the accountability of AI systems by enabling humans to validate accuracy or step in when things go wrong.

Human-in-the-loop requirements also largely relate to AI augmentation. Instead of automating human tasks, AI augmentation means that AI bolsters human capabilities. It's a combination, not a replacement.

Keeping that human in the loop is risk mitigation.

McCrosky pointed to radiology to emphasize successful human-AI collaboration. Years ago, some believed the radiologist's role would be rendered obsolete because machine learning could read X-rays better than humans.

"It turns out, being a radiologist isn't just reading X-rays," McCrosky said. "It's also consulting with doctors about what sorts of scans to do, consulting with patients to explain results [and] digging in when something goes wrong or when a scan doesn't make sense."

By working in tandem with AI, radiologists now spend less time reading scans and more time on other important aspects of their jobs that require human expertise, he said.

While many employees fear job loss due to AI automation, research shows that complete job replacement isn't likely. In fact, AI and humans can be symbiotic: AI can augment human capabilities, and humans can not only oversee AI through human-in-the-loop governance but also complete tasks that aren't suitable for AI.

Some qualities are inherently human that AI can't replicate, said Isabella Loaiza, postdoctoral researcher at the MIT Sloan School of Management. Her past research found that workers more often augment their jobs with AI than are replaced by it. Signs point to human characteristics -- such as empathy, hope and trust -- that AI can't match, meaning elements of work still need humanness even as technology advances.

"Humans are better at navigating novel environments, conditions that you haven't been exposed to before," Loaiza added. "Machines are really good if the present or future looks like the past, but humans are better if that doesn't hold true."

Loaiza's recent paper, "The Limits of AI in Financial Services," found that these human qualities are especially needed in the financial industry. It is often easier for humans to trust other humans than it is to trust technology, and the high-risk industry of money management intensifies this need for human connection.

"The financial sector should be really careful about introducing technology that might erode trust," Loaiza said.

Where humans add the most value to AI workflows

Assessing the need for humans in AI workflows comes down to high-value tasks that require decisions, Blackline's Johar said. In the financial industry, this often refers to the many use cases where accuracy matters.

"Humans must remain in the loop when decisions involve judgment, risk tolerance or regulatory accountability," Johar said. "These are decisions where context, experience and business intent matter as much as the data itself."

When discussing the value of AI augmentation, McCrosky also pointed to financial services and to a recent client who wanted to automate paper processing with AI. Because the content that needed processing comprised bank and tax statements, it was a high-stakes use case; the AI tool couldn't afford to be wrong.

The financial sector should be really careful about introducing technology that might erode trust.

"We ended up building a confidence model to assess how confident the AI was about its answers," McCrosky explained. By identifying which cases the AI was sure about and which it wasn't, they created a hybrid tool that combined automation and human action working in tandem to ensure accuracy.

Aside from the higher-risk instances where accuracy matters greatly, human and AI collaboration can lead to success for other tasks, such as creative work. "There were significantly higher levels of human-AI synergy in creation tasks, so tasks that involve some type of open-ended response, versus decision tasks," Vaccaro said about her recent study findings.

Creation tasks offer an opportunity for easy division of labor, she explained. Many use cases for generative AI in content creation, such as creating images, text and other content, involve repetitive and open-ended tasks. In these cases, humans are adept at directing an AI system on what it should do and leaving other, more human aspects of a task to themselves.

"When it comes to open-ended tasks, especially creation tasks, have humans do part of the task, have AI systems do part of the task, and then have them combine and do some parts together," she said.

Designing human-AI collaboration with intention

It's not always easy to identify where humans should be in AI workflows, what they should do and how heavily they should be involved.

Experts suggest delegating tasks based on expertise and measuring human-centric performance to combine humans and AI in a thoughtful, effective way.

"Oftentimes, humans don't rely on AI appropriately, especially in decision tasks," Vaccaro said. Her research found that when humans alone are good at a task, they are more likely to make successful decisions about when to rely on AI.

"When they're not good at deciding when to or when not to adopt the AI's recommendation, that's part of the reason that there's no synergy in those cases," she said.

Delegation and triage could be a promising avenue to improve reliance between humans and AI in decision-making tasks, Vaccaro said. For example, when assessing medical images, a well-calibrated AI system should handle a task independently when it is confident. When it's uncertain, those cases should be triaged to a human reviewer.

Instead of a simplistic, rote back-and-forth between humans and AI -- for example, an AI system reviews a medical image, provides a recommendation, and the human reviews the recommendation and the image -- delegate some cases to AI systems, some to humans and some to both in tandem.

This can lead to cost and time savings, as well as greater accuracy, she said. Sometimes AI is better at assessment, and sometimes humans are better. Identifying and delegating each instance is important.

Businesses also need to invest in human-centric benchmarks to properly evaluate performance when humans and AI are working in tandem. "There's not enough emphasis on how to measure success," McCrosky said. "If you're not really careful about how you measure success, you can end up being surprised with what you actually end up getting."

Vaccaro also suggested the need for better measurement of collaboration. This can involve teaching employees how to properly assess AI output; evaluating progress on human-AI combinations; and developing human-centered model benchmarks -- rather than relying solely on benchmarks that focus on what AI systems do alone.

The future of human involvement in AI

As AI capabilities grow, humans' need to work with AI will increase, too. Thus, humans are adapting to make room for AI in everyday life and restructuring the status quo.

For instance, McCrosky pointed to the possibility of flipped classrooms -- where students listen to lectures at home and complete discussions and coursework in class -- to help students adapt to a world where AI is a collaborative partner to their education.

But future human-AI collaboration doesn't come without its continued risks and concerns. For instance, AI in the classroom can also be a threat. "If kids are just using ChatGPT to write their essays, what are we going to do?" McCrosky said.

To combat risks and concerns around AI, businesses are increasingly turning to AI governance -- a set of standards, policies and systems in place to ensure safe and successful AI use. Governance measures can include employee upskilling, responsible AI practices, data governance, risk management and the delegation of new roles or programs, such as AI centers of excellence.

AI governance is essential to successful relationships between humans and AI, Johar said. Few business professionals today know enough about governance to ensure it works and promotes collaboration. Effective governance can also mitigate skepticism and build trust.

Building trust is the key, he said. Moving forward, business leaders need to help their workforce embrace AI. This can be achieved by providing tools and information, ensuring data security and investing in trustworthy products.

"Trust is the foundation of AI-human collaboration," Johar said. "Without transparency, controls and accountability, even the most advanced AI will fail to scale in high-stakes environments like finance."

Olivia Wisbey is a site editor for Informa TechTarget's AI & Emerging Tech group. She has experience covering AI, machine learning and software quality topics.