What is natural language understanding (NLU)?

Why is natural language understanding important?

Human language is typically difficult for computers to grasp, as it's filled with complex, subtle and ever-changing meanings. Natural language understanding systems let organizations create products or tools that can both understand words and interpret their meaning.

NLU makes it possible to carry out a dialogue with a computer using a human-based language. This is useful for consumer products or device features, such as voice assistants and speech to text.

How does natural language understanding work?

NLU analyzes data to determine its meaning by using algorithms to reduce human speech into a structured ontology -- a data model consisting of semantics and pragmatics definitions. Two fundamental concepts of NLU are intent and entity recognition:

1. Intent recognition is the process of identifying the user's sentiment in input text and determining their objective. It's the first and most important part of NLU, as it establishes the meaning of the text.
2. Entity recognition is a specific type of NLU that focuses on identifying the entities in a message and then extracting the most important information about those entities. There are two types of entities: named entities and numeric entities. Named entities are grouped into categories, such as people's names, business names and locations. Numeric entities are recognized as quantities, dates, currencies and percentages.

When given a natural language input, NLU splits that input into individual words -- called tokens -- which include punctuation and other symbols. The tokens are run through a dictionary that can identify a word and its part of speech. The tokens are then analyzed for their grammatical structure, including the word's role and different possible ambiguities in meaning.

For example, typing a request into a search engine for an "island camping trip on Vancouver Island Aug. 18" might be broken down as follows:

ferry tickets [intent] / need:
camping lot reservation [intent]
Vancouver Island [location]
Aug. 18 [date]

In this case, the person's objective is to purchase tickets, and the ferry is the most likely form of travel as the campground is on an island. Search results using an NLU-enabled search engine would likely show the ferry schedule and links for purchasing tickets, as the process broke down the initial input into a need, location, intent and time for the program to understand the input. Vancouver Island is the named entity, and Aug. 18 is the numeric entity.

NLU vs. NLP vs. NLG

NLU is a subset of natural language processing (NLP). NLP attempts to analyze and understand the text of a given document, and NLU makes it possible to carry out a dialogue with a computer using natural language.

While both understand human language, NLU communicates with untrained individuals to learn and understand their intent. In addition to understanding words and interpreting meaning, NLU is programmed to understand meaning, despite common human errors, such as mispronunciations or transposed letters and words.

The other subset of NLP is natural language generation (NLG). NLG systems enable computers to automatically generate natural language text, mimicking the way humans naturally communicate -- a departure from traditional computer-generated text.

Generally, computer-generated content lacks the fluidity, emotion and personality that makes human-generated content interesting and engaging. However, NLG can be used with NLP to produce humanlike text in a way that emulates a human writer. This is done by identifying the main topic of a document and then using NLP to determine the most appropriate way to write the document in the user's native language. Text is generated based on this decision.

For example, using NLG, a computer can automatically generate a news article based on a set of data gathered about a specific event or produce a sales letter about a particular product based on a series of product attributes.

Natural language understanding applications

The following applications are designed to understand language as humans do, as opposed to just a list of keywords:

• Interactive voice response (IVR) and message routing. IVR is used for self-service and call routing. Early iterations were strictly touch-tone and didn't involve AI. However, as IVR technology advanced, features such as NLP and NLU have broadened their capabilities so that users can interact with the phone system via voice. The system processes the user's voice, converts the words to text and then parses the grammatical structure of the sentence to determine the probable intent of the caller.
• Customer support and service through intelligent personal assistants. NLU is the technology behind chatbots, which are computer programs that converse with a human in natural language via text or voice. These intelligent personal assistants are a useful addition to customer service. For example, chatbots are used to provide answers to frequently asked questions. Accomplishing this involves layers of different processes in NLU technology, such as feature extraction and classification, entity linking and knowledge management.
• User sentiment and intent. Organizations can use NLU for customer sentiment analysis and intent. For example, NLU can help an organization parse through multiple comments on social media to determine if their customers hold a majority positive or negative sentiment toward the company or an offered product. NLU can gather and understand terms and phrases put into a website's search bar to better understand what a customer's intent is when navigating its website.
• Machine translation. Machine learning is a branch of AI that enables computers to learn and change behavior based on training data. Machine learning algorithms are also used to generate natural language text from scratch. In the case of translation, a machine learning algorithm analyzes millions of pages of text, such as contracts or financial documents, to learn how to translate them into another language. The more documents it analyzes, the more accurate the translation. For example, if a user is translating data with an automatic language tool, such as a dictionary, it performs a word-for-word substitution. However, when using machine translation, it looks up the words in context, which helps return a more accurate translation.
• Data capture. Data capture is the process of gathering and recording information about an object, person or event. For example, if an e-commerce company used NLU, it could ask customers to enter their shipping and billing information verbally. The software would understand what the customer meant and enter the information automatically.
• Conversational interfaces. Many voice-activated devices, including Amazon Alexa and Google Home, enable users to speak naturally. By using NLU, conversational interfaces can understand and respond to human language by segmenting words and sentences, recognizing grammar and using semantic knowledge to infer intent.

Learn more about how NLU and NLP are used to make chatbots smarter.