Disinformation and inappropriate content abound in digital environments, and users might struggle to determine the source of such content or how to filter it out.
Content moderation is commonly used across social media platforms, websites and other digital environments that host user-generated content. It enables the approval or rejection of comments, posts, images, audio and video users create and share. The task involves removing rule-violating content to ensure published material adheres to community guidelines and terms of service.
AI can aid in that process. It searches for, flags and eliminates content -- both human- and AI-generated -- that violates the rules or guidelines of a social media platform, website or organization. This includes any audio, video, text, pictures, posts and comments deemed offensive, vulgar or likely to incite violence.
What is content moderation?
Historically, organizations have moderated content with human moderators who would review most content before it published, said Jason James, CIO at retail software vendor Aptos. The moderators would check the content for appropriateness and either approve and post or disapprove and block it.
Until recently, users often did not know if their content was rejected or, if so, the criteria for the rejection process. The entire process was manual and prevented real-time responses to postings. Approval was also ultimately subject to a single moderator's decisions and leanings.
As a result, many organizations now use a mix of automated and human moderation, James said. AI typically serves as the first layer, filtering out spam and easier-to-identify violations, while humans review the more nuanced cases. That human layer remains important because offensive or misleading content can still slip through automated systems.
Automated moderation occurs when user-generated content (UGC) posted through the platform or website is automatically screened for violating the platform's rules and guidelines. If it does, the platform either removes it altogether or submits it for human moderation, according to Sanjay Venkataraman, former chief transformation officer at ResultsCX, a CX management vendor.
Strong content moderation guidelines can make it easier for organizations to adopt AI moderation tools.
6 types of AI content moderation
Organizations can use six moderation models to scale content review with AI, human moderators or both. Some models rely more on platform review before or after content is published, while others depend more heavily on users to report, rank or filter content.
1. Pre-moderation
To ensure content meets their guidelines, businesses can use NLP to look for words and phrases, including offensive or threatening words and terms. If the content includes those words, it could be automatically rejected, and the user warned or blocked from future postings. This automated approach limits the need for human moderators to review every post.
This type of moderation is an early method of machine learning (ML) for content moderation. The tool can review content against a published blocklist to ensure it does not contain forbidden words or phrases, James said.
An AI-enabled pre-moderation model automatically scans and evaluates content before it publishes, Venkataraman said. AI systems -- including large language models (LLMs), computer vision and content classifiers -- assess text, images, video and audio to determine if content goes against platform guidelines. If it does, like promoting hate speech, explicit imagery or threats, it is either blocked automatically or escalated for human review.
GenAI is not infallible. It can create hallucinations, which include false, misleading or incorrect information.
Jason James, CIO at retail software vendor Aptos
2. Post-moderation
Post-moderation lets users post content in real time without a pre-moderation review. After a user posts something, a moderator reviews the content. With this method, users could see content that violates community guidelines before a moderator notices and blocks it. This lets a user adjust any content deemed in violation so the content can publish afterward, James said.
AI systems and/or human moderators review this content after it publishes. AI automates the review, rapidly scanning new content in real time, flagging potentially harmful material for review or takedown, Venkataraman said.
3. Reactive moderation
This method enables users to serve as moderators, who review posts to determine if they meet or violate community standards. With this method, content could publish prior to moderation. This method crowdsources moderation to the community rather than relying primarily on dedicated human moderators. The community forums of many brands work this way, James said.
With reactive moderation, ML systems can prioritize incoming reports based on severity, content type and user history, Venkataraman said.
4. Distributed moderation
This approach is similar to reactive moderation, where users vote to determine whether a post meets or violates community standards. AI then promotes or suppresses content based on voting behavior, and can detect manipulation patterns or bias, Venkataraman said. The more positive votes received, the more users see it. If enough users report the post as a violation, it is more likely to be blocked from others.
Services like Reddit use this method to allow community engagement on content posted on the site, James said.
5. User-only moderation
This method lets users filter out what they deem inappropriate. Only registered and approved users can moderate content. If several registered users report a post, the system automatically blocks others from seeing it.
These systems are only as fast as the number of moderators available to review content. The greater the number of human moderators, the faster they can review and clear posts, James said.
Users set their own filters or preferences for what they do or don't want to see. Some systems hide content after enough user reports, with limited central oversight. AI can learn from user behavior and automate moderation based on individual preferences, such as muting or keyword filters, Venkataraman said.
6. Hybrid moderation
Generative AI (GenAI) is not infallible, James said. It can create hallucinations, which include false, misleading or incorrect information. With the potential for AI hallucinations, organizations still need humans to review content and make sure it's appropriate and accurate.
The hybrid blend of human and AI moderation enables both speed and accuracy. AI completes faster pre- and post-moderation, and human moderation has the final say to make sure content meets community guidelines while being logical and accurate.
How does AI content moderation work?
AI content moderation uses machine learning models, natural language processing (NLP) and platform-specific data to identify inappropriate UGC, Venkataraman said.
An AI moderation service can automatically make moderation decisions -- refusing, approving or escalating content -- and continuously learns from its choices. Moderation for AI-generated content is complex, and the rules and guidelines are evolving in tandem with the pace of technology, Venkataraman said.
"Content created using generative AI and large language models is very similar to human-generated content," Venkataraman said. "In such a scenario, adapting the current content moderation processes, AI technology, and trust and safety practices become extremely critical and important."
Additionally, AI-generated content is easy to create, and the amount of it online has increased dramatically since AI content tools have become publicly available. Human moderators now have to train to identify massive amounts of AI-generated content in order to weed it out and highlight actual UGC, according to Venkataraman.
As content can be created faster, the need to review and moderate content more quickly also increases.
Jason James, Aptos CIO
"The last thing any brand wants is to have a community area, a website or a platform filled with nothing but AI-created content," Venkataraman said.
As GenAI brings a lot of contextual understanding and adaptability into content generation, moderation tools must be reinforced with advanced AI capabilities to detect nonconformance, Venkataraman said. That includes training the AI models with larger numbers of data sets, using humans to validate a higher sample of content, collaborative filtering with community-generated feedback on published content, and continuous learning and feedback.
AI-generated content is massively increasing, and organizations must adapt to the rapid pace, James said.
"As content can be created faster, the need to review and moderate content more quickly also increases," James said. "Relying on human-only moderators could create a backlog of reviewing content -- thus delaying content creation. The delays created impact collaboration, ultimately resulting in a poor user experience."
GenAI has expanded what AI systems can do for content moderation. For example, multimodal LLMs can better interpret things like sarcasm, coded language or cultural nuance than traditional natural language understanding tools, Venkataraman said.
Hyperscalers, such as Meta, use custom deep learning models, image recognition and cross-modal AI that understands memes -- text plus images. YouTube uses pattern matching and object/image recognition to scan billions of video minutes daily. And TikTok uses multilingual, multimodal AI to do more nuanced tasks like detecting cultural norms, Venkataraman said. Additionally, video moderation tools can scan videos or audio files for copyrighted or inappropriate content.
How AI will affect content moderation
GenAI will continue to accelerate the evolution of content moderation, James said. As organizations produce and process more AI-generated content, they will face greater pressure to invest in moderation tools that can operate at greater scale and speed.
"AI will be more heavily used to not only create content, but [to] respond to postings on social media," James said. "This will require that organizations employ AI-empowered content moderation to not only automate, but also modernize their existing process."
AI can enable faster, more accurate moderation with less subjective review by human moderators, James said. And, as GenAI models evolve and become more advanced, content moderation will become more effective over time.
"Already, [AI] can automatically make highly accurate automated moderation decisions. ... By continuously learning from every decision, [it's] accuracy and usefulness can't help but evolve for expanded usefulness," Venkataraman said.
Some companies are already shifting more of their moderation work toward AI-assisted systems, James said. In 2024, TikTok cut hundreds of moderation-related jobs as it increased its reliance on automated moderation tools. With the massive increase in AI-generated content, that shift reflects a broader push to use AI to help review the content AI can now create at scale.
As AI models become faster and more accurate, organizations may reduce the number of human moderators needed for routine review tasks in the coming years, James said.
Editor's note:This article was originally published in 2023 and updated to reflect changes in the AI tool market.
David Weldon is a freelance writer in the Boston area who covers topics related to IT, data management, infosec, healthcare tech and workforce management.
