AI Moderation

Automatically moderate chat messages using AI to detect and block inappropriate content in real-time.

AI Integration Quick Reference

let textMessage = new CometChat.TextMessage("UID", "Hello", CometChat.RECEIVER_TYPE.USER);

// Send message — check moderation status
CometChat.sendMessage(textMessage).then(message => {
  const status = message.getModerationStatus();
  // CometChat.ModerationStatus.PENDING | APPROVED | DISAPPROVED
});

// Listen for moderation results
CometChat.addMessageListener("MOD_LISTENER", new CometChat.MessageListener({
  onMessageModerated: (message) => {
    const status = message.getModerationStatus();
    // Handle APPROVED or DISAPPROVED
  }
}));

Supported types: Text, Image, Video messages only Statuses: PENDING → APPROVED or DISAPPROVED

AI Moderation automatically reviews messages for inappropriate content in real-time. When a user sends a text, image, or video message, it’s held in a PENDING state while the moderation service analyzes it, then marked as APPROVED or DISAPPROVED via the onMessageModerated event.

getModerationStatus() is available on TextMessage and MediaMessage objects. Custom messages are not subject to moderation.

For configuring moderation rules and managing flagged messages from the dashboard, see the Moderation Overview.

Prerequisites

1. Moderation enabled in the CometChat Dashboard
2. Moderation rules configured under Moderation > Rules
3. CometChat SDK version that supports moderation

How It Works

Step	Description
1. Send Message	App sends a text, image, or video message
2. Pending Status	Message is sent with PENDING moderation status
3. AI Processing	Moderation service analyzes the content
4. Result Event	onMessageModerated event fires with final status

Supported Message Types

Moderation is triggered only for the following message types:

Message Type	Moderated	Notes
Text Messages	✅	Content analyzed for inappropriate text
Image Messages	✅	Images scanned for unsafe content
Video Messages	✅	Videos analyzed for prohibited content
Custom Messages	❌	Not subject to AI moderation
Action Messages	❌	Not subject to AI moderation

Moderation Status

The getModerationStatus() method returns one of the following values:

Status	Enum Value	Description
Pending	CometChat.ModerationStatus.PENDING	Message is being processed by moderation
Approved	CometChat.ModerationStatus.APPROVED	Message passed moderation and is visible
Disapproved	CometChat.ModerationStatus.DISAPPROVED	Message violated rules and was blocked
Unmoderated	CometChat.ModerationStatus.UNMODERATED	Message was not subject to moderation (default)

Implementation

Step 1: Send a Message and Check Initial Status

When you send a text, image, or video message, check the initial moderation status:

TypeScript

const textMessage = new CometChat.TextMessage(
  receiverUID,
  "Hello, how are you?",
  CometChat.RECEIVER_TYPE.USER
);

CometChat.sendMessage(textMessage).then(
  (message: CometChat.TextMessage) => {
    // Check moderation status
    const status: string = message.getModerationStatus();
    
    if (status === CometChat.ModerationStatus.PENDING) {
      console.log("Message is under moderation review");
      // Show pending indicator in UI
    }
  },
  (error: CometChat.CometChatException) => {
    console.log("Message sending failed:", error);
  }
);

JavaScript

const textMessage = new CometChat.TextMessage(
  receiverUID,
  "Hello, how are you?",
  CometChat.RECEIVER_TYPE.USER
);

CometChat.sendMessage(textMessage).then(
  (message) => {
    // Check moderation status
    const status = message.getModerationStatus();
    
    if (status === CometChat.ModerationStatus.PENDING) {
      console.log("Message is under moderation review");
      // Show pending indicator in UI
    }
  },
  (error) => {
    console.log("Message sending failed:", error);
  }
);

Step 2: Listen for Moderation Results

Register a message listener to receive moderation results in real-time. The onMessageModerated callback receives a BaseMessage object:

TypeScript

const listenerID: string = "MODERATION_LISTENER";

CometChat.addMessageListener(
  listenerID,
  new CometChat.MessageListener({
    onMessageModerated: (message: CometChat.BaseMessage) => {
      if (
        message instanceof CometChat.TextMessage ||
        message instanceof CometChat.MediaMessage
      ) {
        const status: string = message.getModerationStatus();
        const messageId: number = message.getId();

        switch (status) {
          case CometChat.ModerationStatus.APPROVED:
            console.log(`Message ${messageId} approved`);
            // Update UI to show message normally
            break;

          case CometChat.ModerationStatus.DISAPPROVED:
            console.log(`Message ${messageId} blocked`);
            // Handle blocked message (hide or show warning)
            handleDisapprovedMessage(message);
            break;
        }
      }
    }
  })
);

// Don't forget to remove the listener when done
// CometChat.removeMessageListener(listenerID);

JavaScript

const listenerID = "MODERATION_LISTENER";

CometChat.addMessageListener(
  listenerID,
  new CometChat.MessageListener({
    onMessageModerated: (message) => {
      if (
        message instanceof CometChat.TextMessage ||
        message instanceof CometChat.MediaMessage
      ) {
        const status = message.getModerationStatus();
        const messageId = message.getId();

        switch (status) {
          case CometChat.ModerationStatus.APPROVED:
            console.log(`Message ${messageId} approved`);
            // Update UI to show message normally
            break;

          case CometChat.ModerationStatus.DISAPPROVED:
            console.log(`Message ${messageId} blocked`);
            // Handle blocked message (hide or show warning)
            handleDisapprovedMessage(message);
            break;
        }
      }
    }
  })
);

// Don't forget to remove the listener when done
// CometChat.removeMessageListener(listenerID);

Step 3: Handle Disapproved Messages

When a message is disapproved, you should handle it appropriately in your UI:

function handleDisapprovedMessage(message) {
  const messageId = message.getId();
  
  // Option 1: Hide the message completely
  hideMessageFromUI(messageId);
  
  // Option 2: Show a placeholder message
  showBlockedPlaceholder(messageId, "This message was blocked by moderation");
  
  // Option 3: Notify the sender (if it's their message)
  if (message.getSender().getUid() === currentUserUID) {
    showNotification("Your message was blocked due to policy violation");
  }
}

Complete Example

Here’s a complete implementation showing the full moderation flow:

TypeScript

    class ModerationHandler {
      constructor() {
        this.pendingMessages = new Map();
        this.setupListener();
      }

      setupListener() {
        CometChat.addMessageListener(
          "MODERATION_LISTENER",
          new CometChat.MessageListener({
            onMessageModerated: (message) => this.onModerated(message)
          })
        );
      }

      async sendMessage(receiverUID, text) {
        const textMessage = new CometChat.TextMessage(
          receiverUID,
          text,
          CometChat.RECEIVER_TYPE.USER
        );

        try {
          const message = await CometChat.sendMessage(textMessage);
          const status = message.getModerationStatus();

          if (status === CometChat.ModerationStatus.PENDING) {
            // Track pending message
            this.pendingMessages.set(message.getId(), message);
            return { success: true, pending: true, message };
          }

          return { success: true, pending: false, message };
        } catch (error) {
          return { success: false, error };
        }
      }

      onModerated(message) {
        const messageId = message.getId();
        const status = message.getModerationStatus();

        // Remove from pending
        this.pendingMessages.delete(messageId);

        // Emit event for UI update
        this.emit("moderationResult", {
          messageId,
          status,
          approved: status === CometChat.ModerationStatus.APPROVED,
          message
        });
      }

      cleanup() {
        CometChat.removeMessageListener("MODERATION_LISTENER");
      }
    }

    // Usage
    const handler = new ModerationHandler();
    const result = await handler.sendMessage("user123", "Hello!");

    if (result.pending) {
      console.log("Message pending moderation...");
    }

JavaScript

    class ModerationHandler {
      constructor() {
        this.pendingMessages = new Map();
        this.setupListener();
      }

      setupListener() {
        CometChat.addMessageListener(
          "MODERATION_LISTENER",
          new CometChat.MessageListener({
            onMessageModerated: (message) => this.onModerated(message)
          })
        );
      }

      async sendMessage(receiverUID, text) {
        const textMessage = new CometChat.TextMessage(
          receiverUID,
          text,
          CometChat.RECEIVER_TYPE.USER
        );

        try {
          const message = await CometChat.sendMessage(textMessage);
          const status = message.getModerationStatus();

          if (status === CometChat.ModerationStatus.PENDING) {
            // Track pending message
            this.pendingMessages.set(message.getId(), message);
            return { success: true, pending: true, message };
          }

          return { success: true, pending: false, message };
        } catch (error) {
          return { success: false, error };
        }
      }

      onModerated(message) {
        const messageId = message.getId();
        const status = message.getModerationStatus();

        // Remove from pending
        this.pendingMessages.delete(messageId);

        // Emit event for UI update
        this.emit("moderationResult", {
          messageId,
          status,
          approved: status === CometChat.ModerationStatus.APPROVED,
          message
        });
      }

      cleanup() {
        CometChat.removeMessageListener("MODERATION_LISTENER");
      }
    }

    // Usage
    const handler = new ModerationHandler();
    const result = await handler.sendMessage("user123", "Hello!");

    if (result.pending) {
      console.log("Message pending moderation...");
    }

Always remove listeners when they’re no longer needed (e.g., on component unmount or page navigation). Failing to remove listeners can cause memory leaks and duplicate event handling.

Next Steps

Flag Message

Allow users to report inappropriate messages manually

AI Agents

Build intelligent automated conversations with AI Agents

AI User Copilot

Smart replies, conversation summaries, and more

Send Messages

Send text, media, and custom messages