Processing time: Usually less than 2 minutes

Smart Topic Builder partially automates the manual topic auditing process described in Test and troubleshoot rules with the Preview panel. Smart Topic Builder audits and annotates up to the first 100 comments available in the preview panel that match an individual topic's rules.

To perform the audit, Smart Topic Builder evaluates matched comments for criteria in the topic description and annotates the comment with Yes if it deems the match correct and No if it does not. GenAI annotations are shown in the preview panel with the generative AI icon. AI-generated annotations can be edited manually. Any AI-generated annotations edited by a human user are no longer considered AI-generated, and no longer display the generative AI icon.

Note: Overriding an AI's annotation does not have any effect on the rules.

The genAI audit identifies under-performing topics and topic rules that are mis-classifying comments so admins can prioritize them for refinement. Smart Topic Builder shows example mis-classified comments and attempts to explain them to further enable manual rule refinement.

The genAI audit generates precision scores for audited topics. The topic precision score is the percentage of Yes annotations out of all annotations. Topic precision scores update automatically if a user manually edits comment annotations, and scores can also be viewed in the Topic Hierarchy tab as a column in the topics table. Topic precision scores are not available in reporting.

To initiate genAI topic auditing, use the Evaluate Comments with AI button displayed in the preview panel's Captured tab. The button is disabled unless the panel shows comment data in the tab. Initiating evaluation clears all AI-generated annotations before generating new ones.

If a Topic Description does not already exist for the topic, the Evaluate Comments with AI dialog appears to prompt the user for the definition components described in Topic rule generation and Topic definition guidance and best practices.

Topics do not need to be activated, and topic changes do not need to be published, to be audited. All rules in the topic are included in the audit; individual rules are not eligible for audit.

GenAI topic audit results can be exported in CSV format for external review. To export audit results, use the optional checkboxes to include Human-, AI-generated, or both types of annotation, then use the export icon next to the precision score. If no options are selected, the export includes both types of annotation.

For example, the following image shows the results of an audit for the Staff - Attitude topic.