Impact of data volume on accuracy and coverage

For high-volume systems, you can read many phrases to identify the words people use most frequently when discussing a certain topic, and then create rules that contain those words. Since there is an abundance of data in high-volume programs, topic implementations for these programs usually score high for both coverage and accuracy.

Alternatively, in systems with low volumes of data, there might not be many phrases about each topic. Consequently, the common words and language patterns in these phrases might not be apparent, and you might need to compromise on coverage in order to ensure that the topics you do create are accurate and useful. In particular, it can be difficult to achieve 80% (or even 70%) coverage for B2B programs, since these types of programs often have less data.

Estimating topic accuracy

Before you begin:

• Check that all topics have been activated and that no topics that should be activated are in draft mode.
• Start a spreadsheet to track the accuracy rates of the topics.
• Verify that a Responses Feed module is displaying Text Analytics data.

To estimate topic accuracy:

1. Navigate to the dashboard with the Responses Feed module.
2. From the Topic Filter, select the topic you want to test.
3. Set the Time Period filter to a time range that displays at least 100 comments. Leave all other filters set to the default.
4. Export the comments to an Excel file.
5. Mark the incorrect comments.
6. Count the incorrect comments and record this number, along with the overall number of comments reviewed for the topic, in a spreadsheet used to track accuracy.
7. For any topics with less than 80% accuracy, refine the rules for the topic and perform the above steps until the topic meets the minimum 80% accuracy rate.

Estimating coverage

After topic accuracy is complete for all topics and every topic is at least 80% accurate, you should boost coverage and perform coverage analysis.

Before you begin:

• Complete topic accuracy testing (see steps above).
• Check that all topics have been activated and that no topics that should be activated are in draft mode.
• Start a spreadsheet to track the coverage rates across all topics.

To test and improve coverage:

1. Boost coverage.

   1. From the responses report, filter for untagged surveys with comments by setting Topic Filter to Untagged surveys, and Custom Query to has_comment=1.
   2. Export 1000 surveys with untagged comments. If there are multiple distinct survey programs, you can opt to export a portion of surveys from each program.
   3. Identify which comments discuss a topic that was built in the system. If there are many comments that belong to a topic, this indicates that topic rules should be added or changed to include a larger percentage of relevant comments.

   4. When you discover untagged comments that should be captured by an existing topic in the system, add the necessary rules to the relevant topic to boost coverage. When adding extra rules to existing topics:

      1. Check that the rules you add are targeting patterns and do not pick up only one untagged comment.
      2. Check the accuracy of every new rule added and ensure it performs at or above 80%.
   5. Once all untagged comments have been reviewed and the necessary rules have been added, activate the edited topics to update the results in reporting.
2. Analyze coverage through recal distribution analysis.
   1. From the responses report, export 1000 surveys with untagged comments to an excel file.
      Tip: Filter for untagged survey records with comments. If there are multiple distinct survey programs, you can opt to export a portion of surveys from each program.

   2. Read each comment and record whether the comment:

      • Has no content (such as N/A, comments that list scores, and so on).
      • Discusses a topic.
      • Is an overly general comment that does not specify the subject of discussion (comments like great or easy that do not specify what was great or easy).
      • Is a new topic area.
   3. Examine the distribution of the comments and extend this distribution to the entire program to estimate the number of surveys with untagged comments that belong to the topic. For instance, if you read 1000 surveys with untagged comments and find that comments from 90 surveys should have been captured by at least one of the topics in your topic list, you can estimate that of the 25,000 total untagged surveys, 2,250 of them should be categorized by at least one topic.
   4. Calculate the true tagging rate and the overall tagging rate, as described above.