Data assessment

Data assessment is an important step for any Action Intelligence powered by Medallia AI implementation, and especially for proof-of-concept implementations for prospective companies. The quality of Action Intelligence results are dependent on comment density and quality, and are therefore vary highly from company to company.

As a result of assessing Action Intelligence data, you:

Determine whether the processed data is accurate enough for company stakeholders.
Identify potential model implementation issues, report configuration errors, or product defects so they can be addressed prior to notifying the the company that the implementation is complete.
Set proper expectations with company stakeholders.
Maximize the likelihood of a successful Action Intelligence implementation, with satisfied company stakeholders.

After completing the data assessment, communicate your results to the Medallia Data Science team.

Overview of the data assessment process

A data assessment includes these steps:

Configure Attention, Effort, and Suggested actions.
Process historical records.
Export data.
Compute metrics in the exported data.
Review data.
If needed, update the Action Intelligence configuration, and then perform another data assessment.

Export data

After processing historical records, export the feedback data. The data assessment should be based on records in English language only, so filter the records in your export accordingly. Ensure that the export includes these fields:


Category	Fields
Basic	The main score field All comment fields configured for AI `a_has_comment` `e_responsedate` `a_surveyid` `a_survey_language` `a_survey_program` `a_survey_type` `e_survey_source` `e_unitid`
Attention scoring	`a_attention_flag` `a_attention_label_legal_action_flag` `a_attention_label_churn_intent_flag` `a_attention_label_eroding_faith_flag` `a_attention_label_unresolved_issues_flag` `a_attention_label_positive_opportunity_flag` `a_alert_type` (if the company is using Closed Loop Feedback alerts) `a_has_alert` (if the company is using CLF alerts)
Customer effort	`a_customer_effort_bucket` `a_customer_effort_score`
Suggested actions	`a_actionability_flag`

For more information, see Exporters.

Compute metrics

Based on the data in your export, compute metrics for attention scoring, customer effort, and suggested actions.

Attention scores

Perform two analyses that determine how many records have been flagged for attention; once for records with comments (a_has_comment = 1), and again for records that have no comments (a_has_comment = 0). In each analysis, compute these metrics:

Number of records processed by the attention model (a_attention_flag is not empty)
Number of records flagged for attention (a_attention_flag = Yes)
Number of records processed and flagged for attention, but have no attention types (a_attention_flag = Yes, a_attention_label_legal_action_flag = No or blank, a_attention_label_churn_intent_flag = No or blank, a_attention_label_eroding_Faith_flag = No or blank, a_attention_label_unresolved_issues_flag = No or blank, a_attention_label_positive_opportunity_flag = No or blank)
Number of records processed and not flagged for attention (a_attention_flag = No).
Number of records processed and tagged with Legal Action attention type (a_attention_label_legal_action_flag = Yes)
Number of records processed and tagged with Churn Intent attention type (a_attention_label_churn_intent_flag = Yes)
Number of records processed and tagged with Eroding Faith attention type (a_attention_label_eroding_Faith_flag = Yes)
Number of records processed and tagged with Unresolved Issues attention type (a_attention_label_unresolved_issues_flag = Yes)
Number of records processed and tagged with Positive Opportunities attention type (a_attention_label_positive_opportunity_flag = Yes)

In records that do not contain comments, any record flagged for attention should have a main score of 6 or less (on a 10 point scale). Additionally, none of these records should have a specific type of attention flag.

If your company is using CLF alerts, perform additional analysis to reveal specific places where attention flagging provides benefit, especially as compared to the company’s existing CLF alerts. Compute this data twice; once for records containing comments (a_has_comment = 1), and again for records that have no comments. (a_has_comment = 0). In each analysis, compute these metrics:

Filter by the main score field to calculate separate metrics for each main score value
Filter by various combinations of a_attention_flag and alert a_has_alert values to calculate:
- Number of records that have an attention flag but no alert
- Number of records that have alerts but no attention flag
- Number of records that have both alerts and an attention flag

Customer effort

Perform analysis to show records based on their customer effort classification. Perform this analysis only on records that contain comments (a_has_comments = 1). Compute the number of survey records with each value of the a_customer_effort_bucket field (very_hard, hard, neutral, easy, and very_easy). Most records should have a value of Neutral.

Suggested actions

When analyzing suggested actions data, focus only on records that have at least one suggestion (a_actionability_flag = 1). Compute these metrics:

Percentage of all records that contain at least one suggestion
Percentage of records with comments (a_has_comment = 1) and that contain at least one suggestion

Review data

Review the data from your calculations in the previous step. Additionally, use the Data Analysis Data Analysis to refine your analysis. Validate these metrics:

Number of records — Ensure all records are reflected in the dashboard.
Number of English records — Use the control panel to show only records with English comments.
Number of survey records with comments — Use the control panel to show only records containing comments. Note that it is possible for a record to contain comments, but to have no comments for the specific fields used by Action Intelligence.
Number of records flagged for attention — Use the control panel to show only records with an Attention value of Yes. In the results, determine the percentage of records with comments that are flagged for attention, and the percentage of all records (with and without comments) that are flagged for attention. This gives a sense of how many records were flagged for attention based on default detractor logic (LTR <= 6) in the absence of comments.
Number of records flagged for attention with no attention types — Identify records without comments (or with comments but none in the fields used by Action Intelligence), that were flagged for attention using default detractor logic.
Number of records with the legal action attention type — Use the control panel to show only records with an Attention Type of Legal Action. This should be expressed as a percentage of English records with comments.
Number of records with the churn intent attention type — Use the control panel to show only records with an Attention Type of Churn Intent. This should be expressed as a percentage of English records with comments.
Number of records with the eroding faith attention type — Use the control panel to show only records with an Attention Type of Eroding Faith. This should be expressed as a percentage of English records with comments.
Number of records with the unresolved issues attention type — Use the control panel to show only records with an Attention Type of Unresolved Issues. This should be expressed as a percentage of English records with comments.
Number of records with the positive opportunities attention type — Use the control panel to show only records with an Attention Type of Positive Opportunities. This should be expressed as a percentage of English records with comments.
Attention flag by main score - For each main score value, this table shows the number of records flagged for attention and not flagged for attention. Use this table to identify these types of records:
- Records with a high main score (7 or more in a 10-point scales) that are flagged for attention are those that would typically be missed by alerts, but might actually require proactive follow-up.
- Records with low main score that are not flagged for attention are those that contain comments that are either not useful or are actually positive comments. In both cases, no follow-up is needed.
Attention flag compared to existing alerts, by main score — If the company has existing alerts defined, se how many records were identified only by their existing alerts, how many were identified only by the attention flag, and how many were identified by both. Use this information to have discussions with company stakeholders regarding:
- Records identified by only the attention flag. This shows the value of using verbatim-based alerting instead of the lower-fidelity alerting criteria most companies use today.
- Records identified by only existing alerts. Review these with company stakeholders to determine whether they actually require follow-up. Most require follow-ups, but some might be records that should have been flagged for attention. In that case, provide the examples to the Medallia Data Science team.
Customer effort metrics — Identify the number of records categorized for each effort category. Every record should have been processed for effort. The sum total of records across all categories should be equal (or very close) to the number of records. Most of the records should be marked for Neutral effort. Very easy and Very hard categories should each represent about 0.5% to 2% of all processed records.
Suggested actions metrics — Identify the number of records for which suggestions were discovered in comments. Use this metric to highlight the value of automated suggestion mining. Usually, only 1-5% of surveys with comments contain suggestions. Action Intelligence automates the analysis of comments to find these suggestions, so you need only to review them, not search for them yourself.
Number of records not processed for attention — Use the control panel to show English records with comments, and with an empty Attention value. This should return 0 records, since the Attention model processes all English records. If this number is relatively large (more than 1% of English records), contact Medallia Support.

During your review, determine whether anything seems very inaccurate or unexpected. Depending on the results of your review, you might need to make adjustments to your Action Intelligence configuration, and then run another assessment.

If no configuration returns accurate results, set proper expectations with company stakeholders. You might need to disable specific Action Intelligence features in the initial implementation, and then enable those features when the Medallia AI processing model has changed, or when the company has enough records to improve the accuracy of Action Intelligence processing.