Add Impact Analysis

Learn how to add impact analysis to an action plan.

Impact analysis enables you to calculate and track the changes in key metrics that result from your action plan. For example, you might want to trial a new service with a small group of customers before making the service available to everyone. You could create an action plan for implementing that trial program, and include analysis that calculates the impact of that service on Net Promoter Score.

Example of impact analysis on an action plan

Important: Enabling impact analysis in Action plans requires configuration to determine which metric fields are available, which fields can be used when defining control and impact groups, and the minimum sample size for displaying impact analysis data. For more information, see Configure action plans.

Milestone dates

Impact analysis requires two periods of time to compare, the baseline period and the test period. The baseline period is a span of time representing the environment before the action plan was implemented, while the test period is the span of time during which the action plan is tested. The difference in selected metrics between the two time periods is the impact of the action plan.

When setting milestone dates, consider these conditions for action plan execution:

  • An action plan starts on Setup Begins date.

  • Metrics collected between the baseline and test periods are ignored, since they do not accurately reflect either period. If you want there to be no gap of time between the baseline and test periods, make sure the Setup Begins and Start Test properties have the same date.

  • The plan implementation is complete and the impact period starts from Start test date onwards. All metrics collected from this point are part of the impact period.

  • Metrics are collected until End Test date.

Define the baseline period by entering dates for the Baseline Begins and Setup Begins properties. Define the test period by entering dates for the Start Test and End Test properties.

Note: Reaching the End Test date does not change the status of the action plan. The status must be changed manually.

Impact calculation

To quantify the impact of an action plan, select up to three fields that have numeric values. The change in these fields during the baseline and test periods represents the impact of the action plan. For example, you might want to see the impact of a plan measured in changes to NPS, Average Overall Satisfaction, and the number of records marked as needing attention.

Control and impact groups

Optionally, you can define control and impact groups of customers to better isolate the effects of the plan when gathering metrics. The impact group is a set of records representing customers affected by the action plan, while the control group is a set of records representing customers not affected by the action plan. This separation is most relevant when the action plan targets a specific audience, affects specific locations, and so on.

Define each group with filtering (based on Topics, Themes, and Segments) that restricts the records considered for that group. For example, you might create a group filtered for a specific unit.

Within each time period (baseline and test), metrics for the control and impact groups are likely to be different. Comparing the baseline score difference to the test score difference determines the overall impact.

Example

To illustrate impact analysis of an an action plan, consider this example:

To improve customer satisfaction, automotive company DriveMe is considering offering customers a complimentary car wash with every scheduled maintenance. To ensure that this additional expense produces the desired improvement in customer satisfaction before implementing the program in every location, DriveMe trials the program with two dealerships, DealerA and DealerB.

Milestone dates are defined as:

  • Baseline period — Begins January 1, 2020 and ends June 1, 2020. During this time, DealerA and DealerB are constructing car wash stations, but are not yet offering car washes to customers.

  • Test period — Begins July 1, 2020, and ends September 30, 2020. At that time, construction of the car wash stations should be complete. The one-month delay between the end of the baseline period and the beginning of the test period is a buffer to ensure that car washes are being offered just before the test period begins.

The metric used to quantify the impact of the plan is NPS.

The impact group includes only the dealerships offering car washes during the test (DealerA and DealerB). The control group includes all other units (DealerC, DealerD, and DealerE).

This impact was calculated as the the action plan progressed:

  • During the baseline period, NPS for the control group was 34, while NPS for the impact group was 26. The difference between groups during the baseline period was -8.

  • During the test period, NPS for the control group was 40, while NPS for the impact group was 38. The difference between groups during the test period was -2.

  • Moving from the baseline period to the test period, the gap between the control group and the impact group narrowed by 6 (from -8 to -2). That six-point positive change represents the overall impact of the action plan.

  1. Turn on the Enable Impact Analysis property.
    Enabling impact analysis when creating an action plan
  2. In the Milestone Dates section, specify the beginning and ending dates of the baseline period in the Baseline Begins and Setup Begins properties.
  3. Specify the beginning and ending dates of the test period in Start Test and End Test.
  4. In the Impact Calculation section, select up to three Metrics for quantifying changes due to the plan.
  5. Optionally, define control groups and impact groups to better understand the impact of the plan.
    For each group, select Topics, Themes, and Segments to act as filters for defining the group.
Continue with the action plan creation process in Create action plans.