Understanding correlation and regression analysis

Understanding the correlation between events is informative, but you need regression analysis to help you prioritize impact.

Correlation

Correlation analysis helps you understand how predictive one variable is of another variable, independent of the size of the effect. It helps you answer the question: How reliable is the relationship between X and Y? This is entirely in relative terms. (For example, how much of the overall variance in Y can X be used to predict?)

To help you understand a limitation with correlation, consider the following example using measurements of leg length and skull size from a population of elephants. It is reasonable to suggest that these two variables are associated in some way, since elephants with short legs tend to have small heads and elephants with long legs tend to have big heads. We could, therefore, demonstrate that an association exists by performing a correlation analysis. However, it is meaningless to apply a causal regression analysis to these variables because they are interdependent. One is not wholly dependent on the other.

With this type of analysis in general, there could be a high correlation because both variables measure the same thing, because one actually causes the other, because a third variable causes both of them, or some other reason.

Regression

Regression analysis (also known as beta) helps you estimate the impact of the predictor variable on the overall metric, assuming there is causality. It helps you answer the question: How big is the relationship between X and Y? (For example, for a 1 point change in X, how much should we expect Y to change?)

To demonstrate the importance of regression analysis, consider the following example using two variables: crop yield and temperature. They are measured independently, one by a weather station thermometer and the other by scales. While correlation analysis would show a high degree of association between these two variables, regression analysis would be able to demonstrate the dependence of crop yield on temperature. The regression identifies the impact independent variables have on the dependent outcome variable.

With the regression analysis available in Action Planner, your customers can identify the impact drivers (such as answers to survey questions) have on key outcomes (such as the Likelihood to Recommend score).

Examples

Suppose a large change in X corresponds very reliably to a small change in Y. This would have a high correlation because knowing X allows you to predict Y very accurately. The beta on X would be low because the impact of X on Y is small). The standard error would be small because our confidence in knowing the precise size of the effect is high.

Suppose a small change in X corresponds on average (but not reliably) to a large change in Y. This would have a low correlation because knowing X does not allow us to predict Y with much confidence. The beta on X would be high because the average impact of X on Y is large. The standard error would be large because our confidence in knowing the precise size of the effect is low.

Key drivers

The most important step of this analysis is to estimate the relative impact of a change in each attribute's score on the overall metric. For example, you might want to understand whether a one-point increase in the room cleanliness score would have a greater or smaller impact on Likelihood to Recommend than a one-point increase in the breakfast quality score.

To do this, Experience Cloud conducts regressions of the key metric on each attribute, and records the coefficient (beta). Experience Cloud then ranks the attributes by their coefficient values from highest (such as Room Cleanliness: .88) to lowest (such as Fitness Center: .24), resulting in a rank order list with those at the top being the key drivers. Experience Cloud can chart these beta values against current scores, and generate a plot to show which drivers are strengths, which are weaknesses, and their relative importance.

Action Planner List report

The Action Planner List report displays a list of drivers and their likely effect on the outcome question, based on improving the drivers by the listed goal. It helps users prioritize improvement plans and estimate the potential impact of improvements. 

The following image shows an example of the Action Planner List report. To run a report, select filter options, choose an Outcome question, and then click Run. By default, the report shows the top 6 areas of improvement for the Outcome question you selected. Click Show More to include any drivers not initially displayed.

The report includes the following columns:

• Your Score — The average score for the corresponding impact attribute in the selected time period, as defined on the Reporting > Report Helpers > Time > Timeperiods screen.
• Change vs. Year Ago — The change for the attribute compared with the same time period, one year ago. If there is no data from one ago, this will be blank.

• Typical Improvement Potential — The year-over-year improvement that can be achieved typically by focusing on each driver, based on the Action Planner general statistical model. The first number represents the 50th percentile (median) level of improvement found among attributes that improved from similar starting points. The second score in the range represents the 90th percentile, an improvement score that would be better than 90% of the improvements found among comparable scores. Note that these values are estimates, and are not industry-specific.

  Note: As a basis for the Typical Improvement Potentials, Experience Cloud performs an empirical analysis of improvement levels across a range of metrics focused on service attributes. As part of this analysis, the improvement levels of the units (such as stores, hotels, agents, and so on) that improved were divided into three tiers: the 50th, 70th, and 90th percentiles of the improvement levels found. In Action Planner, these thresholds are used as the lower (50th percentile) and higher (90th percentile) improvement potentials, and as the Your Goal suggestion (70th percentile).

  For the Business-to-Business sector, Experience Cloud uses a general improvement algorithm based on a common observation: It is more difficult to improve high scores than low scores (and it is impossible to improve a perfect 10). The algorithm used in that case for the 0-10 scale is:

  y = (10-your score) * m%

  where

  m = 13 for the medium improvement (50th percentile), m = 20 for the difficult improvement (70th percentile), and m = 31 for the ambitious improvement (90th percentile).

  For example, a current score of 6.7 for an attribute would result in the following Typical Improvement Potentials:

  Typical Improvement Potential	Calculation
  Lower (50th percentile)	y = (10-6.7) * 13% = 0.43
  Your Goal (70th percentile)	y = (10-6.7) * 20% = 0.66
  Higher (90th percentile)	y = (10-6.7) * 31% = 1.02

• Your Goal — The default number here represents the 70th percentile of improvement. Users change the goal for each driver based on estimates about what improvements the business can achieve. Changing the goal automatically updates the Estimated Impact to your outcome variable. Click Sort to list the drivers from highest to lowest, based on the impact to your outcome variable. Click Reset to restore the original Your Goal values.

• Estimated Impact on [selected outcome variable] — The expected increase in the outcome variable should Your Goal be met. For example, in the screenshot above, you could expect the average rating score on the Likelihood to Recommend outcome variable to increase by .25 points if you increase the average score of the Ease to Resolve Request driver by .43 points.A driver's Estimated Impact is based on the beta obtained through regression analysis. Experience Cloud then weights that value based on the percentage of surveys that have a score for the driver. For example, if only 30% of survey takers answer the Agent Satisfaction question, Experience Cloud weights that attribute less heavily. Even if the driver has a large impact on the 30% of respondents who answered the question, the overall impact for all surveys is reduced. Estimated Impact results are not additive across drivers. The main focus of Action Planner reports is to help prioritize areas for improvement, not to estimate (predict) the total impact of all efforts and developments on the outcome variable.

  Note: For each customer who answered your survey, there exists a pair of points {Scoreattribute;Scoreoutcome} corresponding to the scores this customer gave for a particular driver and for the outcome variable you chose. Answers from the group of customers who took part in your survey then provides the whole sample of {Scoreattribute;Scoreoutcome} pairs. The chart shown below displays the attribute (driver) scores on the horizontal axis and the outcome variable scores on the vertical axis.

  

  A simple linear regression indicates the best fit for the data sample considered. From its slope (β)and the desired change in the attribute’s score (ΔScoreattribute), Experience Cloud determines the estimated impact ΔScoreoutcome(=β*Scoreattribute) on the outcome score. The latter is the Estimated Impact displayed in the Action Planner List report. 

  Regarding multiple regression scenarios and the correlation between drivers, the approach Experience Cloud takes to handle multicollinearity is simply to not run multiple regressions at all. It merely reports the betas from simple regressions of the outcome variance (such as Likelihood to Recommend) on each attribute individually. ( Experience Cloud multiplies the beta by the relative response rate for that question. Essentially, this assumes that a missing value indicates that the attribute is not applicable to that respondent, and therefore has zero impact.) Ultimately, the objective of Action Planner reports is to prioritize possible areas for improvement. They are not designed to create point estimates of how much an outcome would change if you improved multiple attributes at once.

Action Planner Plot report

The Action Planner Plot report helps users visualize quickly and comprehensively where their strengths and weaknesses lie so they can further prioritize where to focus their efforts. It displays the importance of each attribute (vertical axis) as a function of your performance (horizontal axis).

The horizontal axis can display Your Score or the Typical Improvement Potential. Toggle these by clicking an axis label and selecting an option.

The vertical axis can show either the importance (beta) or the importance (correlation). Toggle these by clicking an axis label and selecting an option. Consider referencing both options on the vertical axis. Use the beta to set priorities, which is the main goal of Action Planner reporting as a predictive model. Use the correlation to strengthen (or weaken) this conclusion. The Action Planner model implies a hypothesis that there is a causal relationship between the outcome and driver variable. Especially with small sample sizes, it is important to confirm that the potential driver has a reliable relationship (high correlation) with the outcome variable. If the correlation is not reasonably high, there is a chance that small sample size is driving the high beta. 

The image below shows an example of the Action Planner Plot report. To run a report, select your filter options, choose an Outcome question, and then click Run.

The report sorts drivers into four quadrants, each determined by selecting the median of the score (horizontal axis), and beta (vertical axis), respectively.

• Important Weakness (upper left) — These low score/high importance drivers are priority improvement opportunities.
• Unimportant Weakness (lower left) — These low score/low importance drivers are secondary improvement opportunities.
• Important Strength (upper right) — These high score/high importance drivers are areas of strategic advantage and should be maintained. These items could be used in marketing campaigns, for example.
• Unimportant Strength (lower right) — These high score/low importance drivers are typically business fundamentals or expectations. They might be necessary for business, but might not offer a competitive advantage.

The 6 drivers with the greatest impact on the outcome variable are highlighted with a gold Priority circle. These are the same top 6 drivers shown on the Action Planner List report.

At the bottom of the chart, select or deselect question groups to display or hide them. In the image above, the Agent Attributes, Key Metrics, and Priority groups of questions are visible. Hover over a question group at the bottom of the chart to highlight those drivers on the chart. Hover over a driver circle to view the following details about that driver:

• Your Score — The score for the driver, as submitted by survey respondents. This value is the same as the Your Score value in the horizontal axis of the chart.
• Importance (beta) — The importance (beta) score for the driver, used to set priorities for improvement. This value is the same as the Importance (beta) value in the vertical axis of the chart.
• Typical Improvement Potential — The year-over-year improvement to the score expected when given focus as an area of improvement. For more information, see Action Planner List report, above.
• Importance (correlation) — The importance (correlation) score for the driver, with higher numbers representing a stronger correlation between improving the driver score and improving the selected Outcome.
• Impact on [selected Outcome variable] — The expected increase in the selected Outcome variable should you meet your goal. For more information, see Action Planner List report, above.
• Sample Size — The number of samples used to generate the values displayed for the driver.

For example, the following image shows details for the Knowledge driver, with a selected Outcome of Agent Satisfaction:

The Action Planner Plot chart is interactive:

• Double-click one of the four quadrants to zoom in on that quadrant. Double-click again to zoom back out.

• Hover over a cell in the chart to make the cursor change to a hand, which will allow you to click and drag the chart.
• Click a grid line to drag and resize the respective axis.
• Click Reset at the top right corner of the chart to reset the chart to the default zoom and scale.

Design considerations

To provide accurate results, Action Planners require at least 50 responses with a score for the outcome variable as well as the drivers. If there are fewer than 50 responses, you might see the following error: "There is insufficient data to produce this report. Please widen your selection."

When choosing drivers to use, consider the following items:

• Same sampling scheme — Because Action Planners weight beta coefficients by sample size, only include drivers that have the same sampling scheme. For example, if one driver question is asked of all customers, while another is asked of only a subset of customers, do not compare drivers between the two groups. Doing so gives severely biased results. (An exception is if the question is skipped only for customers for whom the driver is known to be not applicable. In this case, it is safe to include that question alongside questions asked of all customers.) 
• Same level of granularity — Ensure that your drivers are of similar levels of granularity for comparison. For example, Quality of Bathroom Supplies is not at the same level of granularity as Overall Guest Room.
• Must be logical drivers — Only include drivers that logically drive the outcome variable. For example, if Overall Satisfaction is the outcome variable, it does not make sense to include Likelihood to Recommend as a driver since they are essentially two measures of the same concept, not drivers of each other.
• Same direction of scale — Ensure that all drivers have the same scale direction as the outcome variable. For example, a Level of Effort outcome variable where lower scores represent better experience should not be used with drivers where high scores are better (such as Agent Helpfulness, where 0 is not at all helpful and 10 is extremely helpful). In general, higher scores should be better scores for all drivers and the outcome variable. (If you ignore this best practice, some attributes might not appear in Action Planner reports.)

Action Planner reports are coded to hide any drivers that have a negative relationship to the outcome variable. Because answers within a survey are typically highly positively correlated, this problem rarely occurs with real data. If it does occur, it is likely due to one of the following reasons:

• You are using demo data, which is generated randomly and might not show the same positive relationships between variables that we typically see with real data. Some of these randomly generated relationships may be negative, and therefore are automatically hidden.
• You included drivers that have the opposite scale direction as the outcome variable.
• The drivers happen to have no true relationship to the outcome variable, and due to random variable are slightly negative. This is very rare with real data.

Configuring Action Planner reports

To configure Action Planner reports, start by selecting your key outcomes, the drivers of those outcomes, and determine how to measure the impact of your drivers on the outcomes. These are defined in report modules, which are automatically assigned to any Action Planner List and Action Planner Plot reports you create. Next, create the Action Planner reports and make them visible to users by assigning them to subtabs in report navigation. Lastly, assign Action Planner capabilities to the roles that should use Action Planner reports.

1. Select the key outcomes you want to improve:
   1. Open the Reporting > Reports > Standard Report Modules screen.
   2. In the  Filter by Kind / Workspace dropdown select Action Planner Outcome Field. to see which outcomes have already been defined. If you need to define new outcome fields, continue with the following substeps.
   3. Click New Standard Report Module.
   4. For the  Kind property, select Action Planner Outcome Field.
   5. Enter a  Name and a  Priority.
   6. Leave the  Create Skeleton property turned on, and then click Save. Experience Cloud automatically creates a child report item.
   7. In the  Member Roles property, choose the roles that will use this report module.
   8. Click Save.
   9. Select the new child report item.
   10. On the Add/Remove tab of the  Selected Fields table, select a field you want to use as your outcome, and then click Add.
   11. Change the Name to match your outcome field name.
   12. Click Save.
   13. Add other child report items, one each for the outcome fields you want to improve.  
2. Select the questions that drive the improvement of your key outcomes.
   Note: Experience Cloud does not support K-fields as driver fields.
   1. In the  Filter by Kind / Workspace dropdown select Action Planner Driver fields. By default, Experience Cloud includes an Action Planner Driver fields report module defined for all users. You can modify this report module and its child report item (as described below), or create new ones.
   2. Select the existing Action Planner Driver fields report.
   3. Use the MemberRoles property to choose roles that can use this report.
   4. If you want each driver to include its parent name in Action Planner, turn on the  Prefix Parent Text (Action Planner) property. For example, a hotel might have several restaurants and will ask guests to rate some of them. It can be confusing for the user to see "Quality of food & beverage" as a potential candidate for improvement in Action Planner since it could apply to any of the restaurants. The Prefix Parent Text (Action Planner) property clarifies the question by adding the parent name as a prefix, such as "R2 Quality of food & beverage" for restaurant 2. 
   5. Select the child Action Planner Driver fields report item.
   6. On the Add/Remove tab of the  Selected Fields table, select the question fields that drive your outcome, and then click Add.
   7. Click Save.  
3. Assign how to measure the impact of your drivers:
   1. Open the Reporting > Reports > Action Planner screen.
   2. In the list of fields, select one of your drivers.
   3. Select a PlannerMapping that defines how that driver impacts your outcomes. Experience Cloud includes several predefined mappings tailored for common questions (such as Bell Staff Friendliness, Room Cleanliness, and so on). For hospitality companies map your driver questions to similarly named improvement potential thresholds. These thresholds are based on an extensive Action Planner database for determining possible improvements and correlations to the outcome variables. For non-hospitality companies map your driver questions to the appropriate scale range:
      • (1-7 scale, 13-20-30%) for questions with a 1-7 scale
      • (-10 scale, 13-20-31%) for questions with either a 1-10 scale or a 0-10 scale
      • (1-5 scale, 13-20-30%) for questions with a 1-5 scale
   4. Click Save.
   5. Repeat these substeps for each driver field.  
4. Create the Action Planner standard reports:

   1. Open the Reporting > Reports > Standard Reports screen.

   2. Click New.

   3. For the ReportType property, select the Action Planner List report type.

   4. In the MemberRoles property, choose the roles that can use this report by moving them to the right-side box.

   5. Click Save.

   6. If you want to use both types of Action Planner reports, repeat these steps to create a standard report with a ReportType of Action Planner Plot.

5. Assign the standard reports to subtabs in report navigation as described in Navigation.

6. Assign Action Planner capabilities to user roles:
   1. Open the Company > Users > Roles > Roles screen.
   2. Select a role you want to use the Action Planner report modules you created.

   3. In the  MemberCaps property, choose the following capabilities:

      • View Action Planner
      • View Action Planner - ImportancePlot Tab
      • View Action Planner - List Tab
      

Removing access to legacy Action Planner reports

As you migrate away from legacy Action Planner reports to the Action Planner dashboard module in Medallia Alchemy Experience Reporting, consider removing access to the legacy report.

1. Remove roles from each Action Planner system report:
   1. Open the Reporting > Reports > Standard Reports screen.
   2. In the list of reports, select an Action Planner report.
   3. Remove all roles from the MemberRoles property, and then save and publish your changes.
2. Remove role-based permission to legacy Action Planner reports:
   1. Open the Company > Users > Roles > Roles screen.
   2. In the list of roles, select a role.
   3. Remove the following permissions from the MemberCaps property, and then click Save:
      • View Action Planner
      • View Action Planner - ImportancePlot Tab
      • View Action Planner - List Tab