Survey length and structure

When designing your survey, strive to gather useful operational feedback while keeping the survey short. Surveys are an not just a tool for data collection, they are an extension of the experience — only asking questions that are necessary improves that experience and demonstrates respect for respondent's time. Avoid creating survey fatigue and, with it, fatigue with the company.

Shorter surveys have better response rates, as respondents are more likely to complete all the questions. Survey takers are also more likely to be thoughtful in their answers when the survey is not too long. The survey should require less effort to take than the interaction it was about. In addition, you can advertise the fact that the survey is short in the invitation email, resulting in increased click-through rates.

The following structure has been proven to elicit data that helps companies drive action within their organizations.

Overall metric question

The Overall metric question measures the respondent whole experience. You should place this question first to capture respondent's top-of-mind feedback, before they are primed by specific driver questions. Instead of starting the survey with a welcome page, add any context the survey taker needs to either the invitation email or the top of the first survey page.

Types of overall metric questions include:

Open-ended follow-up question

You should pair the overall metric question with an open-ended question, such as Please tell us the reasons for your score. The answer to this question captures actionable unstructured data that can be analyzed by Medallia Text Analytics, providing insight into the "why" behind the respondent's overall metric score.

Key Performance Indicator (KPI) and Driver questions

Surveys should include 1–3 KPI (Key Performance Indicator) questions. KPIs track overall scores for a particular area of business. While the overall metric tracks the overall experience with the company, KPIs ask for information related to individual touchpoints a serve as predictors of the overall metric (for example, a change to the product offering might impact the Likelihood to Recommend score).

Following each KPI question, measure each area of the business by using a set of 2–5 attributes whose performance drives the KPI scores — these attributes are called drivers.

For example, suppose the KPI question is How satisfied were you with the associate in the following areas? The driver attributes for that KPI question might include Knowledge, Attitude, and Professionalism.

Drivers provide insight into performance during specific parts of an experience, and serve as predictors of KPIs (e.g. a change in Knowledge driver might impact the answer to the Satisfaction with Associate KPI). Each driver should be actionable on its own and not highly correlated with other questions. For example, you would not add Courtesy to the group in the image above because it would likely correlate highly with Professionalism and therefore wouldn’t provide much additional insight.

Other questions (Problem, Demographic, and Segmentation questions).

All questions should be actionable, with business objectives in mind. Don't ask questions to which you already know the answers. You should omit segmentation and demographic questions from the survey unless absolutely necessary, as they can increase survey abandonment rates.

Problem questions, such as Please tell us about the issue you experienced, allow survey takers to provide more context to their issues so your organization has a better idea of how to close the loop and resolve them. Problem questions are optional — if your company does not have the bandwidth available for follow-up, consider excluding them from your survey.

Catch-all question

The survey should end with a catch-all question to allow the respondent to provide any additional information. One example of a catch-all question might be: Is there anything else you would like to share with us?

Thank-you page

The final section of the survey should thank the survey taker for their feedback. Optionally, you can also link to a webpage you would like to promote.

Survey structure considerations

The following considerations should be taken into account when defining the survey structure:

• Never make an open-ended question required, as this increases abandonment rates.

• Multi-page surveys are encouraged. These mentally focus the survey taker on the current questions and minimize scrolling. These also grant the ability to create conditions based on previous answers and provides abandonment data segmented by page. Multi-page surveys may seem longer to the user as the exact length can't be seen, but this problem can be mitigated by a progress bar.

• Configure parts of the survey so that they are only shown conditionally, based on answers to previous questions. This allows you to cover more grounds with the survey while not making users answer irrelevant questions.

• Use Quick start to add the first question of the survey in the invitation email.

• Structure the survey as a flex survey, presenting only the key questions first and making the rest of the survey an optional extra step.

Choice set scales

Medallia recommends an 11-point scale from zero to ten. Eleven- point scales provide more granular data with more differentiation than scales with fewer options. For instance, it is possible to tease apart those experiences that elicit a 10 from those that elicit a 9 to highlight top performers.

Keep the numeric scales in answers consistent throughout the survey. This makes it easier for respondents, reduces abandonment, and makes it easier to conduct statistical analysis between questions.

Numbers on the scale should be ascending from left to right, as this adjusts better to the way most survey takers expect.

Add anchors to numbers on the scale so that their meaning becomes less ambiguous, for example Fully agree anchored on 10 and Don't agree anchored on 0. The recommended approach is to place these on the end points of the scale only. Be aware of the phrasing on the anchors, as it affects responses. The words Not at all ... are recommended over Extremely ....

Question wording

Use the following guidelines when writing survey questions:

The experience journey

A journey map is a framework that maps people's experience through their eyes and enables you to improve your CX/EX. Mapping out the journey allows you to understand which touchpoints are relevant for survey design, as well as timing of survey triggers. Defining a journey map can help you:

Defining a journey map

Use stakeholder interviews to first grasp the level of understanding your organization has around the journey. This can determine your approach:

Below are two examples of journeys:

Interactions between the person and the company can be of high engagement and/or of high volume. A high engagement interaction is one that is important to the person, where there is a greater number of ways things could go, which implies a certain risk. Taking a food order online is low engagement, taking an order in person and recommending a dish is high engagement. A high volume interaction is one that occurs with a significant number of individuals. Paying at the checkout stand is high volume, but accepting a refund is (or should be) low volume. Interactions that are high engagement and/or if high volume should be given more importance, and questions in the survey should focus on learning about these.

Quick start

Quick Start bring the first question into the content of the Invitation email. By selecting a number on the scale of this question, users are redirected to the full survey. This practice greatly increases click-through rates and response-rates, as it requires less effort from the user to start taking the survey.

Quick Start surveys work on all devices, automatically rendering a survey optimized for the device type.

Flex surveys

Flex surveys display an initial set of key questions and then offer respondents an opportunity to answer additional questions. This allows the survey to obtain a high response rate for the most important questions (key metrics) with other metrics as optional questions.

Flex surveys give you a more complete picture of what survey takers think, as those who would have opted out will be more likely to give feedback and these are more likely to have a more neutral or negative opinion. As a result of this, restructuring a regular survey into a flex survey tends to make the initial questions rated lower and the additional questions higher, as these are only answered by those with a greater interest in providing feedback.

Users decide whether to take the rest of the survey by answering a Flex Survey question. This question's wording should include the following components:

Including the number of additional questions in the Flex survey portion is not a recommended practice, as this requires additional effort to manually update the survey.

Follow-up surveys

A follow-up survey is a survey that is triggered after an alert has been closed on a previous survey. Its goal is to study follow up actions of frontline users and ensure that dealers have closed the loop properly. Sending these surveys could be automated or manual, depending on the business rules.

The main value that these surveys adds is providing frontline accountability for closing the loop. It allows to understand the impact of closing the loop and find the most effective ways to do it. It also adds incentive for frontline users to be mindful of the full experience.

To reduce respondents' fatigue from answering a second survey, include a permission to contact question on the initial survey and only send follow-up surveys to those that gave their permission to be contacted. If the follow up survey is sent by a specific dealership, make sure this is made clear in the invitation email. If a dealership closes the loop improperly on the initial survey, they should not be the ones to close the loop the second time around. Medallia recommends that the corporate branch team closes the loop in those cases.

Rejecter studies

Rejecter studies are a series of questions asked about dealerships that consumers visited, but where they did not make a purchase. The goal is to provide rejected dealers with information on why they lost out to their competition even though they were likely offered the same product.

To be able to compare dealerships, you need information about the proximity of other dealerships, as Medallia can't infer this on its own. This information either needs to come in the Invite or the Org file.

Rejecter studies have the potential to grow into too many questions. To limit the burden on the consumer, use flex surveys and ask a question like "Do you want to answer more questions about your experience?" before making the survey taker fill out the rejecter portion. Medallia recommends to ask about 3 dealers and up to 5 questions.

If not using flex surveys, keep in mind that survey takers could drop off for the whole survey if they feel the rejecter questions make it too long. Consider implementing autocompletion on the survey.

Below is an example report showing rejecter reasons:

In-the-moment surveys

In-the-moment surveys enable respondents to leave feedback while still in their journey via a tablet or other devices available in the premises.

These surveys can gather data from non-buyers that visit the premises. This might be very valuable data as it helps signal the problems that stopped individuals from buying. They can also help fix problems before the interaction is over, especially in hospitality where the interaction lasts longer. Keep in mind that these types of surveys can have the following challenges:

A/B tests

A/B testing is a structured experiment that compares two versions of something to determine which performs better. Typically, there are two groups:

Ideally, only one feature, product, or element in a program is changed while all other attributes remain the same for both groups to deliberately identify and measure the results and impacts of an update. A/B testing can be used for customer-facing as well as internal applications, such as redesigning a training approach or policy.

Test results provide insights and understanding of impacts without taking on all the risks of a full implementation initially. Note that not every change requires a test, especially in low risk scenarios.

Executing an A/B test

The following table outlines the tasks to perform in the different stages of A/B testing implementation:

This image shows the timeline of the A/B process:

Note that group size can vary depending on risk tolerance and exposure necessary for statistically valid results and do not have to be split 50/50.

Using A/B testing

You may be asked to advise about how a given solution impacts customer satisfaction, loyalty, and/or operating costs. After you perform the required research and identify various solutions, and before making any fundamental change that encompasses significant implementation risk or costs, you should test your solution so that you can be confident recommending it. A/B testing helps to measure the impacts and allow for proper transition, especially for situations where compensation is tied to scores or alert closure SLAs, or if the organization is very sensitive to score changes.

Use the results of your A/B test to continuously improve the experience and understand impacts prior to full scale implementation.

The following table outlines the potential impact and considerations when performing A/B tests:

Examples of A/B test scenarios

The following are typical examples of different A/B testing scenarios:

1. Website shopping cart abandonment — Research shows that abandonment rates are high at checkout, due to a confusing payment process. You decide to run an A/B test to discover if the payment design process results in an increase in purchase conversion. Based on your analysis, you determine that showing the new experience to 5% of shoppers is statistically significant. The remaining 95% will continue to use the current payment experience, until you can confirm that the new design is effective

2. Email subject line — Click-through rate for the Millennial demographic on marketing messages is low and the team has a goal of increasing it by 25%. The team decides to run an A/B test and change the subject line to create a more emotional connection with the specific, targeted audience and then see if this has an impact. The team decides to change the subject line for 25% of Millennials and monitors results in order to make a decision. The remaining 75% will continue to receive the current subject line, until the team can confirm whether the test produces the desired result.

3. Call center training classn — The brand has been experiencing higher than usual call volume, decreased customer satisfaction and high agent turnover. Ahead of the next agent training class, the executive team would like to understand if focusing on building a relationship, instead of the traditional product approach, has any material impact on cost of serving, satisfaction, and employee experience. The brand decides to split the agent training class into two equal groups to execute the A/B test. The first group will continue to receive the traditional training and the second group will receive the updated approach. After measuring results, the team will determine which is most effective, using the predefined success metrics, and make a recommendation for change.