Weighting methodology
A weighted score is a score for a set of surveys, where specific respondent segments have defined levels of importance, calculated as follows:
-
Weighted Average — Σ(Scoresurvey x Weightsurvey) ÷ Σ(Weightsurveys).
-
Weighted Top-Box — Σ(Weightsurvey=10) ÷ Σ(Weightsurveys).
-
Weighted Net Promoter — ( Σ(Weightsurvey>=9) - Σ(Weightsurvey<=6) ) ÷ Σ(Weightsurveys).
It might seem complicated, but this is how Medallia Experience Cloud calculates all scores, even when weighting is not applied. For unweighted survey responses Experience Cloud implicitly weights scores by the number of responses (an implicit weight of 1). You can configure weights to cover standard calendar time periods (day, month, quarter, year, and forever). For example, with a monthly weighting scheme your company could provide different weights for each calendar month.
Weighting schemes are applied when calculations (including those performed by R-fields). When those calculations are applied to a score field, only surveys with a numeric score are considered in the weighting calculation. For example, you might have a weighting scheme applied to an NPS calculation. When that calculation is used in reports, only surveys with a related numeric score are considered when calculating the weighted score. Null values are not considered. Similarly, if the alternative set of the score field contains N/A as a value, surveys with N/A in the score field are not considered. For more information about creating and applying weighting schemes, see Configuring weighting.
General example
The following general example explains the weighting methodology in practice. Assume a company has 3 key survey groups (A, B, and C) based on their organization and/or segments. The company provided total weights for each segment for the month in a weight file:
After the first day, 6 survey responses in the three queues are received, for an unweighted average of (4 + 1 + 2 + 6 + 9 + 5) ÷ 6 = 4.5:
Each provided group weight is distributed evenly to all of the surveys in a group, on the basis of the actual number of responses. In this example, the weighted average is ( (4 x 10) + (1 x 10) + (2 x 10) + (6 x 40) + (9 x 40) + (5 x 90) ) ÷ 200 = 5.6:
As new surveys are added to the system, weights adjust. For example, after survey M7 is received the weights for both surveys M6 and M7 adjust to 45:
( (4 x 10) + (2 x 10) + (9 x 40) + (5 x 90) ) ÷ 150 = 5.8:
Additionally, per-survey weights persist across separate weighting periods (such as months). Each period is weighted according to its total weights. For example:
Weighted scores for Period 1 would be calculated as (4 x 10 + ... + 5 x 90). Weighted scores for Period 2 would be calculated as (9 x 10 + ... 8 x 20). Period 1 receives twice the weight of Period 2, for a calculation of (4 x 10 + ... + 5 x 90) + (9 x 10 + ... 8 x 20) / (200 + 100) = 6.2.
If a survey is junked (e_status = 7) or excluded (e_status = 4), the record is immediately excluded from weighted calculations, and weighted aggregate scores reflect the change immediately. This occurs regardless of the weighting period in which the survey was submitted. For example, assume you are weighting across months, and are currently in the second month. If you junk a survey from the first month, the change affects calculations for both months.
PetMogul example revisited
Returning to the Pet Mogul example started in the main Weighting topic, assume PetMogul has a loyalty program called Pet Plus with about half of their customers signed up. PetMogul has no reason to think that being a member of Pet Plus makes people any more or less likely to answer the survey. Looking at the distribution of customers and respondents finds that the distribution of respondents in Pet Plus matches the distribution in the overall customer base:
Half of the dog customers and all of the cat customers and respondents are Pet Plus members, while none of the hamster or turtle customers are members.
Recall from the initial PetMogul example that all dog customers are unhappy, all hamster customers are ambivalent, and all cat and turtle customers are happy:
| Customer segment | NPS | Customers | Responses | Pet Plus enrollment | Per-survey weight |
|---|---|---|---|---|---|
| Dog lovers | -100 | 100 | 20 | 50% | 100 / 20 = 5.0 |
| Cat lovers | +100 | 50 | 20 | 100% | 50 / 20 = 2.5 |
| Hamster lovers | 0 | 30 | 20 | 0% | 30 / 20 = 1.5 |
| Turtle lovers | +100 | 20 | 20 | 0% | 20 / 20 = 1.0 |
If you were to filter by Pet Plus membership and show unweighted NPS, you would see 10 dog detractors and 20 cat promoters:
| Customer segment | NPS | Responses |
|---|---|---|
| Dog lovers | -100 | 10 |
| Cat lovers | +100 | 20 |
| Total NPS | +33 |
If you were to filter by Pet Plus membership and show weighted NPS you, would still see the same 10 dog detractors and 20 cat promoters. However, from the calculation above we know that the per-survey weight for dog lovers (5.0) is twice that of cat lovers (2.5):
| Customer segment | NPS | Responses | Per-survey weight |
|---|---|---|---|
| Dog lovers | -100 | 10 | 5.0 |
| Cat lovers | +100 | 20 | 2.5 |
| Total weighted NPS | ((-100 x 10 x 5.0) + (100 x 20 x 2.5)) / ((10 x 5.0) + (20 x 2.5 )) = 0.0
| ||
The overall proportion of the segments changes when calculating weights, but the value of the individual customer stays the same.
System fields related to weighting
Some information about each weighting scheme is exposed in the following System fields, where identifier is the Weighting Scheme Identifier you define in the weighting scheme. Each weighting scheme has its own set of System fields.
| System field ID | Data type | Notes |
|---|---|---|
|
| Real number |
This field shows the weight that would be applied if the record was recalculated now. It might not reflect the actual weight that is being used in calculations or stored in the in-memory database (Slug). Note that real number fields cannot be used in calculations. Use this field for exporting and displaying at the individual survey level only. Although you can export this A-field, be aware that the weight of any given survey in a period is dynamic until the last survey within that weighting period has expired. If a company wants the weights to be stored in their data warehouse, consider establishing a special periodic weight export, configured to export data once per period after the weights have been finalized. Do not expose dynamic weights through reporting or Responses List report exports except to users extremely familiar with weighting concepts. |
|
| Integer |
This field shows the internal identifier (segment) that would be applied if the record was recalculated now. It might not reflect the actual segment being used in calculations or stored in the in-memory database (Slug). |
When a segment or unit does not receive a weight in the weighting file, records that belong in that segment or unit have their a_segment_for_weighting_scheme_identifier field set to null, and their a_weight_for_weighting_scheme_identifier field set to 0.
If you are using Org Hierarchy Filtering (OHF), note that when weighting by unit group Experience Cloud sets the record-level weighting fields based on the unit group roll-up of the actual org hierarchy, and not by the record-level a_org_hierarchy field. If you move a unit from one group to another group, the record-level weighting fields of historical records automatically update to reflect the new weights. Therefore, if you are using OHF, make sure that you do not weight by unit group. Instead, consider weighting by record-level segments, where those segments correspond to the same segments as the unit groups you want to see in reports.
Limitations
The following constraints limit how you can use weighting in Experience Cloud:
-
Alternatives — Weighting works only for enumerated fields with a maximum numeric value of 100.
-
Calculations — Weighting works only for standard built-in calculations (Average, NPS, Top Box, % Yes / No).
-
Significant Digits — Experience Cloud can retain 9 significant digits for data.
-
Reporting — Survey scores can be weighted, and weights are available at the individual survey level for export and display as text. However, custom calculations/aggregations cannot be performed on survey weights (such as calculating relative weights for segment aggregations). Additionally, while most reports can display weighted scores, weighting is not available for Action Planner or Text Analytics reports.
-
Maximum weight — Experience Cloud does not have an explicit upper limit for maximum weight. However, the calculation for the weight value of a unit in your import file times the field value in the file is stored as an integer in Experience Cloud. When that calculation produces a result greater than 2,147,483,647, weights for that unit might not be calculated properly or appear in reports. If you encounter these issues during testing, consider dividing all of the weights in your import file by 100 (or 1,000).
System performance
The number of segments in a single weighting scheme should not impact system performance. However, the performance of reporting can be impacted by weighting, and is based on several factors, including the number of fields, complexity of weighting scheme, and number of records. Consider the following tips to reduce performance impacts:
Try to keep the number of fields on Satisfaction reports to 20 or fewer.
Test performance and cube limits on Satisfaction and other reports that use weighting.
Calculate a rough estimate of rendering time:
Time in seconds ~= Dc * [records in the wide set in millions] * 0.005.Dc = number of data cubes; use Satisfaction and log inspector (“Ctrl+I”) using different column splits.
Records in the wide set ~= number of completed records loaded into the in-memory database.
- Consult with a Solutions Architect or your Medallia Expert Services team if you are unsure whether your weighting configuration might impact performance.
Alternative weighting concepts
Conceptually, the term weighting can be applied to any case where a company wants to adjust how a score is calculated. Experience Cloud cannot do this. Weighting in Experience Cloud was designed for a specific use case related to customer experience management. However, there are other common weighting requests that are not what Medallia calls weighting.
Calculating an Index Score (question-level weights)
Some market researchers (such as JD Power) assign weights to individual questions, which are then used to roll up survey-level scores into an index score. For example, an auto company might want to track one metric for franchise compensation and recognition. Based on what management thinks is important in the survey, the researchers create the Best Auto Determination (BAD) Index. The BAD Index is composed of the scores from three questions (A, B, and C), with weights of 30%, 20%, and 50%. An individual survey's BAD score is calculated as:
(Score A x 0.3) + (Score B x 0.2) + (Score C x 0.5)
Although this example above is simple, these types of weighted roll-ups can be complex. Question-level weights can vary on the basis of survey branching, customer attributes, and other factors. Depending on the complexity of the calculation Medallia can meet this requirement through the following methods.
Simple K-fields (as in the example).
Complex calculations / Lookup tables.
Having the company calculate and send the index score.
Relative (Importance Factor) weighting
Certain responses might be considered more important than other responses, although the volume of responses is still considered to be an important factor. For example, a hotel company believes that VIPs are twice as important as other customers, and wants to reflect that in compensation metrics. When calculating a hotel/district/region score, one VIP responses should be treated as two surveys. The overall score calculation is calculated as:
(AverageScoreVIP x ResponseCountVIP x 2) + (AverageScoreStandard x ResponseCountStandard ) ÷ [(ResponseCountVIP x 2) + ResponseCountStandard)]
Medallia recommends that you do not use this type of weighting because it requires Experience Cloud to store the Importance Factor and use it in overriding usual calculations.
The importance factor must be available at the survey level through an E-field or K-field.
Use R-fields or the Calculation dropdown to rewrite the calculations used in reporting.
Medallia can meet this requirement through the following methods:
Calculate and store values in an E-field as part of the import process.
Calculate through K-field logic (time invariant).
Using a K-field to retrieve values from a lookup table.
To reduce the maintenance impact required for this method you can suggest to the company that they weight only the key scores used for compensation.
