Key components of testing

Foremost, to test weighting properly you must understand how weighting works, as explained in Weighting. Although this topic includes steps and an example test plan, you must create your own testing plan based on your unique configuration.

The following image highlights the overall process involved in weighting. Arrows show stages of the process, and boxes show tables and reports you can view and/or configure. Study your inputs and outputs to ensure that the processes are working as expected. If you need assistance, contact your Medallia Expert Services team to review your test plan.

Always build and test your weighting schemes on a sandbox to ensure your familiarity with the product and the testing steps involved. After you have a functioning weighting scheme on your test instance you can transfer it to the company's production instance.

Testing steps

The following steps provide a high level guide for testing a weighting scheme in Medallia Experience Cloud. Steps 1-3 reveal whether weighting has been configured correctly, so run those steps internally before presenting anything to the company. Step 4 verifies that you and the company have the same understanding of how weighting should work, so run that step with the company.

Before you begin, download the weighting segment fields, A-fields, and main score fields for one weighting time period For example, download a full year of data for a yearly weighting scheme, or a month of data for a monthly weighting scheme. Use that data set to perform your tests.

1. Verify that the survey condition is working.

   It is easy to accidentally miss or apply the wrong survey condition. Review your data set to confirm that weighting was applied to the correct surveys. A common problem is to accidentally apply weighting to test surveys.

2. Verify that segments are correctly assigned.

   A common issue is to assign weights using the wrong segment field. For example, you could have three different product type E-fields in the Experience Cloud, where only one is used for weighting.

3. Verify that weights are correctly assigned.

   This is the most critical test, and will tell you quickly whether weighting is working as expected. Sum the weights for a weighting segment in your data set, and compare that to the weighting file. Consider doing this in a pivot table to quickly assess whether weights are correctly assigned.

4. Check offline calculations against Experience Cloud reporting numbers.

   After step 3 you should be confident that Experience Cloud is applying weighting correctly, but make sure that you and the company have the same understanding of how weighting will work. Manually calculate the unweighted and weighted scores offline, and then compare the results to the scores presented in reports. Then, let the company change the weighting file and predict what the scores will be in Experience Cloud. After you have the new score, upload the new weights and check that your prediction matches reports.

The company might request additional tests. They are usually out of scope, and add time to the standard estimates and time lines. Some examples of additional tests include:

• Checking reporting numbers against the company’s historical numbers
• Parallel testing of the weighting scheme with a previous or external weighting program

Common issues when testing

You might encounter the following common issues when testing:

• Weighting a-fields do not appear — A-fields related to new weighting schemes might take a few days to appear in the Exports tab, even if they are available in the A-fields screen. To prevent this from interfering with your testing, configure your weighting schemes as far in advance as possible. As a workaround, you can add the a-fields to a K-field, which will appears in the Exports tab immediately.
• The weighting reporting date is not the same as the date in reporting — Make sure you select the same reporting date as the one used in reporting. For example, if the weighting scheme is based on Creation Date and the reporting is based on Response Date, the number of responses for a given month are likely to be different. This is particularly important if you are using a K-field to handle timezone shifts. In that case, use the same K-field for the weighting configuration.
• Deployment cycle flag or test flags— In programs with E-fields such as Deployment Cycle or test flag, include that field in the weighting scheme (or as a condition) so only live data is weighted.
• The sum of survey weights is not equal to the segment weight for a given time period — This can happen if you configured the weighting scheme with an incorrect weighting time period, causing weights to distributed over a longer or shorter period of time than you expect. For example, if your company provides weights on a yearly basis but wants to report on the scores monthly, pick a monthly time period in the weighting scheme to ensure that the weights are distributed every month. If you were to pick a yearly time period the weights would be distributed over the course of a year.
• Medallia has not distributed weights properly, even though everything seems to be configured correctly — Run a backfill. If you make changes to the weighting scheme settings or upload new historical weights, you need to backfill your data to ensure that the weights are current.

Example test scenario

This section provides a detailed example of how one specific weighting scheme might be tested.

Assumptions

For this scenario assume the following items have been configured:

• Weighting Scheme Identifier — retail.
• Weighting Timeperiod — MONTH.
• Reporting Date — Response Date.
• Survey Condition — This scheme applies only to the Retail touch point, so the scheme uses a condition of e_my_company_touchpoint_alt = 1.
• Segment Fields — The following e-fields determine the segment to which a survey belongs:
  • e_my_company_brand_alt, with alternatives of RigelElectronics (SequenceNumber 1), RigelHome (SequenceNumber 2), and RigelGarden (SequenceNumber 3).
  • e_my_company_product_number_autotext, with an AutoIndexed alternative set supplied by the company in an Invitation File. Expected values are between 0 and 20.
• Questions — The company is interested only in the Weighted NPS calculation, which uses the LTR question (q_my_company_ltr_10na).

The following image shows the completed weighting scheme:

Assume the company has provided the following tabular data for product weights:

Assume the company has provided the following tabular data for brand weights:

Preparation

To test the survey weighting segment assignment and the survey weight assignment, create a survey export with the following fields:

• The field you chose as your Reporting Date when creating your scheme (in this example, e_responsedate)
• a_surveyid
• All fields used in the weighting scheme Survey Condition (in this example, e_my_company_touchpoint_alt)
• All fields used as Segment Fields (in this example, e_my_company_brand_alt and e_my_company_product_number_autotext)
• The survey weighting segment A-field (in this example, a_segment_for_weighting_scheme_retail)
• The survey weight A-field (in this example, a_weight_for_weighting_scheme_retail)
• All question fields that need to be tested (in this example, q_my_company_ltr_10na)

For the export, set your Export Time field to be the same field you chose as your Reporting Date when creating your weighting scheme (e_responsedate in this example). Set your Export Timeperiod to be an interval that corresponds to your Weighting Timeperiod. In this example, the Export Timeperiod is Previous Month because the Weighting Timeperiod is MONTH. Your export should be similar to the following image:

Verify that the survey condition is working

Spot check the spreadsheet to make sure that only the records with touch point 1 (Retail) have a weight in the segment A-field (a_weight_for_weighting_scheme_retail). As shown in the following image, surveys with touchpoint 2 do not have an assigned segment (the A-field is null), confirming that the survey condition works:

Verify that segments are correctly assigned

Test to see if weights are assigned using the right segment by comparing the weighting segment A-field to the unique ID found in the weighting lookup table created by Auto Importer. You can test this thoroughly in Excel using a vlookup function. Alternatively, a spot check might be sufficient.

From the survey export:

From the weighting lookup table:

Verify that weights are correctly assigned

This test involves checking that the weights have been correctly distributed. In this example, summing the individual survey weights for product 0 and brand RigelElectronics should equal the weight in the weighting file (103674.81501). If this test is successful you can be confident that Experience Cloud reports will show the correct weighted score. As shown in the following image the weights are correctly distributed apart from one segment, 1RigelHome. The sum of weights is much lower than the weight in the weighting file, which is something needing further investigation.

Check offline calculations against reporting numbers

To run this test, manually calculate the unweighted and weighted NPS scores and compare these to the scores in Experience Cloud reports. Do this for a range of segments and filters to make sure it’s working as expected. The following image shows calculated unweighted and weighted scores for several segments:

Log in to Experience Cloud and check the NPS score for RigelElectronics. The expected unweighted score is 25 and the expected weighted score is 69.