Sampling

Identifying invitees to receive emails to personalized surveys

Sampling is the process of filtering raw Invitation files to identify invitees to be sent personalized invitation surveys. Sampling occurs after an Importer collects the feedback invitation data, and before an Invitation email is sent. Each sampling process has up to four stages:
  • Opt-out — Excludes invitees who have expressed intent to no longer receive survey emails.
  • Filters — Identifies the emails to include or exclude, usually based on business rules.
  • Quarantine — Avoids sending emails too frequently to the same invitee.
  • Quota — Attempts to achieve a target count of responses for various segments of the company.

    Raw invitations pass through Opt Out, Filter, Quarantine, then Quota before identifying the final survey emails

How sampling works

At any stage, if an Invitation email does not pass a filter, the record is not sampled and never considered again. By default, Medallia Experience Cloud automatically tests all raw invitations against the permanent opt-out list. No other sampling rules happen by default; all other rules are custom to the instance.

Sampling occurs in batches of 5,000 records, which are sampled against both the other records in that same batch and the ones in the database. For example, if in Batch A there two records (R1 and R2) that have the same IDs and share the same conditions, R2 gets sampled out as it is part of the same batch, without being checked against the database. However, if R1 and R2 have different IDs and conditions, they are checked against the database.

See Creating a custom sampler below for details about the steps for implementing a sampler.

Warning: Automatic sampling after import happens when there is a Trigger whose Feed Event is FILE_FEED. By default, Experience Cloud has at least one Trigger whose event is MANUAL, and that trigger is for testing. In production, there needs to be at least one for FILE_FEED.

Additionally, the Auto Importer specification must have a declared Sampling operation, otherwise, no sampling happens to the imported file.

Opt-outs

Opt-outs exclude invitees who have expressed intent to no longer receive emails. When an invitee opts out (from a survey email, the survey, or via survey support) the values of the Sampling Field Setups are copied from the record to the Opt-out Record list. All future raw invitation are compared to that database, and if any of the values matches, the new record is not sampled.

Opt-out records can also be created automatically using Opt-out rules. These rules evaluate each record that enters sampling, and when it meets a rule condition, the information from the record is used to create new opt-out records. Opt-out rule processing happens before testing the record against the existing opt-out records.

Filters

Sampling filters identifies emails to include or exclude during Sampling, usually based on business rules. Experience Cloud can have zero, one, or multiple filters: if any filter excludes a record, that record is permanently excluded, even when other filters might include the record. When there are no filters, all records pass through this stage.

Filters exclude specific raw invitations based on business rules, as opposed to Opt-out rules which permanently exclude invitees.

Quarantines

Quarantines are the third step in Sampling: they are rules to avoid sending surveys to the same invitees too often. The incoming potential emails are compared to a list of recently-sent emails within a recent number of days. Potential emails that match recent ones are excluded from the final survey email set.

Only invitees who were sent emails — or who completed surveys if that option is used — are included in the quarantine list. Potential emails that were excluded for any reason, including quarantine, quota, or other filters, are not included.

Quotas

Quotas are the last step in Sampling: they attempt to achieve a target number of responses for survey emails sent in a given time period. They can limit the count of emails for high-traffic units to allow lower-traffic ones to be statistically significant in the overall scores and results. They can also capture certain proportions of the contact base (by segment), such as to ensure a significant number of high-loyalty customers are included in the results.

During sampling, Experience Cloud compares the current response rate to the target rate necessary to meet the quota. When necessary, Experience Cloud adjusts the number emails sent to better assure the quota number of responses are received for emails sent before the end of the quota period.

Two trend lines: target and current response rate, with the difference show for "today" and a change in the invitation rate to compensate for the difference

For detailed information about creating and working with sampling, see:

Creating a custom sampler

By default, when sampling is enabled for file processing, Experience Cloud automatically tests all raw invitations against the permanent opt-out list. No other sampling rules happen by default; all other rules are custom to the Experience Cloud instance.

To create a Sampler:

  1. In the Auto Importer specification, declare a Sampling operation to pass the records to sampling.
  2. Create a Trigger whose Feed Event is FILE_FEED. By default, Experience Cloud has at least one Trigger whose event is MANUAL, and that trigger is for testing. In production, there needs to be at least one for FILE_FEED.
  3. Optionally, create one or more Opt-out rules to automatically and permanently exclude invitees based on specific criteria.
  4. Optionally, create one or more Sampling filters to exclude emails based on business rules.
  5. Optionally, create one or more Quarantines to avoid sending surveys to the same invitees too often. 
  6. Optionally, create one or more Quotas to attempt to achieve a target number of responses for emails sent in a given time period. To create a quota:
    1. First create a Sampler (if one doesn't already exist).
    2. Create the Quota and reference the Sampler.
    3. In the Auto Importer, identify the Sampler as the one to run for survey processing.

Tips and best practices

For surveys about a one-time event (such as a purchase)

  • Make sure the email is sent soon enough for the invitees to remember everything about the event, but not too soon, as there may be missed opportunities for feedback on problems that haven't cropped up yet.

Surveys about ongoing events

  • Avoid surveying large amounts of invitees in waves, because too much feedback coming in at once makes it difficult for account managers to close the loop and act on the data. Instead, a best practice is to flatten out these waves and send emails continually throughout the year.

  • An account should be surveyed 90 to 180 days prior to their anniversary/renewal. This allows for enough time to take action based on the survey feedback.

  • Typically, a relationship survey is sent once a year. This allows for enough time to close the loop and make changes that are visible to the invitees. If no improvements have been made between surveys, invitees do not see the value in responding again.

Relationship surveys (B2B)

  • It is important to include contacts at all buying influence levels of the company: decision makers, influencers, and end-users.

  • As a best practice, survey all invitees for a single account at the same time. Benefits of this approach include:

    • 360-degree view of an account at a particular point in time.

    • Ability to leverage the client's program team to drive response rates (following up with non-respondents).

    • Manageable closed loop process since focus is on holistic account.

  • Survey each account once every six months to one year, distributing different accounts across that period. This is known as Continuous Feedback Management (CFM), and it prevents peaks and valleys in survey volume, allowing frontline account managers to follow up with each of their accounts in a timely manner. Under CFM, sampling is most often distributed either based on contract renewal date or using dates flagged in campaigns in the company's CRM.

    When a company’s IT or organizational infrastructure is not conducive to CFM sampling, many companies turn to a quarterly wave-based approach where the accounts are divided into quarterly waves of survey emails.