Response Rates

Feedback > Surveying > Sampling > Response Rates

The Response rate is the average proportion of survey takers that typically respond to an invitation; it is the count of responses received divided by count of invitations sent. The rate is used by Quotas to manage the number of invitations to send.

Response rate equals responses received divided by invitations sent

When there are not enough responses within a period of time in the recent past, Experience Cloud estimates the response rate per a specification defined on the Response Rates screen.

Warning: The default values work best in most cases. Do not change these settings before having a full understanding of the expected results. Adjusting the values might be useful for companies with new sampling programs and no useful historical statistics, but setting them wrong can have very undesirable consequences. Review and understand Calculating the response rate before making any changes.

Note that the response rate is company-wide, while quota are defined for individual units.

Response rate properties

Default Response Rate: Response rate to use when there are not enough invitation emails and responses to calculate the rate. The default is 0.15 (15%).

Note: Setting the default response rate a little low is better than setting it a little high.
Default Days to Stable: Count of days to wait before calculating the response rate when the historical period is not active or valid. The default is 14 days.

Important: Set this value to be as least as long as the count of days to expire a survey invitation. Never set this value less than the count invitation expiration days.
Lag Period Length: How many days to include in the historical period. The period be long enough to avoid short-term anomalies, and should not include recent results that are still in flux. The default is 60 days.
Lag Period Start (days ago): How many days ago to start the historical period. This must be larger than the length of the period, and it should be far enough back to allow a period of significant length, but so far as to not be current. The default is 90 days.
Min Invitation Count: How many invitations must have been sent in the historical period to determine the response rate and avoid using the default values. The default is 600 invitations.

Warning: When using per-Unit calculations, set this value low enough to accommodate a lower volume.
Too High Response Rate: Threshold where rates above this percentage are assumed to be fraudulent. In which case, use Default Response Rate instead. Range is 0.01 to 1.00; default is 0.98 (98%).
Min Allowed Response Rate: Minimum rate to use. When the calculated rate is lower, use this percentage instead. Range is 0.01 to 1.00; default is 0.05 (5%).
Max Allowed Response Rate: Maximum rate to use. When the calculated rate is higher, and not too high, use this percentage instead. Range is 0.01 to 1.00; default is 0.9 (90%).
Stable Response Threshold: Percentage of all responses received, and used to calculate the response rate of invitations sent out. Range is 0.01 to 1.00; default is 0.9 (90%). See Calculating the response rate below for details.
Use per-Unit response rate: Calculate the response rate on per-Unit instead of company-wide. When using this option, adjust the Min Invitation Count down to a setting appropriate for Units.
Reset To Default Values: Resets all Response rate properties to the default settings. Use this after Experience Cloud has been running for a while when the start-up settings are no longer needed.
Check Rate and Log: Calculates the current response rate and days-to-stable values. Displays the results.

Calculating the response rate

The Response rate is the average rate that contacts typically respond to . To get an accurate rate, the Experience Cloud looks at a period of time in the recent past when the rate can be considered stable. Invitation emails sent between today and that stable period are in transition and the response rate is unstable. For example, emails sent an hour ago are unlikely to have responses, while emails sent yesterday might have a few responses. The stable period is when Experience Cloud does not expect the response rate to change significantly.

The historically stable period is a period of time in the recent past when all survey responses have been received. This window of time lags behind the current date by a fixed length called the lag start, and the size of the time period is is the lag length. This period is usually the same as the survey expiration length (how long the invitation is valid).

Stable period starts "Lag start" days ago, and is the "Lag length" count of days from the start towards today

Note: Set the Distribution setting to expire after a number of days less than or equal to the length of the unstable period.

The historical period must be big enough to get a good estimate of the rate, and it must be recent enough to match current response rates. By default, the historical period is a 60 day window (Lag Period Length) starting 90 days ago (Lag Period Start), and contains at least 600 responses (Min Invitation Count).

When Experience Cloud has been running less than lag-start days, or when there are less than the minimum count of invitations, Experience Cloud switches the method for calculating the response rate as follows:

When Experience Cloud first starts sending emails it the uses Default Response Rate, which is usually a very low value like 15%.
After several days, Experience Cloud begins to estimate the rate based on the recent responses received. The days to wait is the Default Days to Stable period, and by default is 14 days.

Estimating the response rate

The estimated response rate is a smaller percentage of the current response rate for all emails received to date, and it is based on the Stable Response Threshold (default is 90% of current rate).

Estimated response rate is based on the stable threshold, which is a percentage of the current response rate

When responses are still arriving in Experience Cloud, the current response rate is less than the eventual rate (when all responses have been received). The estimated rate helps make up the difference ensuring enough invitations are sent out to meet the eventual target.

The current response rate is based on "now", and greater than the estimated rate, but less than the eventual rate (when all responses will have been received

Note: Today is "now" on the sampling server, which is set to U.S. Pacific time. The timestamp in the record is Creation date.

Adjusting the response rate for extremes

After determining the historical rate, Quotas can make a final adjustment to the rate. If the calculated rate is outside the allowed range, the rate is changed to match the range extent. Further, if the historical rate is too high, it is probably the result of an attempt to game the results. In that case, Experience Cloud uses the Default Response Rate instead.

Chart showing a column separated in sections: above too high is probable fraud: use default rate (15%); above max allowed is too high: use Max (90%); below min allowed is too low: use Min (5%); all else in the middle use calculated historical rate

Response rate and imported surveys

Generally, a survey response is considered to be the direct result of a survey taker completing the survey in response to an Invitation email. But, some companies have external systems that collect survey responses, and these results are then imported into Medallia Experience Cloud. Those results can be included in the responses-received count, which in turn affects the response rate calculation.

Any record whose Survey source value is 0 (INVITED) is included in the Response rate. This applies to responses from invitations sent from Experience Cloud and to imported survey results. To exclude a record from the calculation, set the field value to 2 (EXTERNAL) before or during import.

Every Quota rule tracks its own response rate for each unit, and a unit can have multiple Quota rules, as identified by the Sampler associated with the record. The Survey sampling setting field identifies the Sampler:

When that field has a value, the record only affects the response rate for those quota rules identified by the sampling setting.
When the field is empty — and which can only happen for an imported survey result — the response applies to all quota rules for the Unit.

Response rate tips

Any record whose Survey source value is 0 (INVITED) is included in the response rate, including to responses from invitations sent from Experience Cloud and to imported survey results.
Response rate statistics are company-wide. They are not calculated for individual Units.
Response rates are tracked in the sampler. If the rate is based on segments (Unit groups) and you add a segment to the quota rule, the response rate resets and the response-calculations restart.