Anti-cheating Engine

Identifying and flagging data anomalies and responses that might be invalid feedback

The Anti-Cheating Engine (ACE) and its associated health check reports manages the integrity of the survey process by identifying and flagging data anomalies and survey responses that might not represent actual feedback. For example, internal employees might submit fake responses to influence their customer loyalty and experience scores, and those responses should not be allowed to influence the impact of legitimate responses.

The ACE service sits outside of Medallia Experience Cloud environments, and it is referred to as Polygraph. It analyzes survey responses and then pipes its results back into Experience Cloud for reports via a flat file in an SFTP folder.

ACE uses a proprietary algorithm to identify and flag questionable responses. The engine generates a report of its findings and the data is included in Experience Cloud reports to review responses marked as suspicious, and quarantine those that should not be included in reports.

Restriction: This feature is not available on Medallia GovCloud. For more information, contact your Medallia representative.

Why and how employees might cheat

In general, when a company has limited cultural incentives (which are healthier and more intrinsic than monetary incentives) to improve the customer experience, employees might feel more inclined to cheat. Cheating is most common when:

  • Employees have an unhealthy focus on survey scores.

  • A company implements negative consequences for employees and teams that do not improve their scores.

  • Employees perceive the effort of improving customer experience scores is too high, leading those employees to look for an easier path.

  • Monetary incentives are attached to scores, which can be set at the wrong levels of a company, such as with the front-line staff.

  • Employees take a shortcut in dealing with their feedback, or even just competing with other colleagues, branches, or regions.

Treating survey responses in this way is not recommended, and usually comes down to a corporate culture focusing on results, rather than a cultural change

How employees might cheat

The following categories summarize the various methods employees might use to cheat. Understanding these methods can help you plan your prevention and management strategies:

  • Imitating respondents — Employees taking surveys intended for changing the customer's email address, or re-sending surveys to expected promoters.

  • Manipulating detractors sample— Changing recipients email addresses so surveys of expected detractors bounce, entering blank email addresses, using the Invitation tab to resend surveys to a different address, or opting recipients out of survey.

  • Following up before the official closed-loop interactions — Calling future respondents before they have submitted their surveys to obtain better scores, or calling or sending a mock survey before the actual survey sent by Experience Cloud.

  • Coaching for high scores— Taking surveys with the respondents, providing the respondents incentives (such as discounts or any other financial incentive) for submitting positive surveys, re-sending survey to potential promoters, or letting the respondent know that positive survey responses are important ("Anything other than all 10s is bad for us").

Prevent and manage cheating

Cheating undermines one of the key capabilities of Operational Customer Experience Management (OCEM): driving accountability at all levels. A program with cheating is less able to be responsive to individuals' real needs, and is at significant risk of not being aligned with the right responsibilities and goals. When a program has damaged accountability, making great people-centric decisions and experience improvements becomes difficult, and ultimately customers are the ones who suffer.

This table summarizes the actions you can take to prevent and manage cheating:

Imitating respondentsManipulating detractors sampleFollowing up before CLFCoaching for higher scores
To prevent it

  • Know the behaviors to motivate.

  • Achievable goal setting process with forward-looking goals.

  • Balance use of monetary and non-monetary incentives.

To manage it

  • Audit email address (repetition and domain).

  • Implement the ACE reports.

  • Limit frontline invitations abilities.

  • Make units accountable for valid email/bounce rates.

  • Mystery shopper.

  • Prevent employees from accessing an anonymous link.

  • Ask a question in the survey.

  • Use Medallia Text Analytics to audit responses.

Tip: No single method or tool can eliminate cheating completely: use a comprehensive approach that prevents and responds to evidence of cheating.

Prevention

Implementing a program-wide, unique global transaction ID is the most reliable preventative measure against survey cheating because it requires that all feedback and activity data be tied to a unique code that would not be knowable externally. This measure has the added benefit of simplifying duplicate checking across personalized and anonymous surveys. A unique global transaction ID requires that all touch points and transaction types in the program have a unique but identically-formatted transaction code (typically an alphanumeric string) with a fixed number of digits. While this ID is appended to invited records, it could also be printed on receipts to allow for receipt-code inputs in anonymous surveys. In this way, you ensure that invited survey recipients, such as rewards members, cannot submit both personalized and anonymous survey responses. With a sufficiently complex validation framework, this method is very reliable to prevent anonymous survey cheating. For example, unless potential cheaters know the exact receipt-code composition and field mapping parameters, it is very difficult for them to mimic a valid code.

Additionally, these behaviors may help design a prevention strategy:

  • Know the employee behaviors to motivate.

  • Implement a trustworthy goal setting process with forward-looking goals.

  • Balance the use of monetary and non-monetary incentives.

Important: Before configuring ACE for a company, explain how preventing cheating with a well built OCEM program is better than managing cheating after it occurs.

Management

Consider using these methods to manage cheating:

  • Request ACE as described below.

  • Audit the access your frontline staff has to survey invitations.

  • Prevent employees from accessing Live links for Anonymous surveys.

  • Audit email addresses (for repetition and by domain).

  • Monitor the bounce rate of survey responses, and keep units accountable for valid rates.

  • Use a mystery shopper program to collect information about how employees interact with customers.

  • Use Text Analytics to audit comments.

Use ACE to monitor possible cheating

ACE manages the integrity of the survey process by identifying and flagging data anomalies and survey responses that might not represent actual feedback. To do so, it generates reports of suspicious records, and highlights units and any user associated with such activity. Managers can review each flagged survey response to decide whether it should be quarantined and removed from reports

ACE applies a set of up to sixteen rules to each record analyzed. If a specific rule deems a record suspicious, then that rule's relative importance is considered when calculating the record's overall suspiciousness score. If the overall score passes the threshold defined for your company, the record is considered suspicious and ACE generates files of records daily. If the overall threshold is not crossed, the record is not considered suspicious, even if one or more individual rules were violated.

ACE also applies various algorithms to reduce the overall possibility of false positives (over-reacting to some suspicious-looking but innocent survey-taking behavior).

For example, an auto dealership receiving unusually high scores in multiple surveys, all coming from the same IP address, and for which there is no on-site surveying being done by customers (because the survey is supposed to be delivered by email at the end of the day) would be flagged as suspicious.

Note: The evidence produced by ACE is not absolute proof of cheating, and might not catch all cheaters. It is an indication of a need to investigate highly suspicious results found by this purposefully conservative, but effective approach. Also, there is no certainty that suspicious records can be attributed to cheating. Experience Cloud provides supporting data in Health Check reports to assist companies in determining the source of potential problems, and the appropriate actions to take based on the information. For example, in some geographies it is common for IP addresses to be shared due to Internet cafes or shared computers. Because of these kinds of phenomena, companies use the reports in different ways. Some companies choose to exclude from their results all records reported by ACE, while others use the data only as a reference to share with individual units.

Only surveys that are submitted through the Survey engine are available for monitoring by ACE. Social media records and imported surveys from other sources are not included.

ACE reports

ACE generates daily Excel reports automatically, and emails those reports to companies. The report format is standard, and cannot be customized. However, some report content can be customized, as determined by the fields you request as part of your ACE implementation. For more information, see Request ACE.

Every night, ACE collects the survey data from the previous day, and generates a Feedback protection report for the surveys that were processed. The timing of report generation is global across all companies and cannot be customized by company.

For more information, see Reporting screen.

ACE app

The Anti-Cheating Engine app is available as a recommended enhancement to standard ACE functionality. It includes:

  • A new ACE report showing trends within ACE results.

  • A Responses Feed module to display ACE results for individual surveys.

  • An Importer to enable the import of ACE data into Experience Cloud.

For more information, see Reports and alerts.

You can request the Anti-Cheating Engine add-on App at any time. For more information, see Request ACE.

ACE historical updates and retention

It is only possible to perform historical updates to data that was collected after ACE was initially enabled.

For example, if ACE was activated on June 1st with a specific set of rules, and those rules were later modified on June 8th, you could perform a historical update for data collected between June 1st and June 7th. However, this functionality does not apply to data predating ACE's activation on June 1st.

Polygraph resides outside of MEC, and pulls data from the survey engine. Polygraph only looks forward, it does not look at things prior to when ACE is activated. That's why it's important to activate ACE as soon as programs are launched.

There is also a retention period for data in ACE. You can not generate reports that go beyond the retention period specified at setup. For example, if that retention period is set to six months, you can not then generate a report showing responses from the last year.

Request ACE

ACE is not configured automatically for every company. To request ACE:

  1. Submit a request through Medallia Support, including:

    • The URL of your Experience Cloud instance. For example, https://company.medallia.com/company.

    • The Medallia-hosted SFTP location for routing ACE reports. For example, https://company@ftp.medallia.com/acereports. If needed, a company can export the reports from Experience Cloud and upload them to a non-Medallia SFTP location.

    • The names of the people on the Medallia Professional Services team that need access to this company's configuration.

      Restriction: Partners cannot access ACE instances (also known as Polygraph).

    The Medallia Production Services Engineering team completes the initial setup, and creates a Polygraph instance for you. This instance is located in one of these data centers:

    Data centerPolygraph server
    SC4 (US)https://polygraph.medallia.com
    SEA1 (US)

    https://polygraph.sea1.medallia.com

    CAN (Canada)

    https://polygraph.medallia.ca

    LON (Europe)https://polygraph.medallia.eu
    SYD1 (Aus)

    https://polygraph.medallia.com.au

    FRA1 (Germany)

    https://polygraph.fra1.medallia.eu

    SIN1https://polygraph.sin1.medallia.com

    Select the correct region when getting access to configure the rules for a client. Once known, request access to the client's configuration via Medallia Support.

  2. After your request has been approved, work with your Medallia expert to ensure that the rule configurations are appropriate for your company.