Traditional audits vs. calibration audits

Traditional audits and calibration audits share a few similarities, but they both serve unique purposes, and one may be more appropriate than the other for your company's audit policies. You can use both types of audits in your organization, but only one type for each review.

Audits

Traditional audits in Agent Connect can be thought of as an exercise in grading reviewers. An auditor scores a reviewer based on a scorecard specifically intended for that purpose. A traditional audit helps to answer questions such as "Did the QA reviewer provide accurate scoring that adheres to our QA policies?" to make sure that reviewers grade agents fairly, consistently, and in line with company policies.

Example

In the following example, you can see how the scorecard used for the review differs from the scorecard used for the audit:

Calibration audits

The new calibration audit takes a different approach. Rather than scoring a reviewer based on a scorecard, auditors score the underlying interaction using the same scorecard as the reviewer. After the reviewer and auditor score the same interaction with the same scorecard, you can compare the results and view the alignment score. This exercise helps you to discover exactly where misalignment occurs in QA Reviews, thus helping to refine your coaching and training for reviewers.

Example

In the following example, the reviewer and the auditor have used the same scorecard, resulting in an alignment score that shows how close the reviewer's score of the interaction matches the auditor's score of the same interaction:

Calibration sessions vs. calibration audits

Agent Connect already supports group calibration sessions, but the new calibration audits feature allows for more granular analysis of individual reviewers' performance.

In a calibration session, a group of reviewers come together to review the same interaction, resulting in a group alignment score. A calibration session is not used to measure individual reviewers' performance, but rather is intended to help a group of reviewers to come closer into alignment with each other and the company's expectations.

If you want to grade individual reviewers to improve their performance over time, use calibration audits, and if you want to bring reviewers together for an exercise in improving alignment, use calibration sessions. Using both at appropriate times can maximize the effectiveness of your QA program.

Perform a calibration audit

Follow the steps below to perform a calibration audit:

1. Open the Completed Reviews tab on the QA Dashboard.

   

2. Use the filters at the top of the page as needed to find a specific review, or simply browse the Completed Reviews table to find one that has not yet been audited.

3. Click Create Audit, then select "Calibration audit" from the dropdown list.

   

   The agent interaction and review scorecard will appear.

4. Use the scorecard on the right side of the page to review the agent interaction, then click Submit Audit when finished.

   
   Note:

   • Remember that you are scoring the agent interaction and not the QA reviewer. The QA Review and the reviewer's name will remain hidden until the calibration audit is complete.

   • Calibration audits cannot be edited or deleted after submission, so make sure to review your answers before moving on.

   If needed, you can click Save Draft to save your audit to be completed later.

View audit results

The Reviewers and Audits pages contain metrics and details of completed calibration audits. In addition, the Audits API endpoint can be used to obtain calibration audit records in bulk.

Request data via API

The QA Audits API endpoint provides up to 1,000 audit records at a time in JSON format. The API returns information about the underlying interactions, reviewers, auditors, questions, scores, and more for all audits in the specified time period.

Review aggregate data

On the Reviewers page, two aggregate metrics provide an overview of calibration audits completed in the selected time range:

• The Audit Alignment Score show the average alignment score, as a percentage, among all calibration audits. Beneath the average score, you can see how the current period's score compares to the previous period.

  

• The total number of calibration audits is displayed as a separate metric. You can click View Calibration Audits in this tile to open the Audits page with the same filters applied.

  

In the Reviewers table, the Audit Alignment Score column shows the average alignment score for each reviewer over the selected time period. You can click Calibration Audits in any row to open the Audits page filtered to a specific reviewer to view more details.

Review individual audits

The Audits page (QA Dashboard > Audits) contains detailed information about completed calibration audits for all reviewers. On this page, you can view an auditor's score next to a reviewer's score for a more granular understanding of where alignment should be adjusted.

In the Audit Type filter, select "Calibration Audit", then set any others filters as needed to find specific audits. In the Audits table, you can see the date of each audit, the reviewer and auditor, the alignment score, and the scorecard used.

Click View in any row to view the underlying interaction, original review, and calibration audit displayed in a three-panel layout:

The alignment score in the right-most column shows how closely the review and audit matched. As you scroll down, you can see exactly where the two differed, including comments and annotations.