Voice signal processing

Medallia Speech uses language models tuned for telephony-based communications such as customer service call center interactions, voicemail, phone sales, and similar audio. This system caters to continuous, spontaneous, uncooperative speech. Speech of this type typically occurs during a phone call between an agent and a caller, or in a voicemail, where it is typical of callers to leave spontaneous messages.

Spontaneous, uncooperative speech is different from other telephony-based situations, such as a receptionist who is practiced in leaving messages (rehearsed speech), someone reading from a script (read speech), or someone interacting with an interactive voice response system (prompted speech).

The following video provides an overview of how Speech transcribes audio calls to text:

Confidence score

Speech provides a confidence score between 0 and 1 for the text that is produced for the agent's voice and the customer's voice. Confidence score is a measure of the system's confidence that the output was transcribed correctly, providing an effective mechanism for filtering output.

Note: The confidence score is not a measure of accuracy. It is a score calculated by Speech that measures the probability Speech has selected the word most likely to be correct out of all the words it believes a region of speech could represent. The confidence score is only relevant to the model being scored, and should not be used to measure performance across multiple models. To measure performance across models, perform accuracy comparisons against a reference set of data for each of the models, as described above.

Confidence scores are available at the word, utterance, and call level. Confidence is calculated for each word, then averaged up to the utterance level, then up to the call level. Confidence scores tend to decrease as calls become more noisy, and as the overall speech signal moves further away from the speech used to train the acoustic and language models. Noise, compression artifacts, and accents all contribute to lower confidence scores.

Diarization

Diarization is a language-independent process for evaluating a mono (single-channel) audio file. Diarization presumes two people are speaking, and separates that mono audio into distinct channels by categorizing speech into two groups. One group is assigned to channel 0 and the other is assigned to channel 1 in the structured transcript. Because in most companies these people are the customer and the call center agent, Speech fields regarding channels are named accordingly.

This process might be less effective when source audio includes hold music, voice recordings, or more than two speakers. Overtalk, which occurs when people are speaking at the same time, might also reduce the overall accuracy. However, for typical agent and caller situations with only two speakers, diarization is effective for separating a call into two distinct channels for enhanced analytics.

Speech also detects and indicates call silence when more than 5 seconds pass without either the customer or the agent speaking.

Medallia recommends using channel-separated audio, which eliminates the possibility of channel-assignment errors.

Stereo and mono audio files

If possible, provide stereo audio files for Speech. Stereo separates the speakers (customer and agent) into distinct channels. With mono, Speech must separate the speakers, and the accuracy of that separation is not always perfect.

Note:

  • Speech only supports two speakers. Multiple speakers on a single channel will be grouped together.

  • Speech currently does not support the identification of an IVR as a separate speaker. If audio contains IVR recordings, those recordings will be presented in the audio channel in which they were found.

  • Phrases spoken by an AI bot will be considered part of the agent channel and speaker.

Audio files recorded in stereo can provide additional metrics not available with mono audio, such as overtalk. Stereo also increases the accuracy of transcripts.

While the insights Speech provides are more detailed and accurate with stereo files, you can still find deep insights with mono files. To improve accuracy when using mono, make sure you build topics that transcend both speaker channels.

Audio data retention

By default, Medallia stores audio and video files for 30 days. If your company needs files stored for longer than 30 days, contact your Medallia expert. This configurable retention period can be longer than 30 days at additional cost. Contact your Medallia expert and your company's IT team if you are considering extending this period.

For more information, see Medallia Media File Transfer screen.

Music detection

Background music can have a significantly negative impact on transcription accuracy. Music detection helps eliminate this issue by excluding music and other high energy non-speech events from transcription results.

Speech detects music in both stereo and mono audio files so it is not included in transcription or reporting.

Language models

Speech uses machine learning components known as models to represent knowledge about speech. This knowledge is applied during transcription. Two types of models used are known as acoustic models and language models. The acoustic model converts audio into a stream of sound symbols specific to a language, such as English or Spanish. The language model is responsible for converting the stream of sound symbols into text.

General-purpose language models are typically trained to understand domains like banking, health insurance, telecommunications, and voicemail. They provide a strong baseline capability that works well without modification.

Accents are incorporated into the process of developing Speech acoustic models. For example, audio from calls originating in Southern, Northeast, Midwest and Western regions of the United States were used to train North American English models, which enables those models to work robustly throughout the USA. For accents that diverge strongly from those found in the USA, Medallia creates new acoustic models as necessary, such as for UK English and Australian English.

For a complete list of supported languages, see Supported languages.

Emotion analysis

Speech applies machine learning techniques to the analysis of emotion, which is determined from the acoustic features of speech. Computer models, trained using thousands of audio and text samples, are used to determine the emotion of each utterance. Since emotion information is captured at the utterance level, Speech can determine how emotion is changing throughout the conversation, and whether the caller is in a more positive state at the end of the call than the beginning.

Each transcribed utterance has its own emotion value, described in the following list:

  • Positive — Only positive emotions were detected.

  • Mostly Positive — Most emotion was positive, but some negative values were also present.

  • Neutral — No emotions were detected, or the numbers of positive and negative phrases were equal.

  • Mostly Negative — Most emotion values were negative but some positive values were also present.

  • Negative — Only negative emotions were detected.

Speech also determines whether emotion has changed during calls lasting 90 seconds or longer. To do this, Speech compares emotion in the first third of the call (without considering the first 30 seconds) to emotion in the last third of the call.

Important: Medallia Speech has altered our acoustic emotion calculation to comply with new regulations in the EU per the EU AI Act. Currently, acoustic emotion is calculated using text-based emotion recognition only, and voice-based processing to determine emotion is discontinued.

Video files

Speech accepts video files for audio transcription through all data ingestion points that accept audio-only files. From the user's perspective, there is no functional difference between Speech processing for audio-video files and audio-only files. No special request parameter or connector configuration is required to process audio-video files. Both the Voci and Amazon Transcribe engines transcribe video files.

Note: Audio and video data must be uploaded in the same file to be processed together. Speech does not support combining audio and video files that have been uploaded separately.

Speech does not redact personally identifiable information (PII) from the video component of the audio-video file, but does redact the audio component. PII redaction for video is handled by the video recording platform.

Video data retention

Audio-only and audio/video files share the same data retention policy. The same setting configures retention for all submitted media. Videos cannot be downloaded, but video playback is available in Reporting everywhere that audio playback is available.

For more information, see Medallia Media File Transfer screen.