Evaluation criteria

Table 1. Language Model Evaluation Criteria

Readability

The ease with which the transcript can be read through while maintaining clarity. This refers to the human perception of the transcript rather than accuracy or word error rate.

Correct: I would like to know the remaining balance in my account.

Incorrect: iPhone like to go remaining balance in Maya count.

Analytics

The accuracy of key phrases that are used in analytics (for example text search, V-Spark applications, etc.) that are present in the call.

Correct: Have you tried doing it over the web portal?

Incorrect: Half you tribe doing it oven the web panel?

Vocabulary

The accuracy of proper nouns and domain-specific terminology.

Correct: I have you confirmed for a flight to Cancun on Lufthansa.

Incorrect: I have you confirmed for a flight to raccoon on lasagna.

Metadata

The accuracy of the metrics displayed on the file details page including agent emotion, client emotion, agent gender, client gender, etc.

Punctuation

The properness of the punctuation including capitalization, periods, question marks, commas, etc.