Import data for Text Analytics

Text Analytics supports importing data from different sources; imported data must be unstructured plain text and comply with format and content requirements

Medallia Text Analytics can analyze data sources besides surveys. These data sources include, but are not limited to:

  • Chat logs — For example, chat messages between customers and support personnel
  • Call center notes — For example, notes from call center representatives on the purpose and outcomes of customer calls
  • Emails — For example, the initial email messages received from a Contact Us link on the company website
  • Social media data — For example, Twitter and Facebook posts
    Note: Comment fields are divided into phrases during parsing, and individual phrases are limited to a maximum of 5000 characters. Phrases that exceed this limit are rare but result in the record's exclusion from Text Analytics parsing, including topic tagging and sentiment analysis. In addition to the phrase character maximum, Medallia AI models process up to a maximum of 60 tokens, or approximately 60 words, per phrase. For more information, see How rules are applied.
Important: Medallia Experience Cloud does not calculate an impact score for source types not associated with an overall satisfaction score.

Importing chat logs and other data

Experience Cloud uses the conversational transcript comment field (q_conversational_transcript_comment) to store two-way chat conversations, such as those between a customer and a call center agent. Several report modules are tailored specifically to display chat conversations in this field as a two-way dialog. For information about importing data to this field and displaying transcripts in reports, see Conversational transcript comment.

To import data from other sources, use Auto Importer to import the new files and then create a new set of Text Analytics topics. Optionally, create or modify reports to display data that does not rely on impact score calculations.

Import file format requirements

Important: Text Analytics is designed to process unstructured plain text data. When importing data from sources with structured or embedded data--for example, email or web form data--you must extract the unstructured plain text before importing the data to Text Analytics.

Importing data with any non-plain-text elements, including HTML tags, mail headers, and base64 encoding from emails, can have significant and unintended consequences for Text Analytics processing.

For example, sensitive data masking may be applied erroneously or not at all to data with HTML tags because the regular expressions that search for sensitive data fields cannot parse HTML. Other potential impacts of importing structured data to Text Analytics include parsing failure and compromised accuracy.

Imported files must also meet the following format requirements:

  • Each import file should have no more than 30 input fields. If there are more, contact Medallia Support for assistance.
  • For emails, the content column should include the email only, and should not include email metadata.

    Email content and metadata

  • For abbreviations in social media data, either replace them with the expanded version of the word or phrase, or remove them if they are not meaningful to the message.
  • For Twitter data, remove most of the user tags (names starting with @) from the tweets. For user tags of the company name (such @Serenity), remove the @ symbol only. Many retweets also contain references to the main user’s tweet, such as ^AD or <AD>. Remove those, as well.

Import file content requirements

At minimum, each record must include the following information. Work with your company to verify that the file has a separate column for each of these fields

  • Unique Record ID — Unique identifier for the record
  • Unit — Unit identifier
  • First name — First name of the customer
  • Last name — Last name of the customer
  • Email — Email address of the customer
  • Interaction date — Date to associate with the record
  • Content — The content of the email, to be analyzed by Text Analytics
  • Source — Identifies the data source, such as call center log, email, and so on
  • Language — Language of the content

For example:

The file has column for each field. Each record is a row.

Remove recurring text

If necessary, create a Regular expression find/replace preprocessor to remove any recurring text in the content. This prevents the text from being tagged by Text Analytics, so it will not influence data in reports. For example, if the beginning of every support call record includes the same introduction, remove that text.

A preprocessor that removes introductory text

For more information about creating preprocessors, see Auto Importer Preprocessors.

Import data using Auto Importer

Use the survey processor in Auto Importer to import the data to Experience Cloud. Map required data to Experience Cloud fields as shown in the following table:

Required dataOutput field
Unique Record IDA company E-field (for example, e_company_unique_record_id).
Unite_unitid
First namee_firstname
Last namee_lastname
Emaile_email
Interaction datee_responsedate
ContentA Q-field (for example, q_company_call_center_txt)
SourceA company E-field (for example e_company_source_alt). You must create and assign an enumerated alternative set to this E-field.
Typee_signal_type which determines the sentiment model used. Must be one of the following values:
  • Chat Transcripts
  • Call Transcripts / Speech Analytics
  • Support Tickets
  • Agent Notes
  • Product Reviews
  • Social Listening
  • Employee Ideas / Public Feedback
  • Forum Data
  • Survey
  • Digital Feedback
  • Mystery Shop Data
  • Account Management
  • Video
  • Messaging
  • Email
  • Digital Experience Analytics
  • Other
Languagesurvey_language

The following example shows how you might format an Auto Importer processor. For more information, see Auto Importer Processors.

<!-- Below is a starter Auto Importer for importing in non-survey data sources for Text Analytics. The fields below are the bare minimum to run TA on, but can always take additional data based on customer needs. -->
<import-spec name="Sample Auto Importer for Non-Survey Data Sources" description="An generic spec to import non-survey data records">
	<input-column heading="UNIQUE_RECORD_ID" />
	<!-- Unique identifier of the record -->
	<input-column heading="UNIT" />
	<input-column heading="FIRST_NAME" />
	<input-column heading="LAST_NAME" />
	<input-column heading="EMAIL" />
	<!-- Some sort of Transaction ID -->
	<input-column heading="INTERACTION_DATE" />
	<input-column heading="COMMENT" />
	<input-column heading="SOURCE" />
	<!-- Source Type (call center notes, chat log, etc.) -->
	<input-column heading="LANGUAGE" />
	<input-column heading="SIGNAL_TYPE" />
	<!-- 2 letter code preferred -->
	<output-column-group pluginName="Survey" recordUpdateMode="CREATE">
		<!-- duplicate check field mapping -->
		<output-column>
			<input-column heading="UNIQUE_RECORD_ID" />
			<target-field fieldId="e_company_unique_record_id" fieldName="Unique Record ID" requiredness="REQUIRED_USED_FOR_DUPLICATE_CHECK" type="STRING">
				<email-field-parse-options />
			</target-field>
		</output-column>
		<!-- unit mapping -->
		<output-column>
			<input-column heading="UNIT" />
			<target-field fieldId="e_unitid" fieldName="Unit" requiredness="REQUIRED" type="UNIT">
				<enumerated-field-parse-options mappingKey="IDENTIFIER" />
			</target-field>
		</output-column>
		<!-- customer detail mapping -->
		<output-column>
			<input-column heading="FIRST_NAME" />
			<target-field fieldId="e_firstname" fieldName="First name" requiredness="OPTIONAL" type="STRING" />
		</output-column>
		<output-column>
			<input-column heading="LAST_NAME" />
			<target-field fieldId="e_lastname" fieldName="Last name" requiredness="OPTIONAL" type="STRING" />
		</output-column>
		<output-column>
			<input-column heading="EMAIL" />
			<target-field fieldId="e_email" fieldName="Email" requiredness="OPTIONAL" type="EMAIL">
				<email-field-parse-options />
			</target-field>
		</output-column>
		<output-column>
			<input-column heading="INTERACTION_DATE" />
			<target-field fieldId="e_responsedate" fieldName="Responsedate" requiredness="OPTIONAL" type="TIMESTAMP">
				<timestamp-field-parse-options format="dd-MMM-yyyy hh:mm:ss a" timeZone="America/Los_Angeles" />
			</target-field>
		</output-column>
		<output-column>
			<input-column heading="COMMENT" />
			<target-field fieldId="e_company_comment_txt" fieldName="Comment Field for Data Source" requiredness="OPTIONAL" type="STRING" />
		</output-column>
		<!-- data source mapping -->
		<output-column>
			<input-column heading="SOURCE" />
			<target-field fieldId="e_company_source_alt" fieldName="Source" requiredness="OPTIONAL" type="ENUMERATED">
				<enumerated-field-parse-options mappingKey="NAME" />
			</target-field>
		</output-column>
		<!-- language mapping -->
		<output-column>
			<input-column heading="LANGUAGE" />
			<target-field fieldId="survey_language" fieldName="Survey Language" requiredness="OPTIONAL" type="ENUMERATED">
				<enumerated-field-parse-options mappingKey="NAME" />
			</target-field>
		</output-column>
		<!-- signal type -->
		<output-column>
			<input-column heading="SIGNAL_TYPE" />
			<target-field fieldId="e_signal_type" fieldName="Signal Type" requiredness="REQUIRED" type="STRING" />
		</output-column>
		<!-- map status of record as completed -->
		<output-column>
			<constant value="COMPLETED" />
			<target-field fieldId="e_status" fieldName="Status" requiredness="REQUIRED" type="ENUMERATED">
				<enumerated-field-parse-options mappingKey="NAME" />
			</target-field>
		</output-column>
		<transfer-to-survey-engine />
		<survey-processor-options />
	</output-column-group>
</import-spec>

Create a new topic set and rules for imported data

Consider creating a new topic set and rules to support the new data being imported to Experience Cloud. You can duplicate an existing topic set and rules for this purpose.

Create a new topic set on the Reporting > Text Processing > Topic & Theme Tagging > Auto Tagging Pools screen, as described in Creating a tag pool. For the MemberSurveyFields property, choose the Q-field you specified for your imported content (for example, q_company_call_center_txt).

Copy the rules associated with each topic from an existing topic set to the new topic set, as described in the following steps.

  1. Open Topic Builder in Medallia Admin Suite.
  2. In the Text Source dropdown, select the topic set containing the rules you want to copy.
  3. Filter the list of topics to show only published topics.

    Show only published topics

  4. Select Topic to select all topics in the topic set.

    Select all topics

  5. Export the selected topics:
    1. Click Export.
    2. Enter a name for the export file.
    3. In the dropdown, select Export topics and rules (XML), and then click Export.

      Export dialog

  6. In the Text Source dropdown, select the new topic set.
  7. Import the topics and rules you exported earlier:
    1. Click Import.
    2. In the dialog, click Choose file, and then select the file you exported.
    3. Click Import.

      Import dialog

  8. Review the imported topics and rules.

    Consider which roles will be able to see these topics by default, and adjust permissions as necessary. For more information about assigning permissions, see Roles.

  9. When you are ready to publish the new topics, select Topic to select all of the topics in the topic set, and then click Publish.

Create or modify reports

As needed, create or modify reports to show data for the new topics. For modules that show scores, make sure impact score is not configured for display. For example, in a TA Topic/Theme Summary module, make sure no impact score metric is selected in the Metrics to Display property.