Feed Pulls

Integrations > Data Import > Importers > Feed Pulls

Feed Pulls import data into Medallia by retrieving files from an external system. Feed Pulls run on a schedule and access the external system via a file transfer protocol (FTP). After retrieving the external file, the Feed Pull can optionally trigger another Feed Pull to retrieve additional file(s) when the files must be processed sequentially.

After the importer successfully updates the database, the feed can optionally trigger an Exporter to send data to an external system.

Feed flow: external system to FTP Feed Pull then to Auto Importer wth optional flow to exporter back to external

Some clients cannot push files to Medallia due to corporate compliance or security regulations; instead, they post files on their servers for retrieval by a Feed Pull.

Important: When files are posted on an S3 Bucket, the S3 policy must allow the ListBucket and GetObject actions.
Tip: Always use unique filenames for the files to pull, and use the File regex to recognize all variations of the filename. When running on a schedule, the Feed Pull looks for files that have been updated since the last time a Feed Pull ran. Only files with newer timestamps are pulled. Once retrieved, the Feed Pull checks to see if there have been any other files pulled with the same name. If the filename is a repeat of a previously pulled file, the system compares the contents of the two files, and then only if they are different, does the file get passed to the importer. To avoid any confusion, it is best to use unique filenames.

For information about the servers, see Medallia feed servers and filestash.

Properties

Name
Name of this feed; use something descriptive of the feed and which distinguishes it from other feeds.
Description
The description of this feed; should include the purpose and information about the expected source system and Auto Importer specification.
Processing mode

Processing state of this feed.

  • Active — Incoming files are processed and records are added to the Experience Cloud.
  • Inactive — Incoming files are not processed.
  • Upload with pretend processing — The feed is active, but records are not added to the database. Use this to test the feed.

After Active or Pretend jobs, the processing report shows the statistics of the event

Importer traffic

Reports statistics about recent active and pretend processing activities for this feed, including successful and failed feeds over time. All times are based on the current time when the page displays — for example, "Past Day" is 24 hours before the page displayed, not everything in the previous day. This can be different from other reports that show activity from the previous day, which would be all traffic that day.

This information appears only after the feed has processed files.

Lifetime of Feed Files
Count of days to retain imported files and processing reports. After this many calendar days, the files are purged from the data centers. Default is to use the Import Deletion Schedule setting on the Company settings screen. See the description of that field for considerations about choosing a value. To be compliant with the E.U. General Data Protection Regulation (GDPR), this value should be 30 days or less.

This setting does NOT affect files retained on company-hosted or Medallia-hosted FTP/SFTP hosts.

Important: To comply with GDPR policies, files are deleted after 30 days.

Pull configuration

Schedule
When the feed will pull files from the source system. Pick one of the defined Schedules, or leave set to None to initiate the pull manually with the Pull feed now button.
Next pull
Timestamp when the next scheduled pull will happen. When there is no scheduled event, the value is N/A (no local FeedPull service).
Pull feed now
Manually initiate the feed pull.
Pull method

File transfer protocol to use to connect to the remote system and transfer the file. One of:

  • FTP — Standard FTP.
  • SFTP (password authentication) — Secure FTP using a password for authentication; encrypts both commands and data.
    Important: Password authentication is deprecated and will be obsolete in a future release. Preferentially, use public key authentication for new connections with Medallia-owned servers or client-owned servers. For information about public key authentication, see Generating SSH Key pairs.
  • SFTP (public key authentication) — SFTP using a key-pair or certificate for authentication.
  • FTPS (explicit mode) — Secure FTP using a root certificate from a certificate authority.
  • Amazon S3 (key authentication) — Secure FTP from an Amazon S3 bucket; requires an AWS access account.
Note: After changing the Pull method, click Save to refresh the screen and see the options specific to the chosen method.
Hostname
Remote host name and optional port number if not the default for the protocol. The format is: hostName:portNumber. Omit the colon and port number for the default port.
Login
Account to access the remote directory.
Warning: Changing the Login value makes existing data inaccessible; specifically, any the already pulled files file's, and all related processing reports and detailed Excel reports will no longer be available.
Password
Password for the Login account.
SFTP private key
Private key for generating signatures (to verify the connection matches Medallia’s Public key). For more information about authentication keys, see SSH key pairs.
Passphrase for the SFTP private key
(optional) Passphrase associated with the key and is necessary to use the private key (when the key was generated with a passpharse), similar to a password.
FTPS certificate
Root certificate to use when comparing the server's certificate.
Directory
Directory on the remote system to retrieve the file.
Bucket name
Name of the Amazon S3 bucket that holds the file. Do not include the file path; instead, use the File regex to locate files in subdirectories.
Tip: To set up a sub-folder in a Feed Pull to Amazon S3, in the account, set the bucket name on the export to "root/subfolder", with "root" being the bucket and "subfolder" being the folder on that bucket.
Amazon S3 Access key ID
Access key for the Amazon Web Service (AWS) account that owns the bucket.
Amazon S3 Secret key
Secret key for the AWS account that owns the bucket.
Amazon S3 file prefix
Only pull files whose keys begin with this prefix (folder path). Use a prefix when the access key does not have access to the root folder, such as when the S3 bucket is not a Medallia account.
Restriction: DO NOT start the prefix with a slash '/', but include one at the end. For example: 'sub/folder/'
Amazon S3 endpoint
Amazon S3 regional endpoint to access the feed; for example, an endpoint that includes a "region" is s3.us-east-1.amazonaws.com. When there is no endpoint, the default is s3.amazonaws.com.

This value does not accept S3 endpoints for China regions (ending in .cn).

Amazon S3 region
OBSOLETE, merged into Amazon S3 endpoint.
File regex (for Amazon pull method only)
Only download files that match this regular expression. For an Amazon S3 pull, the prefix (if included) is automatically prepended to the file regex. For example, consider this file path:
medallia/in/tier3testfile.txt

Both of these prefix and regex combinations access the file:

Prefix: "medallia/in/"
File regex: "tier3testfile.txt"

Prefix: "medallia/in"
File regex: "/tier3testfile.txt"

However, if the accessing account does have access to the root folder, you can omit the Prefix and put the entire file path in the regex.

Prefix: ""
File regex: "medallia/in/tier3testfile.txt"

To search all subdirectories in a bucket, begin the regex with ".+". For example, to find testfile.txt in any subdirectory:

.+testfile.txt
File regex

Regex that identifies files to process. Only files that match the regex are processed by this feed. To match all files, use this expression:

.*

This example

phone_.*.txt.pgp

matches these files:

iphone_20090101.txt.pgp
iphone_20090201.txt.pgp 

but not this file

property_20090101.txt.pgp 

This example

OmniExport-\d+-\d+\.csv

matches these filles:

OmniExport-200901-1201.csv
OmniExport-20090501-01.csv
Pull after date

Only pull files newer than this timestamp. This is useful when there are irrelevant historic files on the same directory.

Note, the feed pull never automatically retrieves the same file more than once.

Directory depth
Count of subdirectory level below the Directory location to search for files to pull. Use 0 (or blank) to search the named directory only. Otherwise, the feed pull searches subdirectories up to the level specified. For example, a depth of 1 searches just child directories, while a depth of 2 also search grandchildren.
Disable dup check across Feed Pulls
Allow the feed to pull a file already retrieved by another feed pull. Otherwise, when off, only one feed pull may retrieve a file.

To determine if two or more files are duplicates, Medallia Experience Cloud compares the files' attributes (filename, size, timestamp), and also compares the files' hash value to determine if they are identical or different in content.

This setting is useful, for example, when a company wants to use the same file for updating the Org Hierarchy and for sending Invitations.

Remove source file after successful transfer
Removes the source file from the server once it has been transferred to Medallia Experience Cloud but before processing occurs. When off, the source file remains on the server and the file's attributes (filename, size, modification timestamp, etc.) determine whether the file should be excluded from future pulls.
Important: For this feature to work as intended, the S3 policy must allow the DeleteObject action.

Record processor

Choose only one processor: either Auto Importer or Lego.
Auto Importer processor
The Auto Importer specification that processes the incoming data.
Lego processor
Lego processor to process the retrieved file.
(Legacy) Convert format

(Deprecated) Convert format to process the retrieved file.

Warning: After setting up a new feed account with LEGO/ConvertFormat processor, test it using Kind UPLOAD_WITH_PRETEND_PROCESSING. Check the processing report carefully to make sure that the feed data is getting processed as expected. This is particularly important with stay files or survey import files (that is, anything that generates or creates new records in Experience Cloud): deleting survey records on a production server is extremely tedious and/or involves scarce engineering resources. But in general, this is a good practice for testing feed accounts that process all kind of data.

Processing options

Primary UnitGroup

(Auto Importer only) Sets the From address of the report e-mails. When this property is set, the Auto-Importer does not determine the survey permission and or do duplicate checking.

(Lego only) Only accept records with Unit IDs in the selected Unit groups. Records for Units not in this Unit group scope are rejected. Applies only to LegoProcessType of SURVEY_IMPORT and SURVEY_UPDATE. It does not work for other process types such as STAY_IMPORT, EMAIL_IMPORT, or OPTOUT_IMPORT.

Additional UnitGroups
(Lego only) Additional Unit Groups to use for the scope of survey imports or updates.
Preprocessor
Only for Auto Importers. Which Auto Importer Preprocessors to use on incoming data before passing it to the Auto Importer processor.
Seconds to wait for grouping

Only for Auto Importers. How long to wait (seconds) when collecting feeds before compiling them into a single feed-file.

For Web Feeds, this option is only available for Feed-file mode.

Warning: Feed-files are always converted to CSV format in the group, regardless of the original format included in the request content. Sometimes columns/fields included in the request do not appear in the CSV files (regardless of the columnsCheck setting in Auto Importer specification). To ensure the intended fields are always included in the CSV file, reference each field with <input-column ...> in the Auto Importer specification.

Use this option for integrations that receive a high volume of requests, but where processing those requests immediately as they arrive could cause potential performance problems, such as when updating the Org Hierarchy (Unit or Unit groups).

Restriction: Grouping is mandatory for new Web Feeds using Auto Importers with Org Hierarchy processors. The recommended minimum value for this parameter is 120 seconds. Existing Web Feeds for Org Hierarchy where grouping is not active can be updated, but Experience Cloud shows a warning message instructing you to activate this feature.

Additional notes:

  • Feed files grouping starts when the first feed-file arrives. For example, when the wait time is 3600 seconds (1 hour), and the first feed file arrives at 1:17 PM, and other feed files arrive at 1:20 PM, 1:36 PM, 1:52 PM; then all 4 feed files are processed at 2:17 PM. If the next file arrives at 2:20 PM, all the files for that hour are executed at 3:20 PM.
  • Generally, the files are processed in the order they arrive. If two files arrive at near the same time, it is possible their order would be reversed.
  • The Feed Files screen shows the grouped files after the group is processed. For a brief period of time, individual files may be shown, but eventually they are collected into the single group.
  • Changing this value on a live instance affects the start of the next group; it does not change the wait time of any groups in progress.
  • If there is a problem with the instance while processing a group, it is possible later files in the group are processed later than the earlier files in the same group. For example, if the instance restarts, the wait-time is reapplied to the remaining files, and they are processed when the new wait-time expires.
  • Pre-processing happens as the files arrive: the pre-processed results are in the group.
  • If you are using this option and change the processing mode (e.g. from pretend to active) while the current grouping is still ongoing, note that, once the grouping period completes, the grouped file will be imported with the current processing mode configured on the web feed. For example: if there is a wait time set to 3,600 seconds, and the first feed file arrives at 1:17 PM, and other feed files arrive at 1:20 PM, 1:36 PM, 1:52 PM, and you change the processing mode from pretend to active at 2:05 PM, all 4 grouped files are imported at 2:17 PM in active processing mode.

No files pulled alert e-mail
(Auto Importer only) A comma-separated list of email addresses to receive a report when the feed pull imports no zero files at the scheduled time.

Email report

Summarized report e-mails
Only for Auto Importers. Email addresses (comma-separated) to receive the summarized processing report of successes and failures.
Lego e-mail template
Only for Lego. E-mail template to use for the processing report.

Post-processing options

Post-processing actions are optional activities to happen after the Auto Importer successfully updates the database. These do not happen when the importer fails to update the database, such as when a record is invalid or when a file is processed with Upload with pretend processing.

Trigger an export after file is processed
Turn on trigger export processing after successfully importing the data. After turning this on and saving the specification, the Trigger section will be available to define the trigger.

Testing

Upload a file (save to submit)

Manually select a file to upload and process, as if the file had been retrieved by this feed pull. Useful for testing.

Tip: Small files upload quickly. However, large files may take awhile: on the order of several minutes. Very large files have been known to take up to 20 minutes to process before they appear in the Feed Files list and before email notifications are sent.

To create a simple test Feed Pull, use these settings

PropertyValue
Name test
Processing mode Upload with pretend processing
Pull method FTP
Hostname test.example.com
Login test
File regex .*

Troubleshooting tips

Corrupt files
All Feed pulls are done in binary mode; do not attempt to use ASCII mode, even for text records. Doing an ASCII transfer of a binary file corrupts the file. Further, success reports typically show twice as many records imported as included in the source file.
Cannot access remote system
Try using a FTP utility to manually connect to the external system. Verify the credentials and settings are the same for the feed pull.
SFTP issues
In the past, some SFTP vendors did not support the SFTP software Medallia Experience Cloud uses (e.g., Tumbleweed has had problems in the past). In those situations, the companies push a file to an intermediate SFTP server hosted by Medallia (Medallia Media File Transfer), and then the Feed Pull accesses the file(s) from that server using SFTP. Accounts using SFTP Pull must use STANDARD COMPLIANT OpenSSH2.