Feed Pulls
Integrations > Data Import > Importers > Feed Pulls
Feed Pulls import data into Medallia by retrieving files from an external system. Feed Pulls run on a schedule and access the external system via a file transfer protocol (FTP). After retrieving the external file, the Feed Pull can optionally trigger another Feed Pull to retrieve additional file(s) when the files must be processed sequentially.
After the importer successfully updates the database, the feed can optionally trigger an Exporter to send data to an external system.
Some clients cannot push files to Medallia due to corporate compliance or security regulations; instead, they post files on their servers for retrieval by a Feed Pull.
ListBucket and GetObject actions.For information about the servers, see Medallia feed servers and filestash.
Properties
- Name
- Name of this feed; use something descriptive of the feed and which distinguishes it from other feeds.
- Description
- The description of this feed; should include the purpose and information about the expected source system and Auto Importer specification.
- Processing mode
-
Processing state of this feed.
- Active — Incoming files are processed and records are added to the Experience Cloud.
- Inactive — Incoming files are not processed.
- Upload with pretend processing — The feed is active, but records are not added to the database. Use this to test the feed.
After Active or Pretend jobs, the processing report shows the statistics of the event
- Importer traffic
-
Reports statistics about recent active and pretend processing activities for this feed, including successful and failed feeds over time. All times are based on the current time when the page displays — for example, "Past Day" is 24 hours before the page displayed, not everything in the previous day. This can be different from other reports that show activity from the previous day, which would be all traffic that day.
This information appears only after the feed has processed files.
- Lifetime of Feed Files
- Count of days to retain imported files and processing reports. After this many calendar days, the files are purged from the data centers. Default is to use the Import Deletion Schedule setting on the Company settings screen. See the description of that field for considerations about choosing a value. To be compliant with the E.U. General Data Protection Regulation (GDPR), this value should be 30 days or less.
This setting does NOT affect files retained on company-hosted or Medallia-hosted FTP/SFTP hosts.
Important: To comply with GDPR policies, files are deleted after 30 days.
Pull configuration
- Schedule
- When the feed will pull files from the source system. Pick one of the defined Schedules, or leave set to None to initiate the pull manually with the Pull feed now button.
- Next pull
- Timestamp when the next scheduled pull will happen. When there is no scheduled event, the value is N/A (no local FeedPull service).
- Pull feed now
- Manually initiate the feed pull.
- Pull method
-
File transfer protocol to use to connect to the remote system and transfer the file. One of:
- FTP — Standard FTP.
- SFTP (password authentication) — Secure FTP using a password for authentication; encrypts both commands and data.Important: Password authentication is deprecated and will be obsolete in a future release. Preferentially, use public key authentication for new connections with Medallia-owned servers or client-owned servers. For information about public key authentication, see Generating SSH Key pairs.
- SFTP (public key authentication) — SFTP using a key-pair or certificate for authentication.
- FTPS (explicit mode) — Secure FTP using a root certificate from a certificate authority.
- Amazon S3 (key authentication) — Secure FTP from an Amazon S3 bucket; requires an AWS access account.
Note: After changing the Pull method, click Save to refresh the screen and see the options specific to the chosen method. - Hostname
- Remote host name and optional port number if not the default for the protocol. The format is: hostName:portNumber. Omit the colon and port number for the default port.
- Login
- Account to access the remote directory.Warning: Changing the Login value makes existing data inaccessible; specifically, any the already pulled files file's, and all related processing reports and detailed Excel reports will no longer be available.
- Password
- Password for the Login account.
- SFTP private key
- Private key for generating signatures (to verify the connection matches Medallia’s Public key). For more information about authentication keys, see SSH key pairs.
- Passphrase for the SFTP private key
- (optional) Passphrase associated with the key and is necessary to use the private key (when the key was generated with a passpharse), similar to a password.
- FTPS certificate
- Root certificate to use when comparing the server's certificate.
- Directory
- Directory on the remote system to retrieve the file.
- Bucket name
- Name of the Amazon S3 bucket that holds the file. Do not include the file path; instead, use the File regex to locate files in subdirectories.Tip: To set up a sub-folder in a Feed Pull to Amazon S3, in the account, set the bucket name on the export to "root/subfolder", with "root" being the bucket and "subfolder" being the folder on that bucket.
- Amazon S3 Access key ID
- Access key for the Amazon Web Service (AWS) account that owns the bucket.
- Amazon S3 Secret key
- Secret key for the AWS account that owns the bucket.
- Amazon S3 file prefix
- Only pull files whose keys begin with this prefix (folder path). Use a prefix when the access key does not have access to the root folder, such as when the S3 bucket is not a Medallia account.Restriction: DO NOT start the prefix with a slash '/', but include one at the end. For example: 'sub/folder/'
- Amazon S3 endpoint
- Amazon S3 regional endpoint to access the feed; for example, an endpoint that includes a "region" is
s3.us-east-1.amazonaws.com. When there is no endpoint, the default iss3.amazonaws.com.This value does not accept S3 endpoints for China regions (ending in .cn).
- Amazon S3 region
- OBSOLETE, merged into Amazon S3 endpoint.
- File regex (for Amazon pull method only)
- Only download files that match this regular expression. For an Amazon S3 pull, the prefix (if included) is automatically prepended to the file regex. For example, consider this file path:
medallia/in/tier3testfile.txtBoth of these prefix and regex combinations access the file:
Prefix: "medallia/in/"
File regex: "tier3testfile.txt"Prefix: "medallia/in"
File regex: "/tier3testfile.txt"However, if the accessing account does have access to the root folder, you can omit the Prefix and put the entire file path in the regex.
Prefix: ""
File regex: "medallia/in/tier3testfile.txt"To search all subdirectories in a bucket, begin the regex with ".+". For example, to find testfile.txt in any subdirectory:
.+testfile.txt - File regex
-
Regex that identifies files to process. Only files that match the regex are processed by this feed. To match all files, use this expression:
.*This example
phone_.*.txt.pgpmatches these files:
iphone_20090101.txt.pgp iphone_20090201.txt.pgpbut not this file
property_20090101.txt.pgpThis example
OmniExport-\d+-\d+\.csvmatches these filles:
OmniExport-200901-1201.csv OmniExport-20090501-01.csv - Pull after date
-
Only pull files newer than this timestamp. This is useful when there are irrelevant historic files on the same directory.
Note, the feed pull never automatically retrieves the same file more than once.
- Directory depth
- Count of subdirectory level below the Directory location to search for files to pull. Use 0 (or blank) to search the named directory only. Otherwise, the feed pull searches subdirectories up to the level specified. For example, a depth of 1 searches just child directories, while a depth of 2 also search grandchildren.
- Disable dup check across Feed Pulls
- Allow the feed to pull a file already retrieved by another feed pull. Otherwise, when off, only one feed pull may retrieve a file.
To determine if two or more files are duplicates, Medallia Experience Cloud compares the files' attributes (filename, size, timestamp), and also compares the files' hash value to determine if they are identical or different in content.
This setting is useful, for example, when a company wants to use the same file for updating the Org Hierarchy and for sending Invitations.
- Remove source file after successful transfer
- Removes the source file from the server once it has been transferred to Medallia Experience Cloud but before processing occurs. When off, the source file remains on the server and the file's attributes (filename, size, modification timestamp, etc.) determine whether the file should be excluded from future pulls.Important: For this feature to work as intended, the S3 policy must allow the
DeleteObjectaction.
Record processor
- Auto Importer processor
- The Auto Importer specification that processes the incoming data.
- Lego processor
- Lego processor to process the retrieved file.
- (Legacy) Convert format
-
(Deprecated) Convert format to process the retrieved file.
Warning: After setting up a new feed account with LEGO/ConvertFormat processor, test it using Kind UPLOAD_WITH_PRETEND_PROCESSING. Check the processing report carefully to make sure that the feed data is getting processed as expected. This is particularly important with stay files or survey import files (that is, anything that generates or creates new records in Experience Cloud): deleting survey records on a production server is extremely tedious and/or involves scarce engineering resources. But in general, this is a good practice for testing feed accounts that process all kind of data.
Processing options
- Primary UnitGroup
-
(Auto Importer only) Sets the From address of the report e-mails. When this property is set, the Auto-Importer does not determine the survey permission and or do duplicate checking.
(Lego only) Only accept records with Unit IDs in the selected Unit groups. Records for Units not in this Unit group scope are rejected. Applies only to LegoProcessType of SURVEY_IMPORT and SURVEY_UPDATE. It does not work for other process types such as STAY_IMPORT, EMAIL_IMPORT, or OPTOUT_IMPORT.
- Additional UnitGroups
- (Lego only) Additional Unit Groups to use for the scope of survey imports or updates.
- Preprocessor
- Only for Auto Importers. Which Auto Importer Preprocessors to use on incoming data before passing it to the Auto Importer processor.
- Seconds to wait for grouping
-
Only for Auto Importers. How long to wait (seconds) when collecting feeds before compiling them into a single feed-file.
For Web Feeds, this option is only available for Feed-file mode.
Warning: Feed-files are always converted to CSV format in the group, regardless of the original format included in the request content. Sometimes columns/fields included in the request do not appear in the CSV files (regardless of thecolumnsChecksetting in Auto Importer specification). To ensure the intended fields are always included in the CSV file, reference each field with<input-column ...>in the Auto Importer specification.Use this option for integrations that receive a high volume of requests, but where processing those requests immediately as they arrive could cause potential performance problems, such as when updating the Org Hierarchy (Unit or Unit groups).
Restriction: Grouping is mandatory for new Web Feeds using Auto Importers with Org Hierarchy processors. The recommended minimum value for this parameter is 120 seconds. Existing Web Feeds for Org Hierarchy where grouping is not active can be updated, but Experience Cloud shows a warning message instructing you to activate this feature.Additional notes:
- Feed files grouping starts when the first feed-file arrives. For example, when the wait time is 3600 seconds (1 hour), and the first feed file arrives at 1:17 PM, and other feed files arrive at 1:20 PM, 1:36 PM, 1:52 PM; then all 4 feed files are processed at 2:17 PM. If the next file arrives at 2:20 PM, all the files for that hour are executed at 3:20 PM.
- Generally, the files are processed in the order they arrive. If two files arrive at near the same time, it is possible their order would be reversed.
- The Feed Files screen shows the grouped files after the group is processed. For a brief period of time, individual files may be shown, but eventually they are collected into the single group.
- Changing this value on a live instance affects the start of the next group; it does not change the wait time of any groups in progress.
- If there is a problem with the instance while processing a group, it is possible later files in the group are processed later than the earlier files in the same group. For example, if the instance restarts, the wait-time is reapplied to the remaining files, and they are processed when the new wait-time expires.
- Pre-processing happens as the files arrive: the pre-processed results are in the group.
If you are using this option and change the processing mode (e.g. from pretend to active) while the current grouping is still ongoing, note that, once the grouping period completes, the grouped file will be imported with the current processing mode configured on the web feed. For example: if there is a wait time set to 3,600 seconds, and the first feed file arrives at 1:17 PM, and other feed files arrive at 1:20 PM, 1:36 PM, 1:52 PM, and you change the processing mode from pretend to active at 2:05 PM, all 4 grouped files are imported at 2:17 PM in active processing mode.
- No files pulled alert e-mail
- (Auto Importer only) A comma-separated list of email addresses to receive a report when the feed pull imports no zero files at the scheduled time.
Email report
- Summarized report e-mails
- Only for Auto Importers. Email addresses (comma-separated) to receive the summarized processing report of successes and failures.
- Lego e-mail template
- Only for Lego. E-mail template to use for the processing report.
Post-processing options
Post-processing actions are optional activities to happen after the Auto Importer successfully updates the database. These do not happen when the importer fails to update the database, such as when a record is invalid or when a file is processed with Upload with pretend processing.
- Trigger an export after file is processed
- Turn on trigger export processing after successfully importing the data. After turning this on and saving the specification, the Trigger section will be available to define the trigger.
Testing
- Upload a file (save to submit)
-
Manually select a file to upload and process, as if the file had been retrieved by this feed pull. Useful for testing.
Tip: Small files upload quickly. However, large files may take awhile: on the order of several minutes. Very large files have been known to take up to 20 minutes to process before they appear in the Feed Files list and before email notifications are sent.To create a simple test Feed Pull, use these settings
Property Value Name testProcessing mode Upload with pretend processing Pull method FTP Hostname test.example.comLogin testFile regex .*
Troubleshooting tips
- Corrupt files
- All Feed pulls are done in binary mode; do not attempt to use ASCII mode, even for text records. Doing an ASCII transfer of a binary file corrupts the file. Further, success reports typically show twice as many records imported as included in the source file.
- Cannot access remote system
- Try using a FTP utility to manually connect to the external system. Verify the credentials and settings are the same for the feed pull.
- SFTP issues
- In the past, some SFTP vendors did not support the SFTP software Medallia Experience Cloud uses (e.g., Tumbleweed has had problems in the past). In those situations, the companies push a file to an intermediate SFTP server hosted by Medallia (Medallia Media File Transfer), and then the Feed Pull accesses the file(s) from that server using SFTP. Accounts using SFTP Pull must use STANDARD COMPLIANT OpenSSH2.
