Amazon S3 FTP Pulls

Amazon S3 Feed Pulls import data into Medallia Experience Cloud by retrieving files from an Amazon S3 bucket. This import method requires an AWS access account. Feed Pulls run on a schedule and access the Amazon S3 bucket via a file transfer protocol (FTP). After retrieving the external file, the Feed Pull can optionally trigger another Feed Pull to retrieve additional file(s) when the files must be processed sequentially.

Feed flow: S3 bucket to FTP Feed Pull.

Some clients cannot push files to Experience Cloud due to corporate compliance or security regulations; instead, they post files on their Amazon S3 buckets for retrieval by a Feed Pull.

Important: The S3 policy must allow the ListBucket and GetObject actions. For the Remove source file after successful transfer feature to work as intended, the S3 policy must additionally allow the DeleteObject action.
Tip: Always use unique filenames for the files to pull, and use the File regex to recognize all variations of the filename. When running on a schedule, the Feed Pull looks for files that have been updated since the last time a Feed Pull ran. Only files with newer timestamps are pulled. Once retrieved, the Feed Pull checks to see if there have been any other files pulled with the same name. If the filename is a repeat of a previously pulled file, the system compares the contents of the two files, and then only if they are different, does the file get passed to the importer. To avoid any confusion, it is best to use unique filenames.

For information about the servers, see Medallia feed servers and filestash.

Creating a Feed Pull

The general steps for creating a feed are:

  1. Create a new import method:

    1. From Import Methods screen, click Add Method. This opens the library of available import methods.

    2. Select the import method from the list. This opens the Method Editor, where you can build your import method.

    3. In the Details tab:

      1. Enter a name and a description.

      2. Provide the Bucket Name, Access Key ID, Secret Key, Endpoint, and File prefix to access the bucket.

    4. In the Schedule tab, configure the schedule for the method.

    5. In the Advanced tab, specify the File regex that matches the name of the file(s) to retrieve from the bucket and to import.

    6. Click Done.

  2. Change the Processing Mode to active.

  3. Inform the company that they may begin feeding files to the Amazon S3 Bucket.

Details

Method Name
Name of this import method.
Description
The description of this method; should include the purpose and information about the expected source system and Importer.
Amazon S3 Bucket Name
Name of the Amazon S3 bucket that holds the file. Do not include the file path; instead, use the File regex to locate files in subdirectories.
Amazon S3 Access Key ID
Access key for the Amazon Web Service (AWS) account that owns the bucket.
Amazon S3 Secret Key
Secret key for the AWS account that owns the bucket.
Amazon S3 Endpoint
Amazon S3 regional endpoint to access the feed; for example, an endpoint that includes a "region" is s3.us-east-1.amazonaws.com. When there is no endpoint, the default is s3.amazonaws.com.

This value does not accept S3 endpoints for China regions (ending in .cn).

Amazon S3 File Prefix
Only pull files whose keys begin with this prefix (folder path). Use a prefix when the access key does not have access to the root folder, such as when the S3 bucket is not a Medallia account.
Restriction: Do not start the prefix with a slash '/', but include one at the end. For example: sub/folder/.

Schedule

Schedules determine when to automatically run an importer and ingest a file. For more details see Schedules.

Days of the week
Either Every day of the week, or only on the selected days.
Days of the month
Either Every day of the month, or only on the selected days.
Months
Either Every month of the year, or only in the selected months.
Time
Either at the specified time, or at the specified minute of the hour, every hour of the day.

Advanced

Send Processing Reports To
Email addresses (comma-separated) to receive the summarized processing report of successes and failures for that particular importer. Processing reports are created at a file level.
Directory
Directory on the remote system to retrieve the file, relative to the login directory.
Note: Start with a slash '/' when specifying an absolute path.
Directory Depth
Count of subdirectory level below the Directory location to search for files to pull. Use 0 (or blank) to search the named directory only. Otherwise, the feed pull searches subdirectories up to the level specified. For example, a depth of 1 searches just child directories, while a depth of 2 also searches grandchildren.
File Lifetime
Count of days to retain imported files and processing reports. After this many calendar days, the files will be purged from the data centers. Default is to use the Import Deletion Schedule setting on the Company settings screen. See the description of that field for considerations about choosing a value. To be compliant with the E.U. General Data Protection Regulation (GDPR), this value should be 30 days or less.

This setting does NOT affect files retained on company-hosted or Medallia-hosted FTP/SFTP hosts.

Important: To comply with GDPR policies, files are deleted after 30 days.
Pull After Date

Only pull files newer than this timestamp. This is useful when there are irrelevant historic files in the same directory.

Note: The feed pull never automatically retrieves the same file more than once.
File Regex

Regex that identifies files to process. Only files that match the regex are processed by this feed. To match all files, use this expression:

.*

This example 

phone_.*.txt.pgp

matches these files:

iphone_20090101.txt.pgp
iphone_20090201.txt.pgp

but not this file:

property_20090101.txt.pgp

This example 

OmniExport-\d+-\d+\.csv

matches these files:

OmniExport-200901-1201.csv
OmniExport-20090501-01.csv
Disable duplicate checking across Feed Pulls
Allow the method to pull a file already retrieved by another import method. Otherwise, when off, only one import method may retrieve a file. This setting is useful, for example, when a company wants to use the same file for updating the Org Hierarchy and for sending invitations.
Remove source file after successful transfer
Removes the source file from the server once it has been transferred to Medallia Experience Cloud but before processing occurs. When off, the source file remains on the server and the file's attributes (filename, size, modification timestamp, etc.) determine whether the file should be excluded from future pulls.