SFTP Feeds

Integration > Data Import > Importers > SFTP Feeds

SFTP Feeds accept files for import from external systems using secure file transfer protocol (SFTP). SFTP Feeds trigger when a file arrives on Medallia's SFTP server, and the file matches the feed specification.

After the importer successfully updates the database, the feed can optionally trigger an Exporter to send data to an external system.

Feed flow: external system to Medallia SFTP server to FTP/SFTP Feed, then to Auto Importer wth optional flow to exporter back to external

Medallia has two general classes of file transfer servers:

  • Feed servers — These are servers that accept files dropped on them.

  • Medallia Media File transfer — MMFT is a scalable, S3-based storage solution hosted by Medallia.

The feed server can hold about 4,000 files. Files can be up to 2GB each. Files are immediately processed by the importer and removed from the feed servers after being processed, but they remain on MMFT for a period of time defined by the Lifetime of Feed Files option, which defaults to 30 days. In situations where you need to feed more files than the server can hold, consider bundling multiple files in one compressed file (either .gz, .bz2, .z, .zip, or .tar format). Experience Cloud detects those compressed files types and automatically extracts the bundled files and passes them to the importer for processing.

For general information about the feed servers, see Medallia feed servers and filestash. Medallia maintains several feed servers, each hosted in separate data centers. To identify the host URL of the feed server for your company's instance, first identify the data center that hosts the instance, then look up the host in Medallia Experience Cloud IP addresses and host names.

Important: Files sent to SFTP/FTP feed accounts — whether from the external system (company) or from an internal Medallia user — must always be encrypted using the Medallia public key due to the security concerns. See File encryption, below, for details.

FTP Feed topics:

Creating an SFTP Feed

The general steps for creating a feed are:

  1. Create a new feed.
    1. Set Processing mode to Upload with pretend processing.
    2. Provide a Login and Password to access the Medallia FTP server.
    3. Specify the File regex that matches the name of the file(s) to retrieve from the FTP host and to import.
    4. Choose the Importer (Auto Importer processor).
    5. Click Save to keep the Feed configuration.
  2. Use an FTP application to verify you can log in to the SFTP server with the Login and Password.
    1. Place a sample file on the FTP server. The name must match the File regex.
    2. Check the Imported Files reports to verify the file was retrieved by the feed in Pretend mode.
  3. Edit the configuration and change Processing mode to Active.
  4. Inform the company that they may begin feeding files to the SFTP server.

Properties

Name
Name of this feed as defined by the Login property.
Description
The description of this feed; should include the purpose and information about the expected source system and Auto Importer specification.
Processing mode

Processing state of this feed.

  • Active — Incoming files are processed and records are added to the Experience Cloud.
  • Inactive — Incoming files are not processed.
  • Upload with pretend processing — The feed is active, but records are not added to the database. Use this to test the feed.

After Active or Pretend jobs, the processing report shows the statistics of the event

Importer traffic

Reports statistics about recent active and pretend processing activities for this feed, including successful and failed feeds over time. All times are based on the current time when the page displays — for example, "Past Day" is 24 hours before the page displayed, not everything in the previous day. This can be different from other reports that show activity from the previous day, which would be all traffic that day.

This information appears only after the feed has processed files.

Lifetime of Feed Files
Count of days to retain imported files and processing reports. After this many calendar days, the files are purged from the data centers. Default is to use the Import Deletion Schedule setting on the Company settings screen. See the description of that field for considerations about choosing a value. To be compliant with the E.U. General Data Protection Regulation (GDPR), this value should be 30 days or less.

This setting does NOT affect files retained on company-hosted or Medallia-hosted FTP/SFTP hosts.

Important: To comply with GDPR policies, files are deleted after 30 days.

Account configuration

Login
Account used by external system to access the Medallia FTP/SFTP host, and also the Name of the feed specification.
Authentication type
Type of authentication used by the external system:

  • Password — System provides the value defined in the Password property.

    Important: Password authentication is deprecated and will be obsolete in a future release. Preferentially, use public key authentication for new connections with Medallia-owned servers or client-owned servers. For information about public key authentication, see Generating SSH Key pairs.

  • Public key — System provides the valued defined in the Authentication public key property.

    Warning: If you are trying to set up an SFTP pull or an SFTP feed with public key authentication and you get a "connection refused" error, check with your network teams. Medallia has no IP firewalls that would be blocking such traffic on our end. We typically see this when a network device on the client side is performing a transparent packet inspection or destination blocking. However, client IP addresses can be blocked if there are more than 5 connection attempts with the wrong authentication key or password.
Authentication public key
Public key to use when Authentication type is Public key.

The public key is a long string of characters provided by administrators of the external system. Enter the key in this property. When the external system makes the SFTP connection to the Medallia platform, it begins by requesting this public key. It then compares the key to its internal private key to authenticate the connection. If the key does not match, the systems will reject the connection.

Restriction: Experience Cloud requires a comment in the key, even though the standard says the Comment field is optional. Medallia recommends including your email address as the comment to add to the keys to identify who generated the key.
Tip: For SFTP feeds, upload the key directly into the SFTP feed configuration within Experience Cloud. Medallia Support only loads keys into filestash.* ("MMFT") directories, not SFTP Feeds. For information, see Import with SFTP/FTP feeds using SSH.

See SSH key pairs for a discussion of how to use and create authentication keys.

Note: Medallia recommends that SSH key pairs use the RSA/4096 encryption algorithm; see Generating SSH Key pairs for a discussion of how to generate key pairs for shorter bit lengths.
Password
The password associated with the Login account. You may enter a custom password or pick one of the suggested strings.
Tip: Make a note of the password before clicking Save. Experience Cloud never shows the password value again.
File regex

Regex that identifies files to process. Only files that match the regex are processed by this feed. To match all files, use this expression:

.*

This example 

phone_.*.txt.pgp

matches these files:

iphone_20090101.txt.pgp
iphone_20090201.txt.pgp

but not this file

property_20090101.txt.pgp

This example 

OmniExport-\d+-\d+\.csv

matches these filles:

OmniExport-200901-1201.csv
OmniExport-20090501-01.csv

Record processor

Choose only one processor: either Auto Importer or Lego.
Auto Importer processor
The Auto Importer specification that processes the incoming data.
Lego processor
Lego processor to process the retrieved file.
(Legacy) Convert format

(Deprecated) Convert format to process the retrieved file.

Warning: After setting up a new feed account with LEGO/ConvertFormat processor, test it using Kind UPLOAD_WITH_PRETEND_PROCESSING. Check the processing report carefully to make sure that the feed data is getting processed as expected. This is particularly important with stay files or survey import files (that is, anything that generates or creates new records in Experience Cloud): deleting survey records on a production server is extremely tedious and/or involves scarce engineering resources. But in general, this is a good practice for testing feed accounts that process all kind of data.

Processing options

Primary UnitGroup

(Auto Importer only) Sets the From address of the report e-mails. When this property is set, the Auto-Importer does not determine the survey permission and or do duplicate checking.

(Lego only) Only accept records with Unit IDs in the selected Unit groups. Records for Units not in this Unit group scope are rejected. Applies only to LegoProcessType of SURVEY_IMPORT and SURVEY_UPDATE. It does not work for other process types such as STAY_IMPORT, EMAIL_IMPORT, or OPTOUT_IMPORT.

Extract archive
Extracts records from an archive (such as a zip file) when the content is a compressed file. This option is on by default, and this option is only available when first creating the Feed. After saving the specification the first time, this option is no longer available. In practice, there is no reason to turn this off.

Email report

Summarized report e-mails
Only for Auto Importers. Email addresses (comma-separated) to receive the summarized processing report of successes and failures.
Lego e-mail template
Only for Lego. E-mail template to use for the processing report.

Post-processing options

Post-processing actions are optional activities to happen after the Auto Importer successfully updates the database. These do not happen when the importer fails to update the database, such as when a record is invalid or when a file is processed with Upload with pretend processing.

Trigger an export after file is processed
Turn on trigger export processing after successfully importing the data. After turning this on and saving the specification, the Trigger section will be available to define the trigger.

Trigger

Trigger options define when and what to trigger as part of the post-processing actions. This section appears after turning on Trigger an export after file is processed, and saving the configuration.

Export to trigger
Exporter to trigger.
Trigger pre-conditions

Files that must have been successfully imported and processed before the export will trigger.

  • The first line is a filename or filename regex identifying a file successfully processed by this feed in the recent past (length varies by system load).
  • (Optional) Each of the following lines begin with an integer indicating a number of hours, followed by a comma, and then a filename or filename regex.
    • Each specified file must have been successfully processed within the number of hours specified.
    • If any of the files was not processed in the specified time, the export is not triggered.
    • These optional files can be processed by any Auto Importer, for any account in the company.

In the following example, each of these conditions must met for the export to trigger:

  • The feed successfully processed a file matching the regex invitation_file.*.csv.
  • In the past 1 hour some importer successfully processed a file matching the regex org_update.*.csv.
  • In the past 3 hours some importer successfully processed a file matching the regex account_update.*.csv.
invitation_file.*.csv
1,org_update.*.csv
3,account_update.*.csv

Remember, the Feed Files reports show the processing status of imported files.

Last trigger attempt
Timestamp of the last time a trigger successfully processed a feed.
Trigger another Feed Pull
(available for Feed Pull importers only) Another Feed Pull to trigger when this one finishes. Use this option to ensure sequential processing of import files.
Preview triggers
Lists exports that have been triggered per the pre-conditions, but which have not yet run. The list appears at the bottom of the screen when

File encryption

Files sent to SFTP/FTP feed accounts — whether from the external system (company) or from an internal Medallia user — must always be encrypted using the Medallia public key due to the security concerns. Note there are different keys depending on the type of installation. See PGP keys for details about the keys.

Important: Sandboxes use the same PGP public key as the production instance. However, Sandboxes have an alternative PGP key that is not available to the production instance. Use that key when you want to ensure that data is decrypted on the sandbox only and not on the production instance.
Restriction: QA instances do not include decryption by default. To decrypt files on a QA instance, contact Medallia Support.

The Auto Importer automatically determines if the file is encrypted, and it automatically decrypts the file. To ensure the system can detect that the file is encrypted:

  • The filename should end with .pgp or .gpg.
  • The first line of an encrypted file is always: 
    -----BEGIN PGP MESSAGE-----

Troubleshooting tips

Corrupt files
All Feed pulls are done in binary mode; do not attempt to use ASCII mode, even for text records. Doing an ASCII transfer of a binary file corrupts the file. Further, success reports typically show twice as many records imported as included in the source file.
Cannot access remote system
Try using a FTP utility to manually connect to the external system. Verify the credentials and settings are the same for the feed pull.
SFTP issues
In the past, some SFTP vendors did not support the SFTP software Medallia Experience Cloud uses (e.g., Tumbleweed has had problems in the past). In those situations, the companies push a file to an intermediate SFTP server hosted by Medallia (Medallia Media File Transfer), and then the Feed Pull accesses the file(s) from that server using SFTP. Accounts using SFTP Pull must use STANDARD COMPLIANT OpenSSH2.