Skip to main content
RichRelevance

Batch Delta Feed File Upload

Overview

Using the FTP upload command, you can upload the delta file segments feed. This allows processing of the delta feed for the segment feed, enabling a subset of users and their segments to be updated in the UPS from the delta file. In this upload method, the batch delta file is processed and only users with modifications/updates in their profiles/segments are updated in the UPS instead of all the users from the file. This is in contrast to the full file upload in which all the users in the feed file are uploaded.

Note: Processing a delta feed file is quite similar to a full feed file with a slight difference.

Scope

Presently delta feed files are processed for modified/updated users only. To remove users, you may still need to use full feed files as users cannot be removed using delta feed files.

Assumptions

The client is expected to send a full feed file to update all users information in one go and the incremental delta feed files on a periodic basis.

Processing Delta Feed File

What is a full feed file? The full feed file consists of all users' details and each user will have full segments and non-segments batch data, user attributes, user preferences and/or user linking details in a single file.

What is a delta feed file? The delta feed file consists of only updated users' details and each user will have full segments and non-segments batch data, user attributes, user preferences and/or user linking details in a single file.

What is a user linking file? The user linking file is a delimited text file and it has a special Unicode character as a delimiter.

Preparing Delta Feed File for Upload

  • Syntax and payload structure in delta feed files is similar to full feed files.
  • Delta feed files should include only updated users.
  • When preparing batch segment delta files, only updated and/or new users' segment details are added.
  • When preparing batch non-segment files and user linking files, only updated users' and/or new users' details are added.

Other details with respect to the processing of these files are provided in the below table.

Delta Feed File vs. Full Feed File

 

Full Feed File

Delta Feed File

Data Processing
  1. After the full feed file is uploaded to the FTP server, all the users' details will be available in the database.
  2. UPS system checks for the new incoming file version with the file available in the database. Only the latest file is processed and persisted in the database.
  3. FTP server will store the incoming batch data into a Zstandard compressed file.
  1. After the delta feed file is uploaded to the FTP server, only a set of users' details are updated in the database.
  2. UPS system checks for the incoming file version with the available full feed file and delta feed file version. Only the latest files are processed and persisted in the database.
  3. FTP server stores the incoming batch data into a Zstandard compressed file.

Note: The delta feed files are uploaded in the same way as mentioned in the Full Feed File column.

Data Restore The client sends a full feed file to sync UPS data with client data. The client sends the full feed file whenever UPS data needs to be synced with client data or to restore the missing data.
UPS Reader
  • Serves the latest full feed file users details.
  • Stalled full feed file versioned user details are ignored.
  • Serves the latest full feed file and/or subsequent all delta feed files users details.
  • Stalled full feed file and/or delta feed file versioned user details are ignored.
Data Compaction

For the same API Key and attribute type, if multiple files are uploaded using the FTP command, the compactor removes the stalled versioned users' data from the database.

  1. For the same API Key and attribute type, if multiple delta feed files are uploaded using the FTP command, the compactor does not remove the stalled delta feed file versioned users' data from the database.
  2. Unless a full feed file is uploaded using the FTP command, all the existing full feed files and delta feed files users' details will be present in the database.
  3. After a full feed file is uploaded using the FTP command, the compactor removes all the stalled versioned full feed files and/or delta feed files users' data from the database.

Sample Data Files

Batch Segment File

rr-segments

{"userId":"00351934487736654412608034933597340001","value":{"Carta-11":"Carta-IDEAPIUEasyCasa","RFM-MEDIO":"RFM-MEDIO","CLTVtot-NULL":"CLTVtot-NULL","CLTVprog-NULL":"CLTVprog-NULL","Bricoleur-NULL":"Bricoleur-NULL"}}
{"userId":"00351934487736654412608034933597340002","value":{"Carta-12":"Carta-IDEAPIUEasyCasa","RFM-NULL":"RFM-NULL","CLTVtot-NULL":"CLTVtot-NULL","CLTVprog-NULL":"CLTVprog-NULL","Bricoleur-NULL":"Bricoleur-NULL"}}

 

Batch User-attributes File

rr-userattributes

{"userId""00351934487736654412608034933597340001","value": {"like" : ["prod102710181","prod102480201"],"disLike" : ["prod10150001","prod101520076"]}}
{"userId""00351934487736654412608034933597340002","value": {"like" : ["prod102710182","prod102480202"],"disLike" : ["prod10150002","prod101520077"]}}

Syntaxes and Commands to Process and Upload Batch Files

Process and Upload File Commands' Syntax

The following syntaxes are used to process the batch files. 

Uploading Full and Delta Feed File by Client

Client uploads full and delta feed files to the FTP server, using the following FTP command:

    put <FileName>

Processing Batch Files

Batch Segments Full Feed File

After uploading the full feed file, run the following command to process the batch segments full feed file:

    site upsbatch <FileName> -attribute <attributeType>

Batch Segments Delta Feed File 

After uploading the delta feed file, run the following command to process the batch segments delta feed file:

    site upsbatch <FileName> -attribute <attributeType> -loadtype delta

User Linkings Full Feed File

After uploading the full feed file, run the following command to process the user linkings full feed file:

    site upslink <FileName>

User Linkings Delta Feed File

After uploading the delta feed file, run the following command to process the user linkings delta feed file:

    site upslink <FileName> -loadtype delta

Upload Commands

Run the following commands to upload and process batch files with the required changes in file names:

  • To upload and process batch segments full feed file:

    put segment1.json

    site upsbatch segment1.json -attribute rr-segments

  • To upload and process batch segments delta feed file:

    put segment2.json

    site upsbatch segment2.json -attribute rr-segments -loadtype delta

  • To upload and process user linkings full feed file:

    put userlink1.txt

    site upslink userlink1.txt

  • To upload and process user linkings delta feed file:

    put userlink2.txt

    site upslink userlink2.txt -loadtype delta

FAQs

What happens when a customer uploads a delta feed file without sending the full feed file at the very first time?

BuildFTP is smart enough to process the incoming delta feed file and all users' data persist in the database. However, It is not recommended to send the delta feed file at first.

Note: It is recommended that you send the delta feed file on top of the full feed file.

What happens when a customer uploads a full feed file as a delta feed file?

All uploaded users' data persist in the database. However, it leads to data inconsistency (All the stalled versions of the users' data should be compacted if you place a full feed file. Because, you have placed it as a delta feed file, there will be no stalled versioned users' data and compaction).

To resolve the above data inconsistency, the customer should resend the actual full feed file.

What happens when a customer uploads a delta feed file as a full feed file?

All uploaded users' data persist in the database. However, it leads to another sort of data inconsistency (No users' data should to be compacted if you place a delta feed file. Because, you have placed it as a full feed file, the previous full and all delta versioned users' data becomes a stalled version and compacted).

To resolve the above data inconsistency, the customer should resend the actual full feed file and then all delta feed files in the same order again if any. Finally, the actual delta feed file should be uploaded.

Does the BuildFTP process support Bi-directional(Grouped) user linking delta feed file?

Yes. The BuildFTP process supports Bi-directional (Grouped) as well as Uni-directional (explicit) mapping with user linking delta feed file.

How does UPS system data sync with customer data?

The customer should send a full feed file so that all user's data will be updated and sync with customer data.

  • Was this article helpful?