Skip to main content
RichRelevance

Batch Delta File Feed Upload

Overview

Using the FTP upload command, you can upload the delta file segments feed. This allows processing of the delta feed for the segment feed, enabling a subset of users and their segments to be updated in the UPS from the delta file. In this upload method, the batch delta file is processed and only users with modifications/updates in their profiles/segments are updated in the UPS instead of all the users from the file. This is in contrast to the full file upload in which all the users in the feed file are uploaded.

Note: Processing a delta feed is quite similar to a full feed file with a slight difference.

Scope

Presently delta files are processed for modified/updated users only. To remove users, you may still need to use full feed files as users cannot be removed using delta files.

Assumptions

The client is expected to send a full file to update all user info in one go and the incremental delta files on a periodic basis.

Processing Delta Feed File

What is a full feed file? The full feed file consists of all users' details and each user will have full segments and non-segments batch data, user attributes, user preferences and/or user linking details in a single file.

What is a delta feed file? The delta feed file consists of only updated users' details and each user will have full segments and non-segments batch data, user attributes, user preferences and/or user linking details in a single file.

What is a user linking file? The user linking file is a delimited text file and it has a special Unicode character as a delimiter.

Preparing Delta File for Upload

  • Syntax and payload structure in delta files is similar to full feed files.
  • Delta files should include only updated users.
  • When preparing batch segment delta files, only updated and/or new users' segment details are added.
  • When preparing batch non-segment files and user linking files, only updated users' and/or new users' details are added.

Other details with respect to the processing of these files are provided in the below table.

Delta Feed File vs. Full Feed File

 

Full Feed File

Delta Feed File

Data Processing
  1. After the full feed file is uploaded to the FTP server, all the users' details will be available in the database.
  2. UPS system checks for the new incoming file version with the file available in the database. Only the latest file is processed and persisted in the database.
  3. FTP server will store the incoming batch data into a Zstandard compressed file.
  1. The delta file is uploaded to the FTP server, only a set of users' details are updated in the database.
  2. UPS systems checks for the incoming file version with the available full file and delta file version. Only the latest files are processed and persisted in the database.
  3. FTP server stores the incoming batch data into a Zstandard compressed file.

Note: The full feed files are uploaded in the same way as mentioned in the Full Feed File column.

Data Restore The client sends a full file to Sync UPS data with client data. The client sends the full file whenever UPS data needs to be synced with client data or to restore the missing data.
UPS Reader
  • Serves the latest full file users details.
  • Stalled full file versioned user details are ignored.
  • Serves the latest full file and/or subsequent all delta files users details.
  • Stalled full file and/or delta file versioned user details are ignored.
Data Compaction

For the same API Key and attribute type, if multiple files are uploaded using the FTP command, the compactor removes the stalled versioned users' data from the database.

  1. For the same API Key and attribute type, if multiple delta files are uploaded using the FTP command, the compactor does not remove the stalled delta file versioned users' data from the database.
  2. Unless a full file is uploaded using the FTP command, all the existing full files and delta files users' details will be present in the database.
  3. After a full file is uploaded using the FTP command, the compactor removes all the stalled versioned full files and/or delta files users' data from the database.

Sample Data Files

Batch Segment File

rr-segments

{"userId":"00351934487736654412608034933597340001","value":{"Carta-11":"Carta-IDEAPIUEasyCasa","RFM-MEDIO":"RFM-MEDIO","CLTVtot-NULL":"CLTVtot-NULL","CLTVprog-NULL":"CLTVprog-NULL","Bricoleur-NULL":"Bricoleur-NULL"}}
{"userId":"00351934487736654412608034933597340002","value":{"Carta-12":"Carta-IDEAPIUEasyCasa","RFM-NULL":"RFM-NULL","CLTVtot-NULL":"CLTVtot-NULL","CLTVprog-NULL":"CLTVprog-NULL","Bricoleur-NULL":"Bricoleur-NULL"}}

 

Batch User-attributes File

rr-userattributes

{"userId""00351934487736654412608034933597340001","value": {"like" : ["prod102710181","prod102480201"],"disLike" : ["prod10150001","prod101520076"]}}
{"userId""00351934487736654412608034933597340002","value": {"like" : ["prod102710182","prod102480202"],"disLike" : ["prod10150002","prod101520077"]}}

Syntaxes and Commands to Process and Upload Batch Files

Process and Upload File Commands' Syntax

The following syntaxes are used to process the batch files. 

Uploading Full and Delta File by Client

Client uploads full and delta files to the FTP server, using the following FTP command:

    put <FileName>

Processing Batch Files

Full File

After uploading the full file, run the following command to process the batch segments full file:

    site upsbatch <FileName> -attribute <attributeType>

Delta File 

After uploading the delta file, run the following command to process the batch segments delta file:

    site upsbatch <FileName> -attribute <attributeType> -loadtype delta

Linkings Full File

After uploading the linking file, run the following command to process the User Linkings Full file:

    site upslink <FileName>

Linkings Delta File

After uploading the delta file, run the following command to process the User Linkings Delta file:

    site upslink <FileName> -loadtype delta

Upload Commands

Run the following commands to upload and process batch files with the required changes in file names:

  • To upload and process batch segments full file:

    put segment1.json

    site upsbatch segment1.json -attribute rr-segments

  • To upload and process batch segments delta file:

    put segment2.json

    site upsbatch segment2.json -attribute rr-segments -loadtype delta

  • To upload and process User Linkings Full file:

    put userlink1.txt

    site upslink userlink1.txt

  • To upload and process User Linkings Delta file:

    put userlink2.txt

    site upslink userlink2.txt -loadtype delta

FAQs

What happens when a customer uploads a delta feed file without sending the full feed file at the very first time?

BuildFTP is smart enough to process the incoming delta feed file and all users' data persist in the database. However, It is not recommended to send the delta feed file at first.

Note: It is recommended that you send the delta feed file on top of the full feed file.

What happens when a customer uploads a full feed file as a delta feed file?

All uploaded users' data persist in the database. However, it leads to data inconsistency (All the stalled versions of the users' data should be compacted if you place a full feed file. Because, you have placed it as a delta feed file, there will be no stalled versioned users' data and compaction).

To resolve the above data inconsistency, the customer should resend the actual full feed file.

What happens when a customer uploads a delta feed file as a full feed file?

All uploaded users' data persist in the database. However, it leads to another sort of data inconsistency (No users' data should to be compacted if you place a delta feed file. Because, you have placed it as a full feed file, the previous full and all delta versioned users' data becomes a stalled version and compacted).

To resolve the above data inconsistency, the customer should resend the actual full feed file and then all delta feed files in the same order again if any. Finally, the actual delta file should be uploaded.

Does the BuildFTP process support Bi-directional(Grouped) user linking delta feed file?

No. For the current release, BuildFTP does not support Bi-directional(Grouped) user linking Delta File.

How does UPS system data sync with customer data?

The customer should send a full feed file so that all user's data will be updated and sync with customer data.

  • Was this article helpful?