Skip to main content
RichRelevance

Scoped Action

Introduction

Scoped Actions provides a means to sync up product information within a snapshot to a new ingestion of items, or sync up the legacy catalog (portal) with a specific snapshot.

 The objective is ensure data integrity and automatically update or replace items and delete "orphan" items that are not updated or replaced.

Deletions are done in batches of 1,000 while updates or replacements are in batches of 50.  That will be reflected in the status for each service that is involved like the streaming-engine, legacy catalog and streaming-find-indexer.  There is a new service, the streaming-scoped-action which will provide a summary of items updated, replaced and deleted.  

There are currently two types of Scoped Actions both of which are part of the snapshot service. A summary of both are below:

Scoped Action to Sync the Legacy Catalog with a Snapshot

The first scoped action is to ensure data integrity for the legacy catalog (portal).  Currently the legacy catalog is not snapshot aware, which means that products and categories from previous snapshots may still be in the legacy catalog and showing up in the catalog.  Items from either complete or active snapshots are accepted by the legacy catalog and because it is not snapshot aware the product information may not be in sync with the current snapshot.  There could be products from older snapshots that are no longer required.  It is possible to clean up the legacy catalog by either deleting the products through the streaming-ingest using "force=true" or one can sync up the legacy catalog to a specified snapshot through this scoped action.  When the scoped action is "opened", the scoped action service replaces all items from the snapshot into the legacy catalog.  The scoped action will close itself once it finishes replacing all items in the legacy catalog and will then delete any "untouched" items in the legacy catalog. 

Scoped Action to Sync a Subset of a Snapshot to an incoming Set of Products during Ingestion

The second type of scoped action happens within one snapshot.  It is when one wants to provide a new set of products for a subset of the current snapshot and have the scoped action determine which items are orphans and delete them from the snapshot as well as update products.  If a user only want to update or replace products, a scoped action is not required as the streaming catalog already compares incoming items to the current items in the snapshot and sends the deltas to engine.out.  

This type of scoped action is only available for a"subset" of the products identified by an attribute.  The attribute must be defined in the product property definition collection with the "scopedActionSettings" parameter. 

One use case is for a B2B who may have stores who are supplying them with a full catalog and they do not know which items are no longer in the current snapshot.  They could have an attribute for "store" and send in the the full catalog for that store.  The scoped action will "touch" each item it updates in the snapshot.  When the ingestion is finished, a "close" command is executed and the service then checks for any "untouched" items in the snapshot and deletes them. 

See more details on the steps for a scoped action within one snapshot.

Steps for a Scoped Action to sync the Legacy Catalog

The first step to sync the Legacy Catalog to a specific snapshot, is to open a scoped action referencing the snapshot and provide a "scopedActionDefinition" in the body of the call.  

A scopedActionDefinition provides instructions and is required for both product and category items.  Both products and categories and their attributes need to be updated in order to keep the legacy catalog and snapshot in sync.  

Once the scoped action is "open", it will then replace all the legacy catalog's products and categories from the snapshot.  It keeps track of items touched.  Once the replacement of all items is completed, the scope action automatically will close the scope action and delete any "untouched" items that are still in the legacy catalog. 

There will be status messages from the streaming-engine, the legacy catalog adaptor service as well as from the streaming-scoped-action-service. The the streaming-scoped-action service details when a scoped action is open up to when it is complete.  

For this scoped action there is only one "open" command to execute.  

Note:

  • Only one scoped action to sync a Legacy Catalog can run at one time. 
  • Deletions are done in batches of 1,000 while replacements are in batches of 50.  
  • Each scoped action has different requirements for the scopedActionDefinition

Scoped Action "open" call and scopedActionDefinition

 The scopedActionDefinition requires specific parameters for action by item type. Both "product" and "category" itemTypes are mandatory and "referenceConntent" is optional.  

onOpen - specify the type of scoped action which is "allLegacyCatalog".  The Legacy Catalog is the target and the action is "replaceAll". If the onOpen is not provided the items will be not be replaced but a comparison is still done and items with the same "id" will be touched.  It is recommended to replaceAll.

onClose - after the products and categories have been updated, the scoped action will automatically close and  delete "untouched" orphans that remain in the Legacy Catalog. 

Syntax for scopedActionDefinition is below.

{
  "<itemType>" : {
       "onOpen": {
            "onExit": "complete",
            "scope": {
                "type": "allLegacyCatalog"
            },
            "action": {
                "type": "replaceAll"
            }
        },
       "onClose" : {
           "scope" : {
              "type" : "allLegacyCatalog"
        },
           "action" : {
              "type" : "deleteUntouched"
        }
      }
    }
 }

 

The API Call to "open" a scoped action:
POST https://<host>/streaming-snapshot/v1/<apiKey>/action/<snapshotId>/<snapshotType>/open

Below is an example with the required body's scopedActionDefinition.  In this case the apiKey is 1234, snapshot ID is 8000, and it is a product snapshot. 

  POST https://<host>/streaming-snapshot/v1/1234/action/8000/product/open
  
  body
  
  {
    "product": {
        "onOpen": {
            "onExit": "complete",
            "scope": {
                "type": "allLegacyCatalog"
            },
            "action": {
                "type": "replaceAll"
            }
        },
        "onClose": {
            "scope": {
                "type": "allLegacyCatalog"
            },
            "action": {
                "type": "deleteUntouched"
            }
        }
    },
    "category": {
        "onOpen": {
            "onExit": "complete",
            "scope": {
                "type": "allLegacyCatalog"
            },
            "action": {
                "type": "replaceAll"
            }
        },
        "onClose": {
            "scope": {
                "type": "allLegacyCatalog"
            },
            "action": {
                "type": "deleteUntouched"
            }
        }
    }
}
  
  
  

Response:

The response includes a scopedActionId, trackingId, state of the scoped action and repeats the body of the call. 

In this case the siteId is 1234, snapshotID is 8000 and the state is "Opening_started".  The scopedActionId is: 1045

{
    "statusTracker": {
        "trackingId": "9df65ae2-bba4-11ea-9929-eb0f60272db0",
        "trackingInstant": "2020-07-01T14:10:28.742013Z"
    },
    "scopedAction": {
        "id": 1045,
        "siteId": 1234,
        "snapshotId": 8000,
        "trackingId": "9df65ae2-bba4-11ea-9929-eb0f60272db0",
        "state": "Opening_started",
        "definition": {
            "product": {
                "onOpen": {
                    "onExit": "complete",
                    "scope": {
                        "type": "allLegacyCatalog"
                    },
                    "action": {
                        "type": "replaceAll"
                    }
                },
                "onClose": {
                    "scope": {
                        "type": "allLegacyCatalog"
                    },
                    "action": {
                        "type": "deleteUntouched"
                    }
                }
            },
            "category": {
                "onOpen": {
                    "onExit": "complete",
                    "scope": {
                        "type": "allLegacyCatalog"
                    },
                    "action": {
                        "type": "replaceAll"
                    }
                },
                "onClose": {
                    "scope": {
                        "type": "allLegacyCatalog"
                    },
                    "action": {
                        "type": "deleteUntouched"
                    }
                }
            }
        },
        "lastModified": "2020-07-01T14:10:28.721655Z",
        "trackingInstant": "2020-07-01T14:10:28.742013Z"
    }
}

Check State of Scoped Action for Legacy Catalog Sync

Check the state of a scoped action through the streaming-snapshot service.  You can find out if it is completed. 

GET https://<host>/streaming-snapshot/v1/<apiKey>/action/<snapshotId>/<snapshotType>/<scopedActionId>

Example when scoped action is complete:

apiKey: 1234,  snapshotId: 8000 and scopedActionID: 1045

GET https://<host>/streaming-snapshot/v1/1234/action/8000/product/1045

Response

{
    "id": 1045,
    "siteId": 1234,
    "snapshotId": 8000,
    "trackingId": "9df65ae2-bba4-11ea-9929-eb0f60272db0",
    "state": "Complete",
    "definition": {
        "product": {
            "onOpen": {
                "onExit": "complete",
                "scope": {
                    "type": "allLegacyCatalog"
                },
                "action": {
                    "type": "replaceAll"
                }
            },
            "onClose": {
                "scope": {
                    "type": "allLegacyCatalog"
                },
                "action": {
                    "type": "deleteUntouched"
                }
            }
        },
        "category": {
            "onOpen": {
                "onExit": "complete",
                "scope": {
                    "type": "allLegacyCatalog"
                },
                "action": {
                    "type": "replaceAll"
                }
            },
            "onClose": {
                "scope": {
                    "type": "allLegacyCatalog"
                },
                "action": {
                    "type": "deleteUntouched"
                }
            }
        }
    },
    "lastModified": "2020-07-01T14:10:38.195171Z",
    "trackingInstant": "2020-07-01T14:10:28.742013Z"
}

Check Status of the Scoped Action's replacements and deletions

A scoped action will send all updates through the streaming engine which will go to the legacy catalog.  This includes both replacements and deletions.  Each service will record a status as it does when using the streaming-ingest service. 

The streaming-scoped-action service provides a summary of actions taken. 

To see the status messages, use the standard streaming-status endpoints.  Below are two examples but further information can be found in the streaming-status section of this manual. 

See all status messages, including legacy catalog, streaming engine and streaming-scoped-action service, based on the trackingId for the scoped action.  

GET https://<host>/streaming-status/v1/<apiKey>/trackingID?<trackingId>

Another option is to return the status messages for one specific service, like the legacy catalog (streaming-consumer-legacy-catalog).  However note that this is not based on trackingId but rather date and you will see all status messages from that day and not just the status messages for the scoped action.

You can restrict it to the streaming-scoped-action to see just the scoped action updates. 

GET https://<host>/streaming-snapshot/v1/<apiKey>/daySnapshotService/<snapshotId>?date=yyyy-mm-dd&service=<streaming service

Example of using the streaming-scoped-action service and date:

There is a status from when the scoped action was opened up to where it is done. 

apiKey: 1234,  snapshotId: 8000

GET https://<host>/streaming-status/v1/1234/daySnapshotService/8000?date=2020-07-01&service=streaming-scoped-action

Response:
[
    {
        "siteId": 1234,
        "snapshotId": 8000,
        "statusId": "a39b0e39-bba4-11ea-b9e0-73b56c2e6889",
        "trackingId": "9df65ae2-bba4-11ea-9929-eb0f60272db0",
        "source": "streaming-scoped-action",
        "statusType": "ScopedActionEvent",
        "message": "Done Closing Scoped Action 1045",
        "level": "SUMMARY",
        "statusData": {
            "stats": {
                "product": {
                    "untouched": 8000,
                    "touched": 12247,
                    "total": 20247
                },
                "category": {
                    "untouched": 7,
                    "touched": 542,
                    "total": 549
                }
            },
            "scopedActionId": 1045
        },
        "datacenter": "qa",
        "trackingInstant": "2020-07-01T14:10:28.742013Z",
        "msSinceRequest": 9468,
        "statusInstant": "2020-07-01T14:10:38.210002500Z"
    },
    {
        "siteId": 1234,
        "snapshotId": 8000,
        "statusId": "a0a35249-bba4-11ea-9aa2-f9a2aaf3c470",
        "trackingId": "9df65ae2-bba4-11ea-9929-eb0f60272db0",
        "source": "streaming-scoped-action",
        "statusType": "ScopedActionEvent",
        "message": "CLOSING ScopedAction 1045",
        "level": "SUMMARY",
        "statusData": {},
        "datacenter": "qa",
        "trackingInstant": "2020-07-01T14:10:28.742013Z",
        "msSinceRequest": 4489,
        "statusInstant": "2020-07-01T14:10:33.231008900Z"
    },
    {
        "siteId": 1234,
        "snapshotId": 8000,
        "statusId": "a0985572-bba4-11ea-b9e0-a55756661850",
        "trackingId": "9df65ae2-bba4-11ea-9929-eb0f60272db0",
        "source": "streaming-scoped-action",
        "statusType": "ScopedActionEvent",
        "message": "Sent Closing_started Scoped Action 1045",
        "level": "SUMMARY",
        "statusData": {},
        "datacenter": "qa",
        "trackingInstant": "2020-07-01T14:10:28.742013Z",
        "msSinceRequest": 4417,
        "statusInstant": "2020-07-01T14:10:33.159000200Z"
    },
    {
        "siteId": 1234,
        "snapshotId": 8000,
        "statusId": "a096a8bf-bba4-11ea-b9e0-6507fd0e4f45",
        "trackingId": "9df65ae2-bba4-11ea-9929-eb0f60272db0",
        "source": "streaming-scoped-action",
        "statusType": "ScopedActionEvent",
        "message": "Sent Open Scoped Action 1045",
        "level": "SUMMARY",
        "statusData": {
            "stats": {
                "product": {
                    "total": 12247,
                    "replaced": 12247
                },
                "category": {
                    "total": 542,
                    "replaced": 542
                }
            },
            "scopedActionId": 1045
        },
        "datacenter": "qa",
        "trackingInstant": "2020-07-01T14:10:28.742013Z",
        "msSinceRequest": 4406,
        "statusInstant": "2020-07-01T14:10:33.148025500Z"
    },
    {
        "siteId": 1234,
        "snapshotId": 8000,
        "statusId": "9e030513-bba4-11ea-9929-035efe94f10c",
        "trackingId": "9df65ae2-bba4-11ea-9929-eb0f60272db0",
        "source": "streaming-scoped-action",
        "statusType": "ScopedAction",
        "message": "ScopedAction 1045 OPENING_STARTED",
        "level": "SUMMARY",
        "statusData": {
            "scopedAction": "Opening_started"
        },
        "datacenter": "qa",
        "trackingInstant": "2020-07-01T14:10:28.742013Z",
        "msSinceRequest": 83,
        "statusInstant": "2020-07-01T14:10:28.825013100Z"
    }
]

Example of using the legacy catalog service and date:

apiKey: 1234,  snapshotId: 8000

GET https://<host>/streaming-status/v1/1234/daySnapshotService/8000?date=2020-07-01&service=streaming-consumer-legacy-catalog

Partial response to show replacements and deletions.  Deletions are sent in batchs of 1,000.
The order below is the default, the most recent to the older status messages. 

[
       
    {
        "siteId": 1234,
        "snapshotId": 8000,
        "statusId": "cae187dc-bba4-11ea-bdb4-fdc35df300ac",
        "trackingId": "9df65ae2-bba4-11ea-9929-eb0f60272db0",
        "source": "streaming-consumer-legacy-catalog",
        "statusType": "InternalEnvelope",
        "message": "Received delete action",
        "kafkaSource": {
            "topic": "streaming.engine.out",
            "partition": 6,
            "offset": 3050978
        },
        "level": "SUMMARY",
        "statusData": {
            "stats": {
                "product": {
                    "total": 1000,
                    "delete": 1000
                }
            }
        },
        "datacenter": "qa",
        "trackingInstant": "2020-07-01T14:10:28.742013Z",
        "msSinceRequest": 75361,
        "statusInstant": "2020-07-01T14:11:44.103010800Z"
    },{
        "siteId": 1234,
        "snapshotId": 8000,
        "statusId": "cadcccc8-bba4-11ea-bdb4-79da9b1a189e",
        "trackingId": "9df65ae2-bba4-11ea-9929-eb0f60272db0",
        "source": "streaming-consumer-legacy-catalog",
        "statusType": "InternalEnvelope",
        "message": "Received replace action",
        "kafkaSource": {
            "topic": "streaming.engine.out",
            "partition": 6,
            "offset": 3050964
        },
        "level": "SUMMARY",
        "statusData": {
            "stats": {
                "category": {
                    "total": 2,
                    "replace": 2
                }
            }
        },
        "datacenter": "qa",
        "trackingInstant": "2020-07-01T14:10:28.742013Z",
        "msSinceRequest": 75330,
        "statusInstant": "2020-07-01T14:11:44.072007200Z"
    },
    {
        
    {
        "siteId": 1234,
        "snapshotId": 8000,
        "statusId": "cadc7ea2-bba4-11ea-bdb4-ededc8264fc0",
        "trackingId": "9df65ae2-bba4-11ea-9929-eb0f60272db0",
        "source": "streaming-consumer-legacy-catalog",
        "statusType": "InternalEnvelope",
        "message": "Received replace action",
        "kafkaSource": {
            "topic": "streaming.engine.out",
            "partition": 6,
            "offset": 3050962
        },
        "level": "SUMMARY",
        "statusData": {
            "stats": {
                "product": {
                    "total": 22,
                    "replace": 22
                }
            }
        },
        "datacenter": "qa",
        "trackingInstant": "2020-07-01T14:10:28.742013Z",
        "msSinceRequest": 75328,
        "statusInstant": "2020-07-01T14:11:44.070006600Z"
    },
    {
        "siteId": 1234,
        "snapshotId": 8000,
        "statusId": "ca730e6f-bba4-11ea-bdb4-f7aba951abf4",
        "trackingId": "9df65ae2-bba4-11ea-9929-eb0f60272db0",
        "source": "streaming-consumer-legacy-catalog",
        "statusType": "InternalEnvelope",
        "message": "Received replace action",
        "kafkaSource": {
            "topic": "streaming.engine.out",
            "partition": 6,
            "offset": 3050961
        },
        "level": "SUMMARY",
        "statusData": {
            "stats": {
                "product": {
                    "total": 29,
                    "replace": 29
                }
            }
        },
   ......

Steps for a Scoped Action within One Snapshot

This scoped action has a specific use case.  It is used when one wants to provide a new set of products for a subset of the current snapshot and have the scoped action determine which items are orphans and delete them from the snapshot as well as update the products and categories.

One use case is for a B2B who may have stores who are supplying them with a full catalog and they do not know which items are no longer in the current snapshot.  They could have an attribute for "store" and send in the the full catalog for that store.  The scoped action will "touch" each item it updates in the snapshot.  When the ingestion is finished, a "close" command is executed and the service then checks for any "untouched" items in the snapshot and deletes them. 

The snapshot can be in “creating”, “complete” or “active” states.  If a snapshot is archived and there is an “open” scopedAction, it will be canceled.  Note that a snapshot in "creating" state does not update Find or the Legacy Catalog.

Multiple scoped actions on one snapshot are supported but they must all reference the same attribute that defines the subset of items. 
 

Property Definition Collection: scopedActionSettings

The first step is to define the attribute that defines a subset of items within a snapshot in the product property definition collection.  Only properties that are String, Boolean, Integer, and Long are supported in this type of scoped action.

In the "scopedActionSettings" set the "propertyMatch" parameter to true. The property definition needs to be published. This needs to be done before creating a snapshot and ingesting items.  For more information on how to create a property definition collection please see the Streaming Property Service manual.

Below is an example of a property called "store" in the product property definition collection.

},
        "store": {
            "dataType": "string",
            "origin": "custom",
            "propertyValueStyle": "baseOnly",
            "required": false,
            "lengthLimit": 1024,
            "scopedActionSettings": {
                "propertyMatch": true
            }
        },

Open a scopedAction for a snapshot

It is assumed that the property definition collection was published and a snapshot created.  It is also assumed that this snapshot has had items ingested into it.  The customer may now have a subset of data to be ingested into the snapshot.  The streaming-ingest by itself will update and handle new items but it doesn't figure out which items in the snapshot need to deleted as they are not in the incoming new ingestion of items.  what  but wants the scoped action to update as well as delete 

The next step is to open an scoped action for a specific snapshot with a scopedActionDefinition in the body of the call. The response will include the scopedActionId

Below is the API endpoint syntax:

POST https://<host>/streaming-snapshot/v1/<apiKey>/<snapshotId>/<snapshotType>/open

The scopedActionDefinition requires specific parameters by item type.  It is required to provide a propertyName and it's value to define the subset of items. The deletions will be restricted within that subset of items.  Updates stream through the streaming-ingest as usual but the scoped action service does touch each item so it can later determine which ones are untouched and are deleted.

onClose -  Type is "propertyMarch" and the action is "deleteUntouched"

Syntax for scopedActionDefinition is below.

{
  "<itemType>" : {
   "onClose" : {
     "scope" : {
        "type" : "propertyMatch",
        "propertyName" : "<propertyName>",
        "propertyValue" : "propertyValue"
     },
     "action" : {
       "type" : "deleteUntouched"
     }
    }
  }
 }

Below is an example open scoped action with the body and response.

apiKey: 1234,  snapshotId: 8000,  propertyName "store", and propertyValue  "one"

POST https://<host>/streaming-snapshot/v1/1234/8000/product/open

Body

{
  "product" : {
   "onClose" : {
     "scope" : {
        "type" : "propertyMatch",
        "propertyName" : "store",
        "propertyValue" : "one"
     },
     "action" : {
       "type" : "deleteUntouched"
     }
    }
  }
 }

 

Response


{
    "statusTracker": {
        "trackingId": "b3a6f40a-5fd4-11ea-833b-7b4db4ad81b4",
        "trackingInstant": "2020-03-06T18:02:53.999002600Z"
    },
    "scopedAction": {
        "id": 180,
        "siteId": 1234,
        "snapshotId": 8000,
        "trackingId": "b3a6f40a-5fd4-11ea-833b-7b4db4ad81b4",
        "state": "Open",
        "definition": {
            "product": {
                "onClose": {
                    "scope": {
                        "type": "propertyMatch",
                        "propertyName": "store",
                        "propertyValue": "one"
                    },
                    "action": {
                        "type": "deleteUntouched"
                    }
                }
            }
        },
        "lastModified": "2020-03-06T18:02:53.997718Z",
        "trackingInstant": "2020-03-06T18:02:53.999002600Z"
    }

Ingest items into snapshot with open Scoped Action

Now that the scoped action is open for a snapshot, it is time to ingest the items that represent the catalog for a subset of the snapshot.  In our example it could be one store who has updated their catalog and may have removed items and don't want to have to figure out what needs to be deleted.  The new ingested catalog is the source of truth. 

The streaming-ingest service is used to ingest items with a reference to the scopedActionId.  It is important that the scopeActionId is included in the streaming-ingest call, otherwise the scoped action won't know to keep track of what is "touched" and could delete everything on the close.

The API endpoint syntax is as follows: 

PUT https://<host>/streaming-ingest/v1/<apiKey>/<itemType>?snapshotId=<snapshotId>&scopedActionId=<scopedActionId>

"Body" will be the regular JSONL for products or categories.  More information on the streaming-ingest can be found in the streaming-ingest-service manual.

Can do as many streaming-ingest calls with the scopeActionId as required to ingest all items.  All items will updated as usual through the streaming-engine, legacy catalog and streaming-find-indexer.  Once the ingestion is finished, the the scoped action can be closed. 

Close the Scoped Action

 

POST https://<host>/streaming-snapshot/v1/<apiKey>/<snapshotId>/<snapshotType>/close

 

 

Check status for “done”.

When a scoped update is “closed”, the “streaming-scoped-action” service provides the status with the stats.  The stats includes how many items were “touched” and “untouched”.  This service then sends the deletes to the streaming-engine which sends them to other services such as streaming-view, Find, Legacy Catalog to be processed. Each service produces their own status. 

Here is an example of a status for a completed closed streaming-scoped-action.

[
    {
        "siteId": 123,
        "snapshotId": 3393,
        "statusId": "01f49eb7-5fd6-11ea-8023-ab1a944371c2",
        "trackingId": "01d30c8c-5fd6-11ea-833b-a134d8882a64",
        "source": "streaming-scoped-action",
        "statusType": "ScopedActionEvent",
        "message": "Done Closing Scoped Action 180",
        "level": "SUMMARY",
        "statusData": {
            "stats": {
                "ALL": {
                    "untouched": 1,
                    "total": 2,
                    "touched": 1
                }
            }
        },

Check streaming-view and other services.

Note: All “untouched Items” will be deleted.

Here is an example of a status that is returned when checking on the status of a closed scoped action will include other services.

{
        "siteId": 123,
        "snapshotId": 3393,
        "statusId": "01f317c7-5fd6-11ea-a01e-d3ca3f50419d",
        "trackingId": "01d30c8c-5fd6-11ea-833b-a134d8882a64",
        "source": "streaming-consumer-view",
        "statusType": "InternalEnvelope",
        "message": "Finished InternalEnvelope",
        "level": "SUMMARY",
        "statusData": {
            "stats": {
                "product": {
                    "total": 1,
                    "delete": 1
                }
            }

Scoped Action Endpoints

Base URL: https://<host>/streaming-snapshot/v1/

Full URL:  https://<host>/streaming-snapshot/v1/<apiKey>/<action>/<snapshotId>/<snapshotType>

Note: Both streaming-snapshot and streaming-ingest services are leveraged for a scopedAction

Action REST API for scoped action within one snapshot

Open a new scopedAction

Same call for both scoped actions but they have different bodies

 

The response will include a scopedActionId.

 

POST https://<host>/streaming-snapshot/v1/<apiKey>/action/<snapshotId>/<snapshotType>/open

Body: (scopedActionDefinition for scoped action within a snapshot)
{
  "<itemType>" : {
   "onClose" : {
     "scope" : {
        "type" : "propertyMatch",
        "propertyName" : "<propertyName>",
        "propertyValue" : "<propertyValue>"
     },
     "action" : {
       "type" : "deleteUntouched"
     }
    }
  }
 }

The response will include a scopedActionId.

Response Codes:
400 - If the scopedActionDefinition is malformed.
400 - If any scope propertyName is missing from the itemType's property definitions.
400 - If the propertyValue cannot be converted into the property's data type.
404 - If the parent snapshot is missing.
409 - Conflict if the snapshot ID contains an open scoped action.
200 - OK

Ingest items

Only for Scoped Action for within a snapshot.

Note: If the scopedActionId is not open then the items will be processed as usual and not as part of the scopedAction 

PUT https://<host>/streaming-ingest/v1/<apiKey>/<itemType>?snapshotId=<snapshotId>&scopedActionId=<scopedActionId>

Body - will have new and/or updated products.

Close a scopedAction

POST https://<host>/streaming-snapshot/v1/<apiKey>/action/<snapshotId>/<snapshotType>/close/<scopedActionId>

Response Codes:
404 - If scopedAction does not exist.
409 - Conflict if the scoped action is not open.
200 - OK.

Cancel a scopedAction.

Can only cancel an “open” scopedAction

POST https://<host>/streaming-snapshot/v1/<apiKey>/action/<snapshotId>/<snapshotType>/cancel/<scopedActionId>

Response Codes:
404 - If there is no scoped action.
409 - Conflict if the scoped action is not open.
200 - OK

Delete a scopedActon
Can only delete a “cancelled” or “complete”
scopedAction.  Cannot delete a scopedAction that is “open” or in the “closing” states. 

POST https://<host>/streaming-snapshot/v1/<apiKey>/action/<snapshotId>/<snapshotType>/delete/scopedActionId>

List scoped actions for a snapshot

GET https://<host>/streaming-snapshot/v1/<apiKey>/action/<snapshotId>/<snapshotType>

Get details for a specific scopedAction GET https://<host>/streaming-snapshot/v1/<apiKey>/action/<snapshotId>/<snapshotType>/scopedActionId

Parameters

 

Name Description Additional Information & Options

<apiKey>

Unique identifier for customer’s environment.

For example, if a customer has multiple environments in production or staging, each environment would have a unique apiKey. There can be many apiKeys associated to a client_ID.

Provided by RR to customer.

<snapshotId>

Unique identifier for a snapshot.

snapshotId is provided in the response to a POST, when creating the snapshot.

<snapshotType>

There are currently two types of snapshots:

product: Requires product and category property definitions.  Product items can have region and language over-rides.  

place: Required for Regions. All updates to regions are through the Place snapshot type.  The Place snapshot should be "active" and requires the Region property definition.

product

place

Scoped Action Commands

The following actions are used for creating a scoped action, closing, cancelling, and deleting a scoped action. 

open, close, cancel, delete

scopedActionDefinition

The definition provided in the body when opening a scoped action.  Require to define each item type. 

Example for Scoped Action within a Snapshot

{
  "<itemType>" : {
   "onClose" : {
     "scope" : {
        "type" : "propertyMatch",
        "propertyName" : "<propertyName>",
        "propertyValue" : "<propertyValue>"
     },
     "action" : {
       "type" : "deleteUntouched"
     }
    }
  }
 }

Example with multiple item types.

 
     
     
     

Examples

Get existing scopedAction for a specific snapshot

GET https://<host>/streaming-snapsho...n/3393/product

Response:

[
    {
        "id": 152,
        "siteId": 123,
        "snapshotId": 3393,
        "trackingId": "f0af2321-5749-11ea-a353-b1118ac84a32",
        "state": "Complete",
        "definition": {
            "product": {
                "onClose": {
                    "scope": {
                        "type": "propertyMatch",
                        "propertyName": "store",
                        "propertyValue": "one"
                    },
                    "action": {
                        "type": "deleteUntouched"
                    }
                }
            }
        },
        "lastModified": "2020-02-24T21:16:16.434831Z",
        "trackingInstant": "2020-02-24T21:09:27.046019300Z"
    },
    {
        "id": 153,
        "siteId": 123,
        "snapshotId": 3393,
        "trackingId": "49f15c25-5761-11ea-a353-33d040339ebe",
        "state": "Complete",
        "definition": {
            "product": {
                "onClose": {
                    "scope": {
                        "type": "propertyMatch",
                        "propertyName": "store",
                        "propertyValue": "one"
                    },
                    "action": {
                        "type": "deleteUntouched"
                    }
                }
            }
        },
        "lastModified": "2020-02-25T00:01:55.755933Z",
        "trackingInstant": "2020-02-24T23:56:35.222019700Z"
    }

Open, Ingest and then close scopedAction

POST https://<host>/streaming-snapshot/v1/<apiKey>/action/3393/product/open

Response:  The snapshotId is 158

{
    "statusTracker": {
        "trackingId": "7b5fbaed-58ce-11ea-8f38-c19a9fad48ee",
        "trackingInstant": "2020-02-26T19:30:44.460004500Z"
    },
    "scopedAction": {
        "id": 158,
        "siteId": 123,
        "snapshotId": 3393,
        "trackingId": "7b5fbaed-58ce-11ea-8f38-c19a9fad48ee",
        "state": "Open",
        "definition": {
            "product": {
                "onClose": {
                    "scope": {
                        "type": "propertyMatch",
                        "propertyName": "store",
                        "propertyValue": "one"
                    },
                    "action": {
                        "type": "deleteUntouched"
                    }
                }
            }
        },
        "lastModified": "2020-02-26T19:30:44.460768Z",
        "trackingInstant": "2020-02-26T19:30:44.460004500Z"
    }
}

Ingest a new product for store “one”

Current view store -  2 products, one (test_ms_3) for store “two” and one (test_ms_4) for store “one” 

GET http://10.3.252.38/streaming-view/v1/<apiLey>/3393/product

{
    "items": [
        {
            "itemType": "product",
            "internalId": "101729812602",
            "externalId": "test_ms_3",
            "trackingId": "4bba6584-5748-11ea-817a-971ac1658db4",
            "item": {
                "properties": {
                    "num_reviews": "10",
                    "image_url": "https://image.migros.ch/fm-xl/452fccb597ac1b6965dbffc5aa1ac952133232e8/maurten-drink-mix-320-getraenkepulver-mit-natriumchlorid_1.jpg(16 kB)",
                    "name": "Drink Mix test_1",
                    "product_recommendable": true,
                    "product_description": "added product 463027402900_test_1",
                    "description": "added product 463027402900_test_1",
                    "in_stock": true,
                    "price": "1001",
                    "product_external_id": "test_0923_1020_1",
                    "store": "two",
                    "product_image_id": "https://image.migros.ch/fm-xl/452fccb597ac1b6965dbffc5aa1ac952133232e8/maurten-drink-mix-320-getraenkepulver-mit-natriumchlorid_1.jpg(16 kB)",
                    "brand": "BrandOneStoreTwo",
                    "product_num_reviews": 10,
                    "product_link_id": "/de/p/463027402900/maurten-drink-mix-320?selectVariant_test_1",
                    "rating": "1",
                    "product_name": "Drink Mix test_1",
                    "product_rating": 1.0,
                    "link_url": "/de/p/463027402900/maurten-drink-mix-320?selectVariant_test_1",
                    "recommendable": "true",
                    "product_pricecents": 100100,
                    "product_brand": "BrandOneStoreTwo"
                }
            }
        },
        {
            "itemType": "product",
            "internalId": "101730595788",
            "externalId": "test_ms_4",
            "trackingId": "4fb064d0-581c-11ea-817a-5569c49e1494",
            "item": {
                "properties": {
                    "num_reviews": "10",
                    "image_url": "https://image.migros.ch/fm-xl/452fccb597ac1b6965dbffc5aa1ac952133232e8/maurten-drink-mix-320-getraenkepulver-mit-natriumchlorid_1.jpg(16 kB)",
                    "name": "Drink Mix test_1",
                    "product_recommendable": true,
                    "product_description": "added product 463027402900_test_1",
                    "description": "added product 463027402900_test_1",
                    "in_stock": true,
                    "price": "100",
                    "product_external_id": "test_0923_1020_1",
                    "store": "one",
                    "product_image_id": "https://image.migros.ch/fm-xl/452fccb597ac1b6965dbffc5aa1ac952133232e8/maurten-drink-mix-320-getraenkepulver-mit-natriumchlorid_1.jpg(16 kB)",
                    "brand": "ScopedAction",
                    "product_num_reviews": 10,
                    "product_link_id": "/de/p/463027402900/maurten-drink-mix-320?selectVariant_test_1",
                    "rating": "1",
                    "product_name": "Drink Mix test_1",
                    "product_rating": 1.0,
                    "link_url": "/de/p/463027402900/maurten-drink-mix-320?selectVariant_test_1",
                    "recommendable": "true",
                    "product_pricecents": 10000,
                    "product_brand": "ScopedAction"
                }
            }
        }
    ],
    "numFound": 2,
    "nextCursorMark": "AoEucDoxMDE3MzA1OTU3ODg=",
    "errors": []

Ingest a new product #8 for store one

Make sure the scopedActionId is included or else the item won’t be counted as “touched” for the scopedAction.
  
snapshotId: 3393
scopedActionId: 158

PUT https://<host>/streaming-ingest/v1/123/product?snapshotId=3393&scopedActionId=158

Body:
{"test_ms_8":{"properties":{"brand":"ScopedBrand","recommendable":true,"price":10,"store":"one","rating":1,
"image_url":"https://image.migros.ch/fm-xl/452fccb597ac1b6965dbffc5aa1ac952133232e8/maurten-drink-mix-320-getraenkepulver-mit-natriumchlorid_1.jpg(16 kB)",
"product_external_id":"test_0923_1020_1","name":"Drink Mix test_1","num_reviews":10,"in_stock":true,
"link_url":"/de/p/463027402900/maurten-drink-mix-320?selectVariant_test_1",
"description":"added product 463027402900_test_1"}}}
 
 Response:
{
    "statusTracker": {
        "trackingId": "2b7b2087-58d1-11ea-8a19-dd50b48edb01",
        "trackingInstant": "2020-02-26T19:49:58.912013500Z"
    }
}

Check Status with trackingId

GET http://10.3.252.38/streaming-status/...9-dd50b48edb01

[
    {
        "siteId": 123,
        "snapshotId": 3393,
        "statusId": "2c4c5380-58d1-11ea-b89c-7fb08cb0548c",
        "trackingId": "2b7b2087-58d1-11ea-8a19-dd50b48edb01",
        "source": "streaming-engine",
        "statusType": "ExternalEnvelope",
        "message": "ItemEvent",
        "level": "SUMMARY",
        "statusData": {
            "stats": {
                "product": {
                    "total": 1,
                    "update": 1
                }
            }
        },
        "datacenter": "qa",
        "statusInstant": "2020-02-26T19:50:00.283020800Z",
        "trackingInstant": "2020-02-26T19:49:58.912013500Z",
        "msSinceRequest": 1371
    },
    {
        "siteId": 123,
        "snapshotId": 3393,
        "statusId": "2ba082e5-58d1-11ea-9778-ff1c509552e2",
        "trackingId": "2b7b2087-58d1-11ea-8a19-dd50b48edb01",
        "source": "find-index-deployer",
        "statusType": "StreamingEvent",
        "message": "Streaming item not processed correctly",
        "level": "WARNING",
        "statusData": {
            "stats": {
                "product": {
                    "total": 1,
                    "product ignored": 1
                }
            }
        },
        "datacenter": "qa",
        "statusInstant": "2020-02-26T19:49:59.157014900Z",
        "trackingInstant": "2020-02-26T19:49:58.912013500Z",
        "msSinceRequest": 245
    },
    {
        "siteId": 123,
        "snapshotId": 3393,
        "statusId": "2b9c8ba9-58d1-11ea-82ca-b756c0072cd9",
        "trackingId": "2b7b2087-58d1-11ea-8a19-dd50b48edb01",
        "source": "streaming-consumer-view",
        "statusType": "InternalEnvelope",
        "message": "Finished InternalEnvelope",
        "level": "SUMMARY",
        "statusData": {
            "stats": {
                "product": {
                    "total": 1,
                    "update": 1
                }
            }
        },
        "datacenter": "qa",
        "statusInstant": "2020-02-26T19:49:59.131024900Z",
        "trackingInstant": "2020-02-26T19:49:58.912013500Z",
        "msSinceRequest": 219
    },
    {
        "siteId": 123,
        "snapshotId": 3393,
        "statusId": "2b93d918-58d1-11ea-82ca-4d753fff94b1",
        "trackingId": "2b7b2087-58d1-11ea-8a19-dd50b48edb01",
        "source": "streaming-consumer-view",
        "statusType": "InternalEnvelope",
        "message": "Received InternalEnvelope",
        "level": "SUMMARY",
        "statusData": null,
        "datacenter": "qa",
        "statusInstant": "2020-02-26T19:49:59.074024800Z",
        "trackingInstant": "2020-02-26T19:49:58.912013500Z",
        "msSinceRequest": 162
    },
    {
        "siteId": 123,
        "snapshotId": 3393,
        "statusId": "2b7c590a-58d1-11ea-8a19-510187ebb49e",
        "trackingId": "2b7b2087-58d1-11ea-8a19-dd50b48edb01",
        "source": "streaming-ingest",
        "statusType": "item",
        "message": "Finished REPLACE action",
        "level": "SUMMARY",
        "statusData": {
            "stats": {
                "product": {
                    "total": 1,
                    "replace": 1
                }
            }
        },
        "datacenter": "qa",
        "statusInstant": "2020-02-26T19:49:58.920013800Z",
        "trackingInstant": "2020-02-26T19:49:58.912013500Z",
        "msSinceRequest": 8
    },
    {
        "siteId": 123,
        "snapshotId": 3393,
        "statusId": "2b7b95b9-58d1-11ea-8a19-f717d0f8a868",
        "trackingId": "2b7b2087-58d1-11ea-8a19-dd50b48edb01",
        "source": "streaming-ingest",
        "statusType": "item",
        "message": "Batch sending 1 item(s)",
        "level": "SUMMARY",
        "statusData": {
            "stats": {
                "product": {
                    "total": 1,
                    "replace": 1
                }
            }
        },
        "datacenter": "qa",
        "statusInstant": "2020-02-26T19:49:58.915013700Z",
        "trackingInstant": "2020-02-26T19:49:58.912013500Z",
        "msSinceRequest": 3
    },
    {
        "siteId": 123,
        "snapshotId": 3393,
        "statusId": "2b7b2088-58d1-11ea-8a19-b3147c9a56a9",
        "trackingId": "2b7b2087-58d1-11ea-8a19-dd50b48edb01",
        "source": "streaming-ingest",
        "statusType": "item",
        "message": "Received REPLACE action",
        "level": "SUMMARY",
        "statusData": null,
        "datacenter": "qa",
        "statusInstant": "2020-02-26T19:49:58.912013600Z",
        "trackingInstant": "2020-02-26T19:49:58.912013500Z",
        "msSinceRequest": 0
    }
]

Check View Store, New Product is Available

GET http://10.3.252.38/streaming-view/v1/123/3393/product

{
    "items": [
        {
            "itemType": "product",
            "internalId": "101729812602",
            "externalId": "test_ms_3",
            "trackingId": "4bba6584-5748-11ea-817a-971ac1658db4",
            "item": {
                "properties": {
                    "num_reviews": "10",
                    "image_url": "https://image.migros.ch/fm-xl/452fccb597ac1b6965dbffc5aa1ac952133232e8/maurten-drink-mix-320-getraenkepulver-mit-natriumchlorid_1.jpg(16 kB)",
                    "name": "Drink Mix test_1",
                    "product_recommendable": true,
                    "product_description": "added product 463027402900_test_1",
                    "description": "added product 463027402900_test_1",
                    "in_stock": true,
                    "price": "1001",
                    "product_external_id": "test_0923_1020_1",
                    "store": "two",
                    "product_image_id": "https://image.migros.ch/fm-xl/452fccb597ac1b6965dbffc5aa1ac952133232e8/maurten-drink-mix-320-getraenkepulver-mit-natriumchlorid_1.jpg(16 kB)",
                    "brand": "BrandOneStoreTwo",
                    "product_num_reviews": 10,
                    "product_link_id": "/de/p/463027402900/maurten-drink-mix-320?selectVariant_test_1",
                    "rating": "1",
                    "product_name": "Drink Mix test_1",
                    "product_rating": 1.0,
                    "link_url": "/de/p/463027402900/maurten-drink-mix-320?selectVariant_test_1",
                    "recommendable": "true",
                    "product_pricecents": 100100,
                    "product_brand": "BrandOneStoreTwo"
                }
            }
        },
        {
            "itemType": "product",
            "internalId": "101730595788",
            "externalId": "test_ms_4",
            "trackingId": "4fb064d0-581c-11ea-817a-5569c49e1494",
            "item": {
                "properties": {
                    "num_reviews": "10",
                    "image_url": "https://image.migros.ch/fm-xl/452fccb597ac1b6965dbffc5aa1ac952133232e8/maurten-drink-mix-320-getraenkepulver-mit-natriumchlorid_1.jpg(16 kB)",
                    "name": "Drink Mix test_1",
                    "product_recommendable": true,
                    "product_description": "added product 463027402900_test_1",
                    "description": "added product 463027402900_test_1",
                    "in_stock": true,
                    "price": "100",
                    "product_external_id": "test_0923_1020_1",
                    "store": "one",
                    "product_image_id": "https://image.migros.ch/fm-xl/452fccb597ac1b6965dbffc5aa1ac952133232e8/maurten-drink-mix-320-getraenkepulver-mit-natriumchlorid_1.jpg(16 kB)",
                    "brand": "ScopedAction",
                    "product_num_reviews": 10,
                    "product_link_id": "/de/p/463027402900/maurten-drink-mix-320?selectVariant_test_1",
                    "rating": "1",
                    "product_name": "Drink Mix test_1",
                    "product_rating": 1.0,
                    "link_url": "/de/p/463027402900/maurten-drink-mix-320?selectVariant_test_1",
                    "recommendable": "true",
                    "product_pricecents": 10000,
                    "product_brand": "ScopedAction"
                }
            }
        },
        {
            "itemType": "product",
            "internalId": "101732217999",
            "externalId": "test_ms_8",
            "trackingId": "2b7b2087-58d1-11ea-8a19-dd50b48edb01",
            "item": {
                "properties": {
                    "num_reviews": "10",
                    "image_url": "https://image.migros.ch/fm-xl/452fccb597ac1b6965dbffc5aa1ac952133232e8/maurten-drink-mix-320-getraenkepulver-mit-natriumchlorid_1.jpg(16 kB)",
                    "name": "Drink Mix test_1",
                    "product_recommendable": true,
                    "product_description": "added product 463027402900_test_1",
                    "description": "added product 463027402900_test_1",
                    "in_stock": true,
                    "price": "10",
                    "product_external_id": "test_0923_1020_1",
                    "store": "one",
                    "product_image_id": "https://image.migros.ch/fm-xl/452fccb597ac1b6965dbffc5aa1ac952133232e8/maurten-drink-mix-320-getraenkepulver-mit-natriumchlorid_1.jpg(16 kB)",
                    "brand": "ScopedBrand",
                    "product_num_reviews": 10,
                    "product_link_id": "/de/p/463027402900/maurten-drink-mix-320?selectVariant_test_1",
                    "rating": "1",
                    "product_name": "Drink Mix test_1",
                    "product_rating": 1.0,
                    "link_url": "/de/p/463027402900/maurten-drink-mix-320?selectVariant_test_1",
                    "recommendable": "true",
                    "product_pricecents": 1000,
                    "product_brand": "ScopedBrand"
                }
            }
        }
    ],
    "numFound": 3,
    "nextCursorMark": "AoEucDoxMDE3MzIyMTc5OTk=",
    "errors": []

Close scopedAction

apiKey: 123, snapshotId: 3393,  scopedActionId: 158

POST https://<host>/streaming-snapshot/v1/123/action/3393/product/close/158

Response:

{
    "statusTracker": {
        "trackingId": "144129d6-58d3-11ea-8f38-f5b421a98d65",
        "trackingInstant": "2020-02-26T20:03:38.938005400Z"
    },
    "scopedAction": {
        "id": 158,
        "siteId": 123,
        "snapshotId": 3393,
        "trackingId": "7b5fbaed-58ce-11ea-8f38-c19a9fad48ee",
        "state": "Closing_started",
        "definition": {
            "product": {
                "onClose": {
                    "scope": {
                        "type": "propertyMatch",
                        "propertyName": "store",
                        "propertyValue": "one"
                    },
                    "action": {
                        "type": "deleteUntouched"
                    }
                }
            }
        },
        "lastModified": "2020-02-26T20:03:38.938939Z",
        "trackingInstant": "2020-02-26T19:30:44.460004500Z"
    }
}

Check Status, Use the trackingId Found Under “statusTracker”

When the scopedAction is complete will see the following status message:

[
    {
        "siteId": 123,
        "snapshotId": 3393,
        "statusId": "0905ada9-5762-11ea-9927-f1c2f7ffeaed",
        "trackingId": "49f15c25-5761-11ea-a353-33d040339ebe",
        "source": "streaming-scoped-action",
        "statusType": "ScopedAction",
        "message": "COMPLETE ScopedAction 158",
        "level": "SUMMARY",
        "statusData": null,
        "datacenter": "qa",
        "statusInstant": "2020-02-25T00:01:55.800004100Z",
        "trackingInstant": "2020-02-24T23:56:35.222019700Z",
        "msSinceRequest": 320578
    },
    {
        "siteId": 123,
        "snapshotId": 3393,
        "statusId": "08f9c6c6-5762-11ea-9927-d799a38b2424",
        "trackingId": "49f15c25-5761-11ea-a353-33d040339ebe",
        "source": "streaming-scoped-action",
        "statusType": "ScopedAction",
        "message": "CLOSING ScopedAction 158",
        "level": "SUMMARY",
        "statusData": null,
        "datacenter": "qa",
        "statusInstant": "2020-02-25T00:01:55.722003800Z",
        "trackingInstant": "2020-02-24T23:56:35.222019700Z",
        "msSinceRequest": 320500
    },
    {
        "siteId": 123,
        "snapshotId": 3393,
        "statusId": "08f4be58-5762-11ea-a353-956b33f0d960",
        "trackingId": "49f15c25-5761-11ea-a353-33d040339ebe",
        "source": "streaming-scoped-action",
        "statusType": "ScopedAction",
        "message": "ScopedAction 158 CLOSING_STARTED",
        "level": "SUMMARY",
        "statusData": {
            "scopedAction": "Closing_started"
        },
        "datacenter": "qa",
        "statusInstant": "2020-02-25T00:01:55.689020Z",
        "trackingInstant": "2020-02-24T23:56:35.222019700Z",
        "msSinceRequest": 320467
    }

 

  • Was this article helpful?