Streaming Snapshot Service

Snapshot Types

Product Snapshot

A product snapshot is required before catalog items can be added through the streaming-ingest service. The product snapshot is associated to both a "published" product and category property definition collection. 

Find requires that the search attributes have already been setup in the property definition product collection. Search Attributes are specific to FIND and are not required for Recommend. 

NOTE:

• The streaming catalog does not use the Portal’s search attributes but there are plans to offer this functionality in the future. 
• Attributes that are searchable are still required in the Portal's search attributes. 

Place Snapshot

The place snapshot is required to define regions. The place snapshot requires the "published" region property definition collection. Region over-rides are still added to the product (at ingest time) and the streaming engine will validate the regions by referencing the region definitions in the place snapshot. 

Note: The Place snapshot is required and must be "active" even if regions are not used.  

Snapshot States

A snapshot has five states: creating, complete, active, archived and deleted.

For each environment there is a limit on the number of snapshot in each state. In production, for each site, there can only be two “creating”, one “completed”, one “active”, three "archived" snapshots at any one time. There are unlimited "deleted" snapshots. Limits details can be found here.

When a snapshot is initially created and associated to the required property definition collections it will be in the "creating" state with an empty catalog. A "creating" snapshot is used for testing how items flows through the streaming catalog but it does not go to Find or the Legacy Catalog. It does go the the view store so transformations can be viewed.

When ingesting data, a snapshot is referenced and the data flows to “engine.out”. How data flows from "engine.out" to Find and the Catalog Database (Recommend & DIscover) depends on the snapshot state and is discussed further below:

Find and Snapshot States

Items that are in the engine.out topic are replicated to the Front End data centers.

• Items in a "creating" snapshot only flow to the streaming-view store and not to Find.  
• The "complete" states triggers the creation of a Find index which is called the "Cutover Snapshot Collection". It is not accessed by the Find API but will index all products ingested into a "complete" snapshot. This provides an opportunity to add catalog items but not have them live (in Find) until the snapshot becomes “active”.  
•  When a "complete" snapshot is “activated”, then that snapshot's index will become the production "Current Snapshot Collection". The previous "active" snapshot is archived.

Recommend & Discover (postgres database):

For Recommend & Discover, the product and category items in snapshot that is either "complete" or "active" state, will flow to the Recommend & Discover database (postgres) via the Legacy Catalog Adaptor. When a snapshot is in the "creating" state, the Legacy Catalog Adaptor will not pick up the items. "Creating" is just for testing items flowing to the view store. 

When a "complete" snapshot is “activated” it doesn't have any specific impact on the Recommend DB. The Legacy Catalog Adaptor is not snapshot aware. This means that items from any "complete" or "active" snapshot maybe in the Legacy Catalog. The Legacy Catalog can be cleaned up by deleting items from another snapshot or a scoped action to sync the legacy catalog to the "active" snapshot can be done. The scoped action will ensure that all items in the Legacy Catalog are the same as in the "active" snapshot. It will delete any items in the Legacy Catalog that are not in the current "active" snapshot. More information on the scoped action to sync the Legacy Catalog can be found in the Scoped Action chapter.

NOTE: For both Find and Recommend customers,

Once satisfied the “creating” snapshot  is working well, a new snapshot is created, with the same “published”  property definition collections (as the "creating" test snapshot). This new “creating” snapshot needs to be set immediately to “complete” before ingesting items. Once set to "complete" ingested items will flow to Find and the Catalog Database (Recommend and Discover) 

This new empty snapshot will be used to ingest items which will flow to Find’s “cutover snapshot collection” as well as to the Legacy Catalog Adaptor (to support Recommend and Discover).

Once the snapshot is “activated”, then the snapshot items flow to the current production FIND Solr collection. When a snapshot is activated, the previously active snapshot is archived.

Parameters

Create a new product snapshot

The property definitions for category and product have already been created and published. 

In this example the apiKey is 123. Note that "name" is optional and can be omitted. Below are examples with and without the "name" parameter in the body. 

POST https://gateway.richrelevance.com/streaming-snapshot/v1/123/create/product
'Authorization:Bearer <tokenValue>’

Body:
{“name”:”test-snapshot”,
"propertyDefinitionCollectionIds": {"category":578, "product":579}}

OR body without "name"
{"propertyDefinitionCollectionIds": {"category":578, "product":579}}

Response:

{
    "statusTracker": {
        "trackingId": "d6b10e13-cacf-11ea-bf3e-cf3f136f672e",
        "trackingInstant": "2020-07-20T21:27:39.719016300Z"
    },
    "snapshot": {
        "id": 400,
        "siteId": 123,
        "state": "creating",
        "propertyDefinitionCollectionIds": {
            "product": 579,
            "category": 578
        },
        "lastModified": "2020-07-20T21:27:39.718651Z",
        "name": "test-snapshot",
        "type": "product"
    }
}

The following is one method to count all products in a specific snapshot. The streaming-view has a responseStyle=count which counts but it is limited to 50,000 products at a time. 

The "count" action

The "count" action will count the number of ingested products, categories, regions, referenceContent and assortments for both the legacy catalog and the view store. The count is provided in a status message associated to the trackingId in the response. This method is recommended for customers who have millions of products.  

Note: a product snapshot needs to be provided as count for the view store is only for that specific product snapshot.   

Syntax is as follows:

POST https://<host>/streaming-snapshot/v1/<apiKey>/count/<snapshotId>/product
'Authorization:Bearer <tokenValue>’

Example where:  apiKey=123, snapshotId=7932

POST https://gateway.richrelevance.com/streaming-snapshot/v1/123/count/7932/product
'Authorization:Bearer <tokenValue>’

Response:

{
    "message": "Kicking off counting for snapshot 7932",
    "trackingId": "71248c63-e1a7-11ea-8565-297c33d675f6",
    "trackingInstant": "2020-08-18T23:06:26.223011500Z"
}

Get the counts from the status service using the trackingId in the response

GET https://gateway.richrelevance.com/streaming-status/v1/123/trackingId/71248c63-e1a7-11ea-8565-297c33d675f6

Response:

[
    {
        "siteId": 123,
        "snapshotId": 7932,
        "statusId": "e6c47203-ee36-11ea-b4e1-49d1afc58bb7",
        "trackingId": "71248c63-e1a7-11ea-8565-297c33d675f6",
        "source": "view-store",
        "statusType": "CountEvent",
        "message": "Snapshot Count",
        "kafkaSource": {
            "topic": "streaming.engine.out",
            "partition": 7,
            "offset": 115560
        },
        "level": "SUMMARY",
        "statusData": {
            "product": {
                "total": 30,
                "recommendable": 10,
                "nonRecommendable": 20
            },
            "category": {
                "total": 0
            },
            "region": {
                "total": 0
            },
            "referenceContent": {
                "total": 0
            },
            "assortment": {
                "total": 0
            }
        },
        "datacenter": "qa",
        "statusInstant": "2020-09-03T22:43:35.550003500Z",
        "trackingInstant": "2020-09-03T22:43:35.218013100Z",
        "msSinceRequest": 332
    },
    {
        "siteId": 123,
        "snapshotId": 7932,
        "statusId": "e6addcc2-ee36-11ea-b4e1-c390e055419c",
        "trackingId": "71248c63-e1a7-11ea-8565-297c33d675f6",
        "source": "legacy-catalog",
        "statusType": "CountEvent",
        "message": "Legacy Catalog Count",
        "kafkaSource": {
            "topic": "streaming.engine.out",
            "partition": 7,
            "offset": 115560
        },
        "level": "SUMMARY",
        "statusData": {
            "product": {
                "total": 6906,
                "recommendable": 10,
                "nonRecommendable": 6896
            },
            "category": {
                "total": 0
            },
            "region": {
                "total": 5
            },
            "referenceContent": {
                "total": 0
            },
            "assortment": {
                "total": 0
            }
        },
        "datacenter": "qa",
        "statusInstant": "2020-09-03T22:43:35.402003400Z",
        "trackingInstant": "2020-09-03T22:43:35.218013100Z",
        "msSinceRequest": 184
    },
    {
        "siteId": 123,
        "snapshotId": 7932,
        "statusId": "e6943a80-ee36-11ea-8dca-01010e2a5661",
        "trackingId": "71248c63-e1a7-11ea-8565-297c33d675f6",
        "source": "streaming-engine",
        "statusType": "CountEvent",
        "message": "CountEvent",
        "kafkaSource": {
            "topic": "streaming.engine.in",
            "partition": 7,
            "offset": 329001
        },
        "level": "SUMMARY",
        "statusData": {},
        "datacenter": "qa",
        "statusInstant": "2020-09-03T22:43:35.234009600Z",
        "trackingInstant": "2020-09-03T22:43:35.218013100Z",
        "msSinceRequest": 16
    },
    {
        "siteId": 123,
        "snapshotId": 7932,
        "statusId": "e6943a41-ee36-11ea-b4e1-41cbe01e3dca",
        "trackingId": "71248c63-e1a7-11ea-8565-297c33d675f6",
        "source": "streaming-engine-sidekick",
        "statusType": "CountEvent",
        "message": "Received CountEvent for site 123 snapshot 7932",
        "kafkaSource": {
            "topic": "streaming.engine.out",
            "partition": 7,
            "offset": 115560
        },
        "level": "SUMMARY",
        "statusData": {},
        "datacenter": "qa",
        "statusInstant": "2020-09-03T22:43:35.234003300Z",
        "trackingInstant": "2020-09-03T22:43:35.218013100Z",
        "msSinceRequest": 16
    },
    {
        "siteId": 123,
        "snapshotId": 7932,
        "statusId": "e69265e4-ee36-11ea-8425-19e8e00e8fd3",
        "trackingId": "71248c63-e1a7-11ea-8565-297c33d675f6",
        "source": "streaming-snapshot",
        "message": "Counting snapshot 9113",
        "level": "SUMMARY",
        "statusData": {},
        "datacenter": "qa",
        "statusInstant": "2020-09-03T22:43:35.222013200Z",
        "trackingInstant": "2020-09-03T22:43:35.218013100Z",
        "msSinceRequest": 4
    }
]

Note that the counts for the place and referenceContent will count all snapshots for those types. There should only be one "active" place snapshot.