Streaming Status Service
Introduction
Each streaming catalog service processes each item independently of the other streaming catalog services. Each service will provide a status of the number of items successfully processed as well as the invalid items. Information on why an item is invalid is sent to the “invalid item store” which is retrieved through the status service. Note that messages in the invalid item store has a time to live of 200 hours.
The streaming catalog currently creates batches of 50 items for a payload associated to a trackingId. There is a status message for each batch as well as a final "finished" status message for all items associated to the trackingId.
Pagination is available to assist in paging through status messages as well as the invalid item store. It is possible to page by 1,000 to 10,000 status messages per page. If pagination is not used, a call to the status service may time out if there are thousands of status messages. By default status messages are ordered from most recent to oldest. There is an "order" parameter to reverse the order from oldest to most recent.
Status Message Levels
There are four levels of status messages.
-
ERROR
-
WARNING
-
SUMMARY
-
DETAIL
Status Level Descriptions
SUMMARY
A final "finished" status message is recorded for all items associated to a trackingId, which are successfully processed. If there are any invalid items then the final "finished" status message will be at the WARNING level.
For all services except streaming-ingest, successful batch processing is at the SUMMARY level. Streaming-ingest records successful batch processing at the DETAIL level. If there is an invalid item it will be at the WARNING level for the batch.
For each payload, items are sent through the streaming catalog in batches. A batch will have a variable number of items as the size limit for one batch is 1.5 MB and can hold up to 800 items. It is common for the batch to be between 500KB and 1MB. One item cannot be larger than 1 million bytes.
Below are descriptions for each status level:
WARNING
For all streaming catalog services, if there is an error during processing, a WARNING status message will be recorded. The count of the successful processed status items as well as the count of the invalid items will be in the stats section of the status message.
For example, the streaming-ingest validates the item's format:
If the JSONL format is invalid the following messages are recorded:
- WARNING status message for a batch - includes number of successful items and invalid items.
- WARNING status message for the final "finished" status message - includes number of successful items and invalid items.
- DETAIL status message for the individual invalid item.
The streaming ingest will not send the invalid items items to the streaming-engine. It will send those items directly to the invalid item store. It will only send the successful items to the streaming-engine.
The next service, the streaming engine, will read all items and verify the properties. If any properties are invalid, then the item is rejected and will be tabulated as part of the “finish items as invalid. The streaming-engine will provide statistics of successful and invalid items.
DETAIL
Streaming-ingest service records successful processing for each batch at the DETAIL level. If there is an invalid item it will be at the WARNING level for the batch.
ERROR
ERROR message can be a result of an internal issue. For example an application is down and processing has stopped. It is possible that the system may self-recover (retry) and proceed where it left off. If it doesn’t resolve itself contact Algonomy.
Streaming Status Endpoints and Parameters
Status messages are generated by each service. They can be retrieved by trackingId, date, or date&service.
Due to the volume of status messages generated, they have a specific time to live (TTL):
- status messages by trackingId - 30 days
- status messages by date - 10 days
- debug messages (internal to RR) - 2 days.
Both the "level" parameter and the pagination parameters can be applied to any of the Status Service endpoints. They are discussed below.
Status “Level” Parameter
For each of the endpoints below status messages returned by default are: SUMMARY, WARNING and ERROR.
The “level” parameter can be used with any of the following streaming-status endpoints to specify one or more specific status message levels to be returned.
The following examples result in the same status messages levels returning:
?level=summary, error,warning
?level=summary&level=error&level=warning
?level=summary,error&level=warning
There are optional parameter values that map to multiple status levels:
warningPlus → maps to WARNING + ERROR
summaryPlus → maps to SUMMARY + WARNING + ERROR
detailPlus → maps to DETAIL + SUMMARY + WARNING + ERROR
Status Service Pagination Parameters
If the number of status messages is in the thousands it is recommended to page through the list of status messages, otherwise the status service may timeout. There are three parameters "rows", "page" and "order". By default the order is returned by the most recent status messages to the oldest.
Pagination is optional and is only invoked when the "rows" parameter is included in the call. If the "rows" parameter is not included, all status messages will be returned.
If the “rows” parameter is included, then the list of status messages will be paginated and a pagination JSON is included at the beginning of the response. The pagination response will include the number of status messages returned as well as the nextPage token. It is possible for the number of rows in the pagination JSON to be different from the rows requested. Currently the filtering of the status message returned happens after retrieving the status messages.
See example below:
GET https://<host>/streaming-status/v1/<apikey>/trackingId/<trackingId>?rows=1000
Response:
{
"pagination": {
"rows": 1000,
"nextPage": "28a42e01-6390-11ea-80c7-edc39b8269af"
},
"results": [
{
"siteId": 123,
"snapshotId": 3740,
"statusId": "5aea15f7-63a7-11ea-b671-a3965c025a90",
"trackingId": "25c72599-6390-11ea-b436-870e85ddcd60",
"source": "find-index-deployer",
"statusType": "StreamingEvent",
"message": "Streaming item processed correctly ",
"level": "SUMMARY",
"statusData": {
....
the 1000 status messages are included in the results.
To get to the next page, use the nextPage token with the "page" parameter as illustrated below. Note that in the response if there isn't a nextPage token it means there are no more status messages.
GET https://<host>/streaming-status/v1/<apikey>/trackingId/<trackingId>?rows=1000&page="28a42e01-6390-11ea-80c7-edc39b8269af"
Response:
{
"pagination": {
"rows": 300,
},
"results": [
{
"siteId": 123,
"snapshotId": 3740,
"statusId": "470c4c14-6390-11ea-b182-d9144468c897",
"trackingId": "25c72599-6390-11ea-b436-870e85ddcd60",
"source": "streaming-consumer-legacy-catalog",
"statusType": "InternalEnvelope",
"message": "Received update action",
"level": "SUMMARY",
"statusData": {
"stats": {
"product": {
"total": 50,
"update": 50
}
}
},
"datacenter": "qa",
"statusInstant": "2020-03-11T12:03:10.665013200Z",
"trackingInstant": "2020-03-11T12:02:14.847016900Z",
"msSinceRequest": 55818
},
{
"siteId": 123,
"snapshotId": 3740,
"statusId": "470b61b1-6390-11ea-b182-47122ae1803e",
"trackingId": "25c72599-6390-11ea-b436-870e85ddcd60",
"source": "streaming-consumer-legacy-catalog",
"statusType": "InternalEnvelope",
"message": "Received update action",
"level": "SUMMARY",
"statusData": {
"stats": {
"product": {
"total": 50,
"update": 50
}
}
},
"datacenter": "qa",
"statusInstant": "2020-03-11T12:03:10.659012900Z",
"trackingInstant": "2020-03-11T12:02:14.847016900Z",
"msSinceRequest": 55812
},
....
the rest of the 300 status messages
Streaming Status Endpoint Syntax
Base URL: https://<host>/streaming-status/v1/
'Authorization:Bearer <tokenValue>’
Full URL: https://<host>/streaming-status/v1/<apiKey>/<statusType>/[<snapshotId>]/[?level=<level>]
'Authorization:Bearer <tokenValue>’
| Actions | API Syntax |
|---|---|
|
Get all status for a specific tracking ID. Default status levels - SUMMARY, WARNING and ERROR are be returned for all streaming services. |
GET {baseURL}/<apiKey>/trackingId/<trackingId> Specify one or more status levels: GET {baseURL}/<apiKey>/trackingId/<trackingId>?level=<level> |
|
Check status of all transactions for one day for a specific snapshot.
|
GET {baseURL}/<apiKey>/daySnapshot/<snapshotId>?date=yyyy-mm-dd
|
| Check status of all transactions for one streaming service (such as streaming-ingest, streaming-snapshot) for a specific day, and snapshot. | GET {baseURL}/<apiKey>/daySnapshotService/<snapshotId>?date=yyyy-mm-dd&service=<streaming service> |
|
View invalid items in the invalid item store. Note: Invalid items have a time to live (TTL) of 200 hours. |
GET {baseURL}/<apiKey>/invalidItems/<trackingId> |
Parameters
| Name | Description | Values and Additional Information |
|---|---|---|
| <statusType> |
statusType provides a means to bring back statuses for either a specific trackingId, for one day (specific snapshot), or for a specific service but restricting it to one snapshot. This is a string that needs to be provided. The invalidItems value will return information on each invalid item for a specific trackingId. |
trackingId daySnapshot daySnapshotService invalidItems |
| <apiKey> | Unique identifier for a customer’s environment. For example, if a customer has multiple environments in production or staging, each would have a unique apiKey. There can be many apiKeys associated to a client_ID | Provided by Algonomy to customer |
| <snapshotId> | Unique identifier for the snapshot. | snapshotId is provided in the response to a POST when creating the snapshot |
| service |
Parameter to request status for a specific Streaming Catalog service. Descriptions of each service can be found after the parameters section below. |
streaming-property streaming-snapshot streaming-ingest streaming-engine streaming-sidekick streaming-consumer-legacy-catalog streaming-find-indexer streaming-invalid-item streaming-scoped-action
|
| level |
Level of status message to be retrieved from the streaming status service. The default returns SUMMARY, WARNING and ERROR. Can specify one or more levels. Example: ?level=summary All of the following will have the same result. ?level=summary, error,warning ?level=summary&level=error&level=warning ?level=summary,error&level=warning |
One or more of the following status levels can be specified: error warning summary detail
Optionally any of the following parameter values can be used, each of which maps to multiple status levels: Parameter values: warningPlus → maps to WARNING + ERROR summaryPlus → maps to SUMMARY + WARNING + ERROR detailPlus → maps to DETAIL + SUMMARY + WARNING + ERROR |
| date |
Parameter to restrict status messages to a specific day.
|
date=yyyy-dd-mm |
| rows |
Number of results per page to return for pagination. Max is 10,000. Integer. A negative number produces a 400 error. Rows can equal 0. |
&rows=<statusMessagesPerPage> Example: rows=1,000 |
| page | The nextPage token is returned in the pagination JSON when the parameter "rows" "is used. The nextPage token is used to return the next page of status messages. String. | page=<nextPage> |
| order | By default the order of the status messages returned is from the most recent to the oldest. The reverse option orders by the oldest to the most recent. Order must be used in conjunction with the "rows" parameter. It only works with pagination. |
order=reverse default: natural |
Streaming Status Response Elements
Each status message returned, for each source (service), is organized as illustrated in the example below.
The first section has general information including the source. The list of sources is provided after this example.
StatusData - Demarcation
Each service, except for the streaming-find-indexer, in the statusData section will provide the number of batches processed and if the service is finished processing the payload is finished. The "finished" element is either "true" or "false". If "false" it is still processing. "True" is finished processed.
In the above example, the streaming-engine has finished processing 5 batches.
To determine the number of batches in a payload, check the streaming-ingest status. It will list the number of batches to be processed.
This feature is extremely useful to determine when the legacy catalog (streaming-consume-legacy-catalog) has processed a payload. Below is an example of the status for a finished streaming-catalog-legacy-catalog.
"siteId": 608,
"snapshotId": 9758,
"statusId": "a019a4d7-681c-11eb-bee7-cbd1128b70de",
"trackingId": "9d5e2dd7-681c-11eb-920a-27a53355597f",
"source": "streaming-consumer-legacy-catalog",
"statusType": "InternalEnvelope",
"message": "Finished InternalEnvelope",
"kafkaSource": {
"topic": "streaming.engine.out",
"partition": 7,
"offset": 119661
},
"level": "SUMMARY",
"statusData": {
"cursor": {
"batch": 1,
"finished": true
}
},
"datacenter": "qa",
"statusInstant": "2021-02-06T01:42:51.934024700Z",
"trackingInstant": "2021-02-06T01:42:47.350011900Z",
"msSinceRequest": 4584
},
"Source" Descriptions
| Source | Description |
|---|---|
|
streaming-property |
The streaming-property service is responsible for the creation and publishing of property definition collections. |
|
streaming-snapshot |
The streaming-snapshot service creates and validates that the required published property definition collections are available. |
| streaming-ingest | It is through the streaming-ingest service that items are ingested via a JSONL format. The streaming-ingest validates the JSONL format. |
| streaming-engine | The streaming engine service is responsible for first validating and transforming data as required. It then determines the differences (deltas) for each ingested item and sends the deltas to the view store and engine.out. Find and the Legacy catalog adaptor both pick up the items from engine.out. |
| streaming-sidekick |
The streaming-sidekick does the background work for the streaming engine. This service is involved in closing a scoped action.
|
| streaming-consumer-legacy-catalog |
The Legacy Catalog Adaptor picks up items from engine.out and sends them to the catalog database which is used by both Recommend and Discover. Note: A batch job is still required for Recommend and Discover to process the updates. |
| streaming-find-indexer |
The streaming-find-indexer creates the Find index on each of the 12 front end data centers. Data that is sent to engine.out is replicated to each of the 12 data centers to be indexed by Find. Any item that fails to index are sent to the invalid item store. The data center is also included on each status message. Note: It is now possible to restrict customer's data to go to specific data centers. |
| streaming-invalid-item |
The streaming-invalid-item writes each invalid item to the invalid-item-store. Note: There is a limit of 50 invalid items in a period of 5 minutes. Processing of valid items will continue but additional invalid items will not be written to the invalid item store for the 5 minute period. Will resume after 5 minutes. This limit is by site and not by snapshot. |
| streaming-scoped-action |
Scoped actions are part of both the snapshot service and ingest service. A scoped action is "open" through the snapshot service, ingestion of items is managed through the ingestion service. Streaming-ingest-action keeps track of all "touched" and "untouched" items. Upon closing a scoped action, it will determine which items were :untouched" and will send delete events to engine.out.
|
Examples for each streaming catalog service
Below are examples of status for each streaming catalog service. Note that when requesting a status to be returned all streaming catalog services' statuses are returned.
Streaming-property service status message (Success)
The following example involves only the streaming-property service. In this example a product property collection is published.
apiKey: 123
Product property collection: 9563
Below is the streaming-property request to publish a product property collection. The response includes the trackingId
POST https://gateway.richrelevance.com/streaming-property/v1/123/collections/product/9563/publish
Response:
{
"statusTracker": {
"trackingId": "f2bfe141-4c4f-11ea-bdfb-7d1710347a3f",
"trackingInstant": "2020-02-10T21:54:44.701011300Z"
}
}
Check on the status of the published property collection:
GET https://<host>.com/streaming-status/v1/123/trackingId/f2bfe141-4c4f-11ea-bdfb-7d1710347a3f
Response:
[
{
"statusId": "f2c38ac2-4c4f-11ea-bdfb-99d563adc033",
"siteId": 123,
"trackingId": "f2bfe141-4c4f-11ea-bdfb-7d1710347a3f",
"snapshotId": 0,
"source": "streaming-property",
"statusType": "PropertyDefinitionCollectionEvent",
"message": "PropertyDefinitionCollection product 9563 published",
"level": "SUMMARY",
"statusData": null,
"statusInstant": "2020-02-10T21:54:44.725011400Z",
"msSinceRequest": 24,
"trackingInstant": "2020-02-10T21:54:44.701011300Z"
}
]
Ingest products - status messages for each service (success)
In the following example, one product was ingested into snapshot 3199. Each streaming catalog service will provide its own status messages and will note if it is successful or not. Note that in the response, the latest status from a streaming catalog service is at the top of the response. For the purposes of explaining the status messages, they are not in their original order but by streaming catalog service. Go to “response” to see the original order of the messages.
apiKey: 123
snapshotId:3199
trackingId:: 187c7b6e-4d13-11ea-b4c0-350a29a5f200
Get the status messages:
GET https://<host>/streaming-status/v1/123/trackingId/f187c7b6e-4d13-11ea-b4c0-350a29a5f200 'Authorization:Bearer <tokenValue>’
RESPONSE:
Below are status messages organized by each streaming catalog service.
Streaming-ingest status messages:
The streaming ingest service has received the item.
{
"statusId": "187c7b6f-4d13-11ea-b4c0-f58ae4f095e2",
"siteId": 123,
"trackingId": "187c7b6e-4d13-11ea-b4c0-350a29a5f200",
"snapshotId": 3199,
"source": "streaming-ingest",
"statusType": "item",
"message": "Received REPLACE action",
"level": "SUMMARY",
"statusData": null,
"statusInstant": "2020-02-11T21:11:39.875006300Z",
"msSinceRequest": 0,
"trackingInstant": "2020-02-11T21:11:39.875006200Z"
}
The streaming-ingest service will validate the JSON is correct and also batch the items. Current batch size is 50.
{
"statusId": "187ca280-4d13-11ea-b4c0-fb4795316a50",
"siteId": 123,
"trackingId": "187c7b6e-4d13-11ea-b4c0-350a29a5f200",
"snapshotId": 3199,
"source": "streaming-ingest",
"statusType": "item",
"message": "Batch sending 1 item(s)",
"level": "SUMMARY",
"statusData": {
"stats": {
"product": {
"total": 1,
"replace": 1
}
}
},
"statusInstant": "2020-02-11T21:11:39.876006400Z",
"msSinceRequest": 1,
"trackingInstant": "2020-02-11T21:11:39.875006200Z"
},
The streaming-ingest indicates it is “finished” and has sent the batch to the streaming-engine where the items will be processed.
{
"statusId": "187d3ec1-4d13-11ea-b4c0-f9c7a09dff37",
"siteId": 123,
"trackingId": "187c7b6e-4d13-11ea-b4c0-350a29a5f200",
"snapshotId": 3199,
"source": "streaming-ingest",
"statusType": "item",
"message": "Finished REPLACE action",
"level": "SUMMARY",
"statusData": {
"stats": {
"product": {
"total": 1,
"replace": 1
}
}
},
"statusInstant": "2020-02-11T21:11:39.880006500Z",
"msSinceRequest": 5,
"trackingInstant": "2020-02-11T21:11:39.875006200Z"
Streaming-engine status:
The streaming-engine validates that the properties are valid, that they exist in the product property collection. Streaming-engine also determines the delta of information to be provided to engine.out. Note that “stats” are provided on the total number of items and those that were successfully processed.
All other streaming service consumers such streaming-view, Find and the Legacy catalog adaptor process the items from engine.out. These services will update, delete, and add new products, categories and regions.
{
"statusId": "189c1142-4d13-11ea-b937-03ad09d513d3",
"siteId": 123,
"trackingId": "187c7b6e-4d13-11ea-b4c0-350a29a5f200",
"snapshotId": 3199,
"source": "streaming-engine",
"statusType": "ExternalEnvelope",
"message": "ItemEvent",
"level": "SUMMARY",
"statusData": {
"stats": {
"product": {
"total": 1,
"update": 1
}
}
},
"statusInstant": "2020-02-11T21:11:40.082003400Z",
"msSinceRequest": 207,
"trackingInstant": "2020-02-11T21:11:39.875006200Z"
},
Streaming-view status:
Streaming-view receives updates, deletes and new items from the engine.out. This status acknowledges information has been received.
{
"statusId": "189c1198-4d13-11ea-845e-37a48a79f79c",
"siteId": 123,
"trackingId": "187c7b6e-4d13-11ea-b4c0-350a29a5f200",
"snapshotId": 3199,
"source": "streaming-consumer-view",
"statusType": "InternalEnvelope",
"message": "Received InternalEnvelope",
"level": "SUMMARY",
"statusData": null,
"statusInstant": "2020-02-11T21:11:40.082012Z",
"msSinceRequest": 207,
"trackingInstant": "2020-02-11T21:11:39.875006200Z"
},
The streaming-view has been successfully updated. "Finished" status message.
{
"statusId": "18b62949-4d13-11ea-845e-cf23941c8686",
"siteId": 123,
"trackingId": "187c7b6e-4d13-11ea-b4c0-350a29a5f200",
"snapshotId": 3199,
"source": "streaming-consumer-view",
"statusType": "InternalEnvelope",
"message": "Finished InternalEnvelope",
"level": "SUMMARY",
"statusData": {
"stats": {
"product": {
"total": 1,
"update": 1
}
},
"statusInstant": "2020-02-11T21:11:40.253012100Z",
"msSinceRequest": 378,
"trackingInstant": "2020-02-11T21:11:39.875006200Z"
},
Invalid Item Store
To verify that items are accepted and processed, it is possible to do the following:
-
Check the status service
-
Every service will have a status. It will also include a “finish” message.
-
Check the “statsData”
-
If there is an issue, then the item will go to the invalid item store
-
For each service, a “statsData” is provided that lists all successful and invalid items. The level for a message with an invalid item is “WARNING”.
-
-
Valid items will contintue to flow through to the streaming-view store. Can check the streaming-view to view what has flowed through.
-
Invalid items go to the invalid item store which is accessed through the streaming-status service.
Example - ingested category had a JSON error. apiKey is 123.
GET https://<host>/streaming-status/v1/123/invalidItems/329ff7b5-d8b7-11e9-9c98-df4ddcfbc245 'Authorization:Bearer <tokenValue>’
Response:
[
{
"externalId": "",
"externalItemJson": "{\"test_new_category2\":{\"properties\":{\"name\":\"test_new_category name\",\"category_link_url\": \"/link/url/test_new_category\"}} ",
"internalItem": null,
"itemType": "category",
"message": "Invalid json syntax: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte[])\"{\"test_new_category2\":{\"properties\":{\"name\":\"test_new_category name\",\"category_link_url\": \"/link/url/test_new_category\"}} \"; line: 1, column: 1])\n at [Source: (byte[])\"{\"test_new_category2\":{\"properties\":{\"name\":\"test_new_category name\",\"category_link_url\": \"/link/url/test_new_category\"}} \"; line: 1, column: 245]",
"siteId": 123,
"snapshotId": 200,
"trackingId": "329ff7b5-d8b7-11e9-9c98-df4ddcfbc245",
"ts": "32a5c410-d8b7-11e9-a8f5-add7163ced05"
}
=
Example of status when 50 invalid items are written to the invalid item store in a 5 minute period. Additional invalid items will not be written to the invalid items store. This is to avoid flooding the system with invalid items.
GET https://<host>/streaming-status/v1/123/invalidItems/67a1751c-c723-11ec-ab38-f7e58ab8184a 'Authorization:Bearer <tokenValue>’
[
{
"siteId": 1779,
"snapshotId": 16223,
"statusId": "67b5c096-c723-11ec-b7c5-4f8a3d001df1",
"trackingId": "67a1751c-c723-11ec-ab38-f7e58ab8184a",
"source": "streaming-consumer-invalid-item",
"statusType": "InvalidItem",
"message": "site 1779 has over 50 invalid items were received in the last 300 seconds, further invalid items will not be stored until the rate of invalid items arriving falls below the threshold",
"kafkaSource": {
"topic": "streaming.engine.invalid-item",
"partition": 4,
"offset": 6758393
},
"level": "WARNING",
"statusData": {},
"datacenter": "qa",
"trackingInstant": "2022-04-28T18:45:41.806006Z",
"statusInstant": "2022-04-28T18:45:41.939010200Z",
"msSinceRequest": 133
}
=
Full Response for a status service by tracking ID
Below are the same messages from above but in the actual order: This is for one product being ingested. Note that there was a problem with the find-depoyer-index (Find), so the status message is a warning. Legacy Catalog Adaptor service was not included as the snapshot was in the "creating" state. Legacy catalog adaptor requires a snapshot in the "complete" or "active" state.
GET https://gateway.richrelevance.com/streaming-status/v1/123/trackingId/f187c7b6e-4d13-11ea-b4c0-350a29a5f200
Response:
[
{
"statusId": "18b62949-4d13-11ea-845e-cf23941c8686",
"siteId": 123,
"trackingId": "187c7b6e-4d13-11ea-b4c0-350a29a5f200",
"snapshotId": 3199,
"source": "streaming-consumer-view",
"statusType": "InternalEnvelope",
"message": "Finished InternalEnvelope",
"level": "SUMMARY",
"statusData": {
"stats": {
"product": {
"total": 1,
"update": 1
}
}
},
"statusInstant": "2020-02-11T21:11:40.253012100Z",
"msSinceRequest": 378,
"trackingInstant": "2020-02-11T21:11:39.875006200Z"
},
{
"statusId": "189c5ffa-4d13-11ea-8009-4b3869866863",
"siteId": 123,
"trackingId": "187c7b6e-4d13-11ea-b4c0-350a29a5f200",
"snapshotId": 3199,
"source": "find-index-deployer",
"statusType": "StreamingEvent",
"message": "Streaming item not processed correctly",
"level": "WARNING",
"statusData": {
"stats": {
"product": {
"total": 1,
"product ignored": 1
}
}
},
"statusInstant": "2020-02-11T21:11:40.084018600Z",
"msSinceRequest": 209,
"trackingInstant": "2020-02-11T21:11:39.875006200Z"
},
{
"statusId": "189c1198-4d13-11ea-845e-37a48a79f79c",
"siteId": 123,
"trackingId": "187c7b6e-4d13-11ea-b4c0-350a29a5f200",
"snapshotId": 3199,
"source": "streaming-consumer-view",
"statusType": "InternalEnvelope",
"message": "Received InternalEnvelope",
"level": "SUMMARY",
"statusData": null,
"statusInstant": "2020-02-11T21:11:40.082012Z",
"msSinceRequest": 207,
"trackingInstant": "2020-02-11T21:11:39.875006200Z"
},
{
"statusId": "189c1142-4d13-11ea-b937-03ad09d513d3",
"siteId": 123,
"trackingId": "187c7b6e-4d13-11ea-b4c0-350a29a5f200",
"snapshotId": 3199,
"source": "streaming-engine",
"statusType": "ExternalEnvelope",
"message": "ItemEvent",
"level": "SUMMARY",
"statusData": {
"stats": {
"product": {
"total": 1,
"update": 1
}
}
},
"statusInstant": "2020-02-11T21:11:40.082003400Z",
"msSinceRequest": 207,
"trackingInstant": "2020-02-11T21:11:39.875006200Z"
},
{
"statusId": "187d3ec1-4d13-11ea-b4c0-f9c7a09dff37",
"siteId": 123,
"trackingId": "187c7b6e-4d13-11ea-b4c0-350a29a5f200",
"snapshotId": 3199,
"source": "streaming-ingest",
"statusType": "item",
"message": "Finished REPLACE action",
"level": "SUMMARY",
"statusData": {
"stats": {
"product": {
"total": 1,
"replace": 1
}
}
},
"statusInstant": "2020-02-11T21:11:39.880006500Z",
"msSinceRequest": 5,
"trackingInstant": "2020-02-11T21:11:39.875006200Z"
},
{
"statusId": "187ca280-4d13-11ea-b4c0-fb4795316a50",
"siteId": 123,
"trackingId": "187c7b6e-4d13-11ea-b4c0-350a29a5f200",
"snapshotId": 3199,
"source": "streaming-ingest",
"statusType": "item",
"message": "Batch sending 1 item(s)",
"level": "SUMMARY",
"statusData": {
"stats": {
"product": {
"total": 1,
"replace": 1
}
}
},
"statusInstant": "2020-02-11T21:11:39.876006400Z",
"msSinceRequest": 1,
"trackingInstant": "2020-02-11T21:11:39.875006200Z"
},
{
"statusId": "187c7b6f-4d13-11ea-b4c0-f58ae4f095e2",
"siteId": 123,
"trackingId": "187c7b6e-4d13-11ea-b4c0-350a29a5f200",
"snapshotId": 3199,
"source": "streaming-ingest",
"statusType": "item",
"message": "Received REPLACE action",
"level": "SUMMARY",
"statusData": null,
"statusInstant": "2020-02-11T21:11:39.875006300Z",
"msSinceRequest": 0,
"trackingInstant": "2020-02-11T21:11:39.875006200Z"
}