Data Exchange Enhancements and Changes

Summary

The following enhancements to STEP's data exchange functionality have been made as part of the 11.1 release. These updates are outlined below and described in the Details section that follows:

  • Updates to the Kafka integration include improvements for Kafka Receiver functionality and support for the latest version.

  • Excel and CSV data exchange has new mapping options for references and product-to-classification links (including their metadata) and data containers. These new mapping options result in a better user experience during mapping and an easier-to-understand layout in the Excel or CSV file.

  • Exports using the CSV file format now allow removing headers to streamline data transfer and avoid manual manipulation of files for downstream systems that cannot consume a header.

  • The asset push sidecar client can now communicate with the STEP server via HTTP, HTTPS, and SOCKS proxy servers.

  • New configurable IIEP properties support automatic cleanup of files and allow admins to automatically delete successful files while errored files can be retained for troubleshooting.

  • New automated error handling and improved notification options on inbound and outbound integration endpoints (IEPs) and event processors (EPs) reduce process failures, reduce required manual intervention for failed processing, and improve management of processors.

  • Outbound integration endpoints (OIEPs) and event processors (EPs) now include a better estimate for the number of events displayed in the 'Unread events (approximated)' parameter on SaaS systems.

  • The OIEP Business Rule Based Message Processor now processes the nodes within a collection when the data source is Select Objects, and a collection object type is selected.

  • Multiple certificates and/or keystores can be defined and individually selected for outbound integration endpoints (OIEPs) and gateway integration endpoints (GIEPs) that use Mutual Transport Layer Security (mTLS) authentication. Admins can now define a certificate / keystore for each integration, including those for a third party.

  • Industry standard updates include support for the newest versions of ECLASS Basic, ETIM, UNSPSC, and GS1 GDSN.

  • The ECLASS Advanced Editor Screen is now available for editing ECLASS data. This screen improves upon and supersedes the view-only ECLASS Advanced Screen.

  • The IEP Transactional Setting parameter for processing in Chain mode is planned to be deprecated in a future release.

Details

Kafka integration improvements

Updates for the Kafka Receiver used in the workbench Inbound Integration Endpoint (IIEP) Wizard dialog now allows admins to monitor and modify the Topic Offset. The Topic Offset tracks the sequential order that messages are received by Kafka topics. Disabling the IIEP allows the offset to be changed, which gives admins the ability to reprocess failed messages.

The Kafka Receiver execution report is also improved for these scenarios:

  • When the Kafka broker is not running, the IIEP fails, and the number of logged entries is reduced since no messages are processed.

  • When the Kafka broker is running but no messages are retrieved, the logged entry reports the number of events, the Topic, and the Topic Offset.

Additionally, both the Kafka Receiver and the Kafka Delivery Method now support the latest versions of Apache Kafka software.

For more information, refer to the Kafka Receiver topic (here) and the Kafka Delivery Method topic (here), both in the Data Exchange documentation.

Improved format for complex data imports and exports using Excel and CSV

Data exchange using the Excel and CSV formats can now take advantage of a better file layout for references and product-classification links (and any metadata) and data containers owned by the objects being imported or exported. The new easily readable file layout makes user-validation and editing faster and simpler by allowing users to easily associate complex data structures with their corresponding values when using import manager, export manager, and inbound and outbound integration endpoints.

The new Excel or CSV file layout for both inbound and outbound displays each object on an individual row. The new 'Data Type' source column indicates how the data is related. Target data objects for the imported or exported object are displayed on individual rows following the 'Data Type' source identified by the 'NODE' text. Metadata attribute values are displayed in the appropriate column for the rows to which it belongs.

The 'Map Data' step for Import Manager, Export Manager, inbound and outbound integration endpoints includes new options for products, entities, classifications, and assets that make it easier to identify the data being received or generated by STEP.

Using new data source options, the format layout improvements include the ability to import and export:

  • Reference target ID, name, key, and metadata on the reference

  • Product-classification link target ID, name, key, and metadata on the link

  • Data container values

Also, using the new data sources, now data container references (the target ID and name) can also be imported and exported in the improved file layout.

For additional documentation, refer to the Outbound Map Data - Data Source (here) and the Inbound Map Data - Map (here) topics, both in the Data Exchange documentation.

For more information on this functionality, click the video below:

 

Remove header for CSV format

The new 'Remove Header' parameter in the CSV file format in Export Manager and outbound integration endpoint streamlines data transfer and avoids manual manipulation of files for downstream systems that cannot consume a header. When a user exports data using CSV file format with the Remove Header parameter selected, the mapped data is exported and separated by the delimiter, but the headers are not exported. By default, the Remove Header option is not selected.

For more information, refer to the CSV Format topic in the Data Exchange documentation here.

HTTP, HTTPS, and SOCKS proxy support available for asset push sidecars

The asset push sidecar client can now communicate with the STEP server via HTTP, HTTPS, and SOCKS proxy servers with optional authentication.

For more information, refer to the Installing an Asset Push Sidecar topic in the Digital Assets documentation here.

Automatic cleanup of files for inbound integration endpoints

Typically, there is little need to retain inbound files that are processed successfully by an inbound integration endpoint (IIEP), particularly since high volumes of files can take up a lot of space on the server. Retaining files that failed, however, is beneficial since it allows time to troubleshoot and process the file again. Previously, additional maintenance effort was required to manage files delivered for processing via the hotfolder and REST receivers.

Now, for the following IIEP receiver methods, new parameters allow automatic deletion of files in the Save and the Failed folders with separate retention settings:

  • Hotfolder Receiver

  • Hotfolder using file sequence

  • Hotfolder using meta files

  • REST Receiver

The new automatic cleanup functionality in the Choose Receiver wizard step includes settings for:

  • The Save folder - the 'Number of files to keep in save' parameter can save up to 1,000 files and the 'Time to keep files in save (in days)' parameter can save files for up to 365 days.

  • The Failed folder - the 'Number of files to keep in failed' parameter can save up to 1,000 files.

The maximum settings are automatically applied to every IIEP upon upgrade to 11.1. Upon upgrade to 11.1, only 1,000 IIEP import files, and none older than 365 days, are retained. All other IIEP files are deleted. Before performing the 11.1 upgrade, review the IIEP files and store any files to be kept in a location outside of the defined Save and Failed folders.

For more information, refer to the Inbound Integration Endpoints topic in the Data Exchange documentation here.

New automated error handling and improved notification options on IEPs and EPs

Inbound and outbound integration endpoints (IEPs) and event processors (EPs) include updates to reduce the overall downtime. The improvements include:

  • Reduced manual intervention required to restart processing after connection errors via the new 'Retry Connection' parameter.

  • Reduced manual intervention required to restart processing after concurrency errors via 'Retry Concurrency Errors' parameter.

  • Notification of processing error and warning alerts sent to the appropriate users via expanded email notification options.

  • Eliminated process failures related to deleted objects via the new 'Skip Deleted Nodes' option as well as embedded deletion handling.

Automated retries for connection errors

When configured, connection errors (not authentication issues) that are encountered in inbound and outbound integration endpoints (IEPs) and event processors (EPs) can be automatically retried.

The new Connection Error Handling parameters are included on IEPs (shown below) and EPs (shown in the following section) that allow for an external connection. When the external system is unavailable, the IEP and EP can now initiate automated reconnection attempts. Connection error handling can be used when credentials are successfully authenticated but delivery fails on HTTP-based receiver and delivery options.

Set the Retry Connection parameter to 'Yes' and set the Retry Duration to define how long each retry process should continue while the endpoint or event processor is in a 'Failed (retrying)' state. Wait times are increased between each retry to avoid degraded performance, improve the likelihood that the issue can be resolved within the 'Retry Duration' time frame, and allow other processes to run as the system has available resources. Once the 'Retry Duration' has expired, if the issue continues, the IEP or EP fails, and email alerts are distributed based on the error reporter settings.

Upon upgrade, review existing endpoints and event processors with the following HTTP-based delivery or receiver options to determine the suitability of enabling the Retry Connection option: Amazon SQS, Cloud Blob Storage, Elastic Search, Git Delivery, Kafka, REST, REST Direct, and SFTP. (SFTP and Git Delivery connection retries fail immediately if connections are interrupted mid-transfer or if there are slow server response times, since these types of issues cannot be distinguished from authentication errors.)

  • For all new and existing IEPs, connection error handling is disabled by default so that administrators can decide when and where to apply this feature based on the selected delivery or receiver support for automated connection retries.

  • For all new and existing EPs, connection error handling is enabled by default with a 30 day Retry Duration to support backwards compatibility with some processors that already included built in connection retry mechanisms. Review and change the Retry Duration to suit each configured EP.

  • Entries in the sharedconfig.properties file that control connection retries (such as those for the Email, REST Direct, and JMS delivery methods) continue to be effective and are applied prior to the new 'Retry Connection' parameter settings.

Automated retries for concurrency errors

The new Concurrency Error Handling parameters are valid for EPs and allow for automatic retries of transactions that encounter optimistic locking errors. For example, concurrency errors can occur when an event processor attempts to commit a transaction, but the object is already being modified by another user or process. These typically occur when there is a high level of activity on the system and/or long-running business actions.

Set the Retry Concurrency Errors parameter to 'Yes' and set the Retry Duration to define how long each retry process should continue while the endpoint or event processor is in a 'Failed (retrying)' state. Wait times are increased between each retry to avoid degraded performance, improve the likelihood that the issue can be resolved within the 'Retry Duration' time frame, and allow other processes to run as the system has available resources. Once the 'Retry Duration' has expired, if the issue continues, the IEP or EP fails, and email alerts are distributed based on the error reporter settings.

Since enabling concurrency error handing can result in reduced throughput and a perceived decrease in performance, it is recommended to first investigate reported concurrency issues from the 'Optimistic Lock Recovery' health check in Performance Analysis Tools (defined in the Performance Analysis topic here) to determine if business rule logic can be improved to eliminate the errors.

For all new and existing EPs, concurrency error handling is disabled by default so that administrators can decide when and where to apply this feature. As concurrency error handling does introduce wait times when enabled, if concurrency issues occur frequently, performance is impacted.

Expanded email notification options for processing errors and warnings

Error handling for both inbound and outbound integration endpoints (IEPs) and event processors (EPs) is simplified with improved email notification tools as defined below.

  • Admins can now specify separate email addresses to send error alerts based on the type of error encountered when the 'Send Error Report' option is selected for the 'Error Handling & Reporting' step in the configuration. This means email notifications can be distributed to two groups of users: those monitoring fatal processing errors and those who can address data warnings and/or errors that do not cause the process to stop. Additionally, a new parameter allows notifications to be generated for errors only, or to alert on errors and warnings.

    Multiple users can be indicated for each group by separating the email addresses with a semicolon. Also, consider using email groups created on the email system to handle address validation outside of STEP, which also allows for changes in the list of group members without a corresponding edit to the configuration.

    For EPs and inbound IEPs, the error reporter can be configured in the wizard.

    For outbound IEPs, the error reporter can be configured from the editor.

    Upon upgrade, IEPs and EPs with the 'Send Error Report' option configured are updated so that all types of alerts are emailed to the specified existing email address(es). IEPs and EPs without error reporting configured are unchanged.

  • Users can now identify the system that is reporting errors via the email subject line. This improves troubleshooting efforts since the system name can help the email recipient(s) prioritize the urgency of the error. For example, the 'ddc-dev' system highlighted in the following image identifies that a failure happened on a lower environment, namely a dev system, not on a production system. Additionally, the Execution Report now indicates the email address(es) used when sending error reports.

Improved handling for deleted objects in EPs

Event processors (EPs) with the 'Select Processor' parameter on the 'Configure Event Processor' step set to 'Execute Business Action' or 'Execute Business Action for Event Batch' have a new 'Skip Deleted Nodes' parameter. This eliminates the need to handle deleted objects via JavaScript. All other EPs automatically skip deleted nodes, which means that objects in the recycle bin are automatically skipped and do not cause a failure.

The 'Execute Business Action' event processor wizard displays the new parameter in the Configure Processing Plugin step:

The 'Execute Business Action for Event Batch' event processor wizard also displays the new parameter in the Configure Processing Plugin step:

For all new and existing EPs that run business actions, the default 'Skip Deleted Nodes' parameter setting is 'No' to allow the admin to decide when to implement the new functionality. This also maintains the previous functionality where deleted object handling is addressed via JavaScript or where the business action EP fails when attempting to process events for deleted objects. Manually update the parameter to implement the new functionality.

Additionally, all new and existing EPs using the Data Sufficiency Calculator processor no longer fail with an error when encountering an object that is in the recycle bin. Now, the event is automatically skipped, allowing calculations to continue uninterrupted.

Refer to the following topics for more information on error handling and reporting improvements:

  • OIEP - Event-Based - Manual Configuration (here), OIEP - Configuration Section (here), and the individual delivery method topics linked from the OIEP - Delivery Method Section topic (here), all in the Data Exchange documentation

  • IIEP - Error Handling & Reporting (here) or the individual receiver topics linked from the IIEP - Choose Receiver topic (here), both in the Data Exchange documentation

  • EP - Error Handling and Reporting (here) in the System Setup documentation

Refer to the following topics in the System Setup documentation for more information on the 'Skip Deleted Nodes' parameter:

  • Execute Business Action Processing Plugin Parameters and Triggers topic (here)

  • Execute Business Action for Event Batch Processing Plugin Parameters and Triggers topic (here)

OIEP and EP 'Unread events (approximated)' parameter improvement for SaaS systems

Outbound integration endpoints (OIEPs) and event processors (EPs) running on SaaS systems now employ improved methods for estimating events remaining in the queue resulting in a more accurate count of events in the 'Unread events (approximated)' parameter. Previously, for SaaS systems, a count of no more than 1,000 was displayed. With this improvement, admins can make a better estimate of the size of the event queue, consider approximately how long it will take to clear the events (based on experience with processing speed history), and then allow the events to process or decide on another course of action. For example, if an event queue has a large backlog of events that are either no longer relevant or that can easily be recreated, it may be beneficial to purge the event queue, optimize processing or resolve configuration errors, and then republish or recreate the events.

For more information, refer to the Event-Based OIEP Queued Events topic (here) in the Data Exchange documentation and the EP - Event Processor Tab topic (here) in the System Setup documentation.

Updated Business Rule Based Message Processor in OIEP gets nodes within a collection

The Business Rule Based Message Processor in the outbound integration endpoint (OIEP) is updated to get nodes within a collection. When a user configures a collection object in an OIEP, the Business Rule Based Message Processor processes the nodes under the particular collection. Previously, if a collection was configured in the OIEP, the nodes were not processed in the exported file.

For more information, refer to the Outbound Integration Endpoint topic in the Data Exchange documentation here.

Mutual Transport Layer Security (mTLS) now allows specifying a certificate and/or keystore per endpoint

For outbound integration endpoints (OIEPs) using delivery via SSL and REST Direct and for gateway integration endpoints (GIEPs) using delivery via REST, multiple certificates / keystores can be added within the sharedconfig.properties file. Having the ability to select from more than one certificate / keystore is useful when integrating with multiple systems that use different CNames (the STEP system name as defined by another system) or when each integration requires a different certificate.

Selection of the appropriate certificate / keystore is now possible on the OIEP 'Delivery Method' flipper and on the GIEP 'Gateway Configuration' flipper.

For single certificate scenarios, the default setup and 'SSL.Default' properties (shown below) are unchanged. For multiple certificate scenarios, the properties for additional certificates can be set based on the requirements for each delivery method and are identified with a user-defined name for the additional certificates that replace 'Default' in the 'SSL' properties.

The user-defined options displayed above in the 'MTLS Authentication KeyStore' configuration dropdown are determined by the 'Location' properties in the sharedconfig.properties file below. In this example, two additional certificates ('Cert1' and 'Cert2') are applied. The user-defined labels align with the file names of the keystores / certificates. 

SSL.Default.KeyStore.Password=password
SSL.Default.KeyStore.Location=/shared/customer-config/keystore/keystore.jks
			
SSL.Cert1.KeyStore.Location==/shared/customer-config/keystore/Cert1.jks
SSL.Cert1.KeyStore.Password=password
SSL.Cert2.KeyStore.Location==/shared/customer-config/keystore/Cert2.jks
SSL.Cert2.KeyStore.Password=password

RestDirect.Mtls.CertificateKeyStores=,Cert1,Cert2
RestDirectDeliveryURL=1=https://DeliveryUrl1.com,2=https://DeliveryUrl2.com 

RestGateway.Mtls.CertificateKeyStores=,Cert1,Cert2
RESTGateway.ServerURL=1=https://DeliveryUrl1.com,2=https://DeliveryUrl2.com

Administrators can apply keystores / certificates for on premise systems. Activating new or changed keystores / certificates on the application server requires a system restart. For SaaS systems, contact Stibo Systems Technical Support to apply keystores / certificates.

For more information, refer to the Mutual Transport Layer Security topic in the Data Exchange documentation here.

Support for the latest industry standard formats

Updates include support for:

  • ECLASS Basic 13

  • ETIM 9

  • GS1 GDSN 3.1.20

  • UNSPSC 24 and 25

For more information, refer to the ECLASS Format (here), ETIM Format (here), and UNSPSC Format (here) topics in the Data Exchange section of the documentation.

For more information on GS1 GDSN, refer to the GDSN Receiver Solution Enablement in the Solution Enablement documentation here.

ECLASS Advanced Editor Screen updates

ECLASS Advanced Editor Screen is a new screen in the Web UI that improves upon and supersedes the existing ECLASS Advanced Screen, which will be deprecated and withdrawn in a future release. In the past, users could only view the ECLASS Advanced data from the Web UI. But now, it is possible to not only maintain products already linked to the ECLASS Advanced hierarchy, but also link existing or new products to one more version of the ECLASS Advanced hierarchy. The ECLASS Advanced Editor Screen will allow adding aspects and blocks to the product supporting polymorphism and cardinality. You must first have an ECLASS Advanced commercial license enabled to access the functionality.

For more information, refer to the ECLASS Advanced Editor Screen topic here.

IEP Transactional Setting 'Chain' option is planned to be deprecated

For inbound integration endpoints and for Select Object outbound integration endpoints, 'transactional settings' specify how background processes will be generated, how the messages will be processed, and what will happen upon failure. The legacy 'Chain' option is planned to be deprecated in a future release since it generally has slower processing speeds compared to using the 'Strict' option with similar features for enforcing the order of import files. For more information, refer to the Integration Endpoint Transactional Settings topic in the Data Exchange documentation here.