Data Exchange Enhancements and Changes

Summary

The following enhancements to STEP's data exchange functionality have been made as part of the 11.2 (2023.2) release. These updates are outlined below and described in the Details section that follows:

  • The Import Manager and IIEPs now include an out-of-the-box Generic JSON format for imports. Users can import a JSON file structure including mapping and transformations as needed.

  • Data Sufficiency Messages and Calculation Dates can now be exported in human-readable format in Excel and CSV, allowing these messages and calculation dates usable for external analysis and communication with users outside of STEP.

  • Logging information is now available in the Performance Analysis Tools for EPs and OIEPs running in Efficient Mode to help STEP admins determine how much time is spent analyzing and removing duplicate events. The information is useful in determining which mode is best to use for different use cases.

  • Support for inbound mTLS is now available on SaaS environments. Using mTLS provides another level of security for system protection, ensuring that traffic is secure and trusted in both directions.

  • The option to have Kafka manage the topic offset for Kafka Receiver IIEPs is now available. Now, Kafka will be aware of which messages have been consumed.

  • Collections used in Scheduled Data Export will no longer deliver an empty file.

  • Configured PDX Channels can now be set up by customers, partners, and consultants, eliminating the need for custom development by the PDX team. This speeds up the process and also increases retailers’ business agility with a deployment that is adaptable to their changing needs.

  • The ECLASS Advanced solution has been expanded to include an easy setup, importing and exporting of ECLASS Advanced data in BMEcat 2005.1 format, and JavaScript and workbench actions.

  • The importing and exporting of ECLASS Advanced data in BMEcat 2005.1 format has been added to support the industry standards capabilities.

  • SaaS customers can utilize Stibo Systems GDSN Receiver securely with an AS2 connection, and a new GDSN Receiver version is available.

  • A new Data Onboarding solution simplifies data exchange between different data models or STEP objects.

Refer to the Web UI Enhancements and Changes release note (here) for additional 11.2 (2023.2) information regarding Web UI-specific components / functionality. Also, check the 'Bugfixes' section of the Miscellaneous Bugfixes topic (here) for specific fixes related to Data Exchange.

Details

Generic JSON out-of-the-box import option

In release 11.0, the Generic JSON format was introduced for export via the Export Manager and outbound integration endpoints (OIEPs). Now, the Generic JSON format is supported for imports via the Import Manager and inbound integration endpoints (IIEPs). This new functionality significantly simplifies the work for STEP administrators and enterprise architects as they can now import data in JSON format without requiring scripting, extensions, or other complex configurations. Additionally, having out-of-the-box solutions for Generic JSON import and export helps simplify the integration landscape via the exchange of data in the same format throughout the customer ecosystem and with external data trading partners.

Existing methods of importing data in JSON format are still available to support non-standard use cases, but customers should attempt to use the out-of-box format whenever possible to reduce extensions, experience smoother upgrades, and optimize inbound performance.

The new Generic JSON documentation includes examples of input files, templates, and the resulting new objects in STEP. Refer to the Generic JSON Inbound Processing Instructions topic in the Data Exchange documentation here.

Data Sufficiency error messages now exportable in human-readable format

The exported data of Data Sufficiency messages and calculation dates are now available in human-readable format in Excel and CSV. Sufficiency calculation dates and error messages are stored in an encoded format, which is decoded for display in the Sufficiency Panel in the Web UI. This information is usable for external analysis and communication with users outside of STEP.

This supports users who are correcting data on a product-by-product level in the Web UI, but does not support cases where external management or analysis of the data is desired. For example, to review errors across a set of products in bulk and import data changes accordingly, or to perform further analysis to identify repeating or frequent problem areas.

To address these cases, two new transformations are available in the export mapper that transform the date and time information from epoch to ISO format and one that decodes the sufficiency messages, allowing the data to be extracted in human readable format in both Excel and CSV during export.

For more information, refer to the Sufficiency Configuration Type topic in the Data Governance documentation (here) and Attribute Transformations topic in the System Setup documentation (here).

Logging added to help determine the best Event Mode for EPs and OIEPs

In the Performance Analysis Tool Activity tree, logging has been added for the time spent deduplicating events when an event processor (EP) or outbound integration endpoint (OIEP) is configured with the Efficient Event Mode (renamed 'Deduplicate' in release 2023.4). Using this information, customers can make data-driven decisions regarding which is the best mode to use (Efficient or Detailed (renamed 'Standard' in release 2023.4)), as the decision is highly dependent on the underlying use case and overall expected throughput. In some cases the time it takes to complete the deduplication of events outweighs the potential performance gains in processing, while in other cases the processing itself is the bottleneck. Therefore, being able to compare the time spent deduplicating against the overall run time supports administrators in selecting the setting that will lead to the most optimal performance for their particular scenario.

For more information regarding OIEP setup and configuration, refer to these topics in the Outbound Integration Endpoints section of the Data Exchange documentation:

  • Creating an Event-Based Outbound Integration Endpoint (here)

  • OIEP - Event-Based - Manual Configuration (here)

For more information on configuring an EP, refer to the EP - Event Processor Tab topic in the Event Processor section of the System Setup documentation here.

New SaaS inbound mTLS authentication capabilities

TLS (Transport Layer Security) is a well-established one-way security protocol, and mTLS (Mutual TLS) is the two-way version that requires both parties involved in the communication (server and client) to authenticate each other. To continue to support the latest security and authentication protocols, mTLS for inbound data is now available for SaaS environments via GraphQL, SOAP, RESTv2, and RESTv1.

'Inbound' refers to requests made externally from STEP by another system. The existing options relevant for mTLS authentication are S3 / Azure Cloud Blob Storage and Kafka. Additionally, like the outbound mTLS solution, multiple certificates and/or keystores can be defined and individually selected.

For more information on mTLS, refer to the Mutual Transport Layer Security topic in the Data Exchange documentation here.

New option for Kafka to manage the topic offset for Kafka Receiver IIEPs

When enabled via a new 'Use Kafka to Manage Offset' parameter available for Kafka Receiver inbound integration endpoints (IIEPs), the Kafka offset management mechanism is used instead of the STEP-owned one. This means that STEP commits the offset back to Kafka as soon as the message is read by STEP, although reading the message does not necessarily equate to successfully processing it. However, committing the offset when read makes it easier to manage the overall throughput, and when combined with the retention period setting in Kafka, can determine when messages can be purged in Kafka. Previously, the offset was managed completely in STEP with no commits done to Kafka.

When switching from having STEP manage the offset to having Kafka do it, upon invoking the IIEP, all messages that Kafka has available for a given topic will be re-fetched. This can be prevented on the Kafka side via manual intervention that is described in online help.

For more information, refer to the Kafka Receiver topic (here) and the Kafka Delivery Method topic (here), both in the Data Exchange documentation.

Exclude empty files generated from a collection via the Scheduled Data Export

For the Scheduled Data Export, when the export is based on a collection and the collection is set to refresh at each run, empty collection files will no longer be delivered. If the user is using an empty collection file as a way to monitor the status of the background process, then they need to handle this differently by using external systems to do the monitoring instead.

Previously, when a scheduled export (from Export Manager) was based on a collection, and the collection was set to refresh at each run. If the refreshed collection was empty, a file was still produced but it was empty (i.e., with no content). This was confusing for end users as the typical use case for a collection-driven scheduled export is to export a set of objects needing action, such as all products in a specific workflow state, or all objects with a particular attribute value (such as an upcoming publishing or expiration date). End users receiving an empty file were therefore distracted when no action was needed and/or were confused as to whether the export had failed or if the empty content was accurate. Further, it was inconsistent with the behavior in outbound integration endpoints where an empty file would not be sent as a result of a refreshed empty collection.

Non delivery of the empty files is consistent with other export methods, including outbound integration endpoints (OIEPs). This streamlines the process for end users by preventing confusion, improving productivity, and creating a consistent experience across exports.

For more information, refer to the Scheduling a Data Export topic in the Export Manager documentation here and Monitoring an OIEP via External Systems in the Export Manager documentation here.

New feature to set up PDX Onboarding channels in STEP

A new PDX Onboarding Channel Configurator solution has been introduced in STEP, enabling users to create PDX Onboarding channel configurations. This feature aims to simplify the PDX channel creation and its management in the Web UI, making it possible for more STEP implementations to quickly adopt PDX as the vendor facing product data onboarding solution.

With the new PDX Onboarding Channel Configurator component, which is designed to be standardized and driven by user input, retailers and their customers will benefit from an exclusive channel management solution. Implementing this new feature in the Web UI will allow customers to experience a more straightforward adoption process, faster implementation, and a quicker realization of its value. In order to take advantage of the PDX Onboarding Channel Configurator for PDX channel creation, some prerequisites must be in place: It is recommended that a discovery / data model analysis is completed before implementing this component as adjustments may be required for the setup process.

This new feature will significantly improve the ability to easily create onboarding channels, providing a more efficient solution for implementing and maintaining an onboarding channel within the STEP system.

Using the functionality requires the enablement of the Product Data Exchange Syndication commercial license.

Note: The PDX Onboarding Channel Configurator is compatible with STEP versions 10.3, 11.0, 11.1, and 11.2.

For more information, refer to the PDX Onboarding Channel Configurator topic here.

For more information on the PDX Onboarding Channel Configurator component, click the video below:

 

Expanded ECLASS Advanced industry standard solution

The Product Master Data Management (Product MDM) solution has recently expanded its functionality by including support for the ECLASS Advanced Industry Standard. This means that users can now easily import ECLASS Advanced files such as the ECLASS Unit, ECLASS Dictionary and BMECAT 2005.1 files into the system.

To use this functionality, it is necessary to have the ECLASS ADVANCED commercial license enabled. This functionality has also been backported to eclass-component-model component versions that are compatible with the last maintenance patches for STEP 11.1 and 10.3. Existing customers are advised to reach out to Stibo Systems Support prior to upgrading to newer versions.

The Easy Set-up action can automatically create the ECLASS Advanced data model, Inbound Integration Endpoints that facilitate the import of Unit, Dictionary, and BMEcat data, import widgets, and standard users for importing. This makes the process of importing and exporting ECLASS Advanced data much simpler and more efficient for business users who are responsible for managing product data in the system.

New importing / exporting of BMEcat 2005.1 industry standard data

It is now possible to import and export BMEcat 2005.1 industry standard data in the STEP system. BMEcat 2005.1 is a format designed specifically for the transport of ECLASS Advanced data, and therefore the importer for BMEcat 2005.1 can only be used in conjunction with ECLASS Advanced. To use this functionality, it is necessary to have the ECLASS ADVANCED commercial license enabled.

It is important to note that BMEcat data corresponds to certain ECLASS Advanced version(s) and it is necessary that the corresponding ECLASS Advanced version has been imported before the BMEcat data import takes place.

Additional industry standard updates

Additional industry standard support includes:

  • With 11.2, SaaS customers can utilize our GDSN Receiver securely with an AS2 connection. Open an issue (i.e., ticket) in the Stibo Systems Service Portal if you would like to have this connection set up. SaaS Operations and Technical Services will reach out to get the specific information needed to complete the configuration.

  • The GS1 Maintenance Release 3.1.23 (May 2023) for the STEP GDSN Receiver solution is available. Read the release notes before importing the files. They can be found on the Release Information dashboard of the Stibo Systems Service Portal. This is a Stibo Systems product release for GDSN Receiver. The GS1 maintenance release is on May 20, 2023.

New Data Onboarding solution

Industries have varying data exchange standards, and STEP facilitates the import and export of files in multiple industry standards. However, each standard has a distinct data model for storing and handling data. Organizing products across different standards and data models can be challenging when dealing with imported files from multiple standards.

The Data Onboarding solution offers a valuable tool for simplifying data exchange between different data models or STEP objects, leading to greater efficiency in data maintenance within the STEP system. This solution optimizes the process of importing supplier data and mapping it into standard and proprietary data models. With the Mapper functionality, users can effortlessly move data, such as ID, name, and attribute values, between STEP objects, significantly reducing the configuration process's complexity. Data mapping can be executed via business rules or background processes, providing users with greater flexibility in managing their data.

To optimize the user experience of the Data Onboarding solution, it features several Web UI components such as the Comparison screen, Stacked mapper action, and Onboarding action. Previously, this solution was integrated as an embedded component in the PMDM for Automotive STEP instance, and now it has been enhanced to support all industry standards.

The functionality requires the enablement of the Onboarding Mapper license / component.