Data Exchange Enhancements and Changes

Summary

The following enhancements to STEP's data exchange functionality have been made as part of the 2023.4 release. These updates are outlined below and described in the Details section that follows:

  • The Kafka Receiver and Kafka Delivery methods have been upgraded to work with Kafka 3.5.1.

  • Configuration management is improved with inbound integration for change packages exported to VCS (version control systems) which now allows files to be combined to import a change package object in a target system, including user-defined installation instructions, custom attributes, and modifications made outside of STEP. This improves management of project-based initiatives running various configurations simultaneously across development, test, acceptance, and production (DTAP) environments.

  • Gateway integration endpoints now allow proxy support with Amazon S3 Blob Storage. This is used by on-premises systems where the AWS S3 Storage is hosted externally.

  • Support added for the latest industry standards, including ECLASS 14 (Basic and Advanced) and UNSPSC version 26.

  • A new feature in ECLASS Advanced version 14 allows for the sorting of cardinality blocks.

  • The Classification-Specific Attribute Value Group Component that is used for viewing and editing ETIM data now allows to define the display order of ETIM attributes.

  • The Industry Standard Mapper, integrated with Data Onboarding, automates the interpretation of Industry Standard Mapping files, creating Mapper Configuration setup entities. This enables seamless ECLASS version mapping, offering both automated and manual onboarding capabilities, and streamlining standard upgrades within STEP.

  • Receiver and delivery methods in STEP that are related to the IBM WebSphere solution are being deprecated and customers should transition to using Dynamic JMS Receiver and Dynamic JMS Delivery as replacements.

  • Event Mode options for outbound integration endpoints and event processors have been renamed to 'Standard' and 'Deduplicate' to better describe the functionality.

  • Authentication and encryption are strengthened for the PDX integration to allow the OAuth 2.0 token-based authorization protocol in both inbound and outbound integration endpoints. Additionally, the 'Product Data Exchange 2' delivery method now allows excluding asset binary content, while still sending asset metadata.

Details

Kafka integrations upgraded

The Kafka Receiver available in inbound integration endpoints (IIEPs) and the Kafka Delivery method available in the outbound integration endpoints (OIEPs) are upgraded to use Kafka 3.5.1.

Both the receiver and the delivery method enable a STEP platform integrated with Apache Kafka to use a Kafka queue. For more information, refer to the Considerations for Setting Up Kafka Receiver or the Considerations for Setting Up Kafka Delivery, both in the Data Exchange documentation.

Configuration management improvements for change packages and VCS repositories

Using change packages with an external version control system (VCS, like Git) in a development, test, acceptance, or production (DTAP) environment, now enables better management of systems in the DTAP environment by allowing external modification and impact analysis before installation of the change package versus directly updating objects in a target system via STEPXML.

Now, users can construct a change package containing all the changes previously exported to a VCS. This provides a better configuration management user experience by allowing users to review the impact of an update on the target system by running an impact analysis before it is installed. Improvements in this release include:

  • The new STEPXML Joiner for Change Packages pre-processor collects and validates the required files (STEPXML, .js, and .json) from a .ZIP file sent to an inbound integration endpoint (IIEP) from a VCS or other location, and creates the change package version of STEPXML for import, where changes are not applied until it is installed. This pre-processor provides data to the 'Primary Items' and 'Secondary Items' flippers in the imported change package, as well as the custom change package attributes and instructions optionally included by the user who created the change package on the source system. Data for the 'Items Required for Transfer' and 'Possibly Impacted Items' flippers are not exported to the VCS and are not included when the change package is created in a target system.

  • The Git Delivery method for outbound integration endpoints (OIEPs) now includes a 'Git Access Method' parameter with radio buttons for HTTPS and SSH. To reduce user error during setup, based on radio button selection, only the applicable parameters are activated.

  • To reduce troubleshooting time for both IIEPs and OIEPs, error messages have been improved to include detailed information related to the specific problem.

  • To improve usability on the Output Templates section when configuring an OIEP for use with change packages, the Object-Eventtype 'Verified package contents' option has been renamed to 'Change Package (Sealed)'. The newly named option can now be found using the Browse or Search tabs on the 'Conditions for output template' dialog.

For more information, refer to the Version Control System Integration topic or the VCSI: Change Package Git Delivery Method in OIEP topic, both in the Configuration Management documentation.

Gateway integration endpoints allow proxy support for Amazon S3 Blob Storage

In some scenarios, particularly on-premises systems where the AWS S3 Storage is hosted externally, a proxy server is used to restrict internet connectivity from STEP. Now, gateway integration endpoints (GIEPs) using the Amazon S3 Blob Storage option support proxy configuration with the new 'Proxy config' parameter.

For more information, refer to the HTTP Proxy Configurations topic or the Configuring a Gateway Integration Endpoint - Amazon S3 Blob Storage topic, both in the Data Exchange documentation.

Support for the latest industry standard formats

Updates include support for:

  • ECLASS 14: STEP now supports ECLASS version 14, both Basic and Advanced. For more information, refer to the ECLASS Format topic in the Data Exchange documentation.

  • UNSPSC 26: STEP now supports UNSPSC version 26. With this update, UNSPSC users can import and have access to the most efficient, accurate classification system UNSPSC has to offer. For more information, refer to the UNSPSC Format topic in the Data Exchange documentation.

Added feature for sorting of cardinality blocks in ECLASS Advanced version 14

Enhanced support for ECLASS version 14 (Basic and Advanced) with a new feature facilitating the sorting of cardinality blocks. The system now facilitates sorting based on the FORDER approach.

In support of this solution, the following implementations have been made:

Running the ECLASS ADVANCED Model easy setup action creates a description attribute, FORDER (ID = eClassAdv_FORDER), valid for the 'eClass Advanced Product Block' object type. The importer populates the sequence number of the block within this attribute. Having this attribute as part of the model facilitates to import and maintain the sequence of the cardinality blocks.

Furthermore, the BMECat exporter has been updated to incorporate the FORDER value within the <FORDER> element in the exported file.

It is strongly recommended to initiate the import process by importing ECLASS Advanced files (Unit and Dictionary) first, followed by the import of ECLASS Basic files. This sequence ensures proper data handling and adherence to the system's requirements.

Improved display order of ETIM attributes in Web UI

This enhancement is for those using the Classification-Specific Attribute Value Group Component to view classification-specific attribute values in Web UI. Users using this component (which is generally used for viewing and editing ETIM data) can now define the attribute display order using the newly introduced parameter: 'Classification Link Display Sequence Attribute.' The parameter is to be configured with a metadata attribute that determines the display order of ETIM attributes. This facilitates the ETIM users to view the ETIM attributes in accordance with the ETIM specifications.

For more information, refer to the Classification-Specific Attribute Value Components topic in the Web User Interfaces documentation.

New Industry Standard Mapper

Industry standards evolve through periodic releases, introducing updates to their reference files. The new Industry Standard Mapper solution addresses challenges associated with those evolving industry standards. STEP now supports seamless mapping between different versions of a standard, allowing users to automate reclassification and data transfer tasks. This enhancement ensures an automated approach to handling standard upgrades.

Every time there is a version upgrade, many industry standards provide a description file that contains an instruction on how to upgrade from one of their versions to their next major version, which Stibo Systems refers to as Industry Standard Mapping files. The new solution interprets those Industry Standard Mapping files to create Mapper Configuration setup entities with necessary mapping instructions. These created Mapper Configuration setup entities later facilitate onboarding of data from one version to another version.

Currently the new Industry Standard Mapper solution supports only the ECLASS BASIC. For optimal functionality of this solution, it is required that the system utilizes the version-specific object type structure for ECLASS classifications. This can be achieved by selecting the 'Version specific object types' option on the Advanced step of the ECLASS Classification Importer during an import.

Additionally, the Data Onboarding solution has been enhanced to read the Industry Standard Mapping files and automatically generate Mapper Configuration setup entities, minimizing manual efforts. Users can now efficiently map between ECLASS Basic versions, benefiting from both automated and manual mapping capabilities.

The Industry Standard Mapper solution works in conjunction with the existing Data Onboarding solution in STEP. Implementation of the Industry Standard Mapper is dependent upon the prior implementation of the Data Onboarding solution.

In support of this solution, the following components have been newly implemented:

  • A component model called 'Industry Standards Mapping Model' that when run creates all the necessary folders, object types, attributes, LOVs, etc.

  • A processing engine called 'Eclass Mapping File Importer' that when configured in an IIEP will import the Industry Standard Mapping files into the system.

  • A Web UI screen called 'Industry Standard Mapping Creator' that provides an interface to create Mapper Configuration setup entities.

  • A Web UI component called 'Manual handling Component' which supports handling mapping issues that requires manual handling as deletion and specialization (splits) of classifications.

Documentation outlining how the solution can be configured and deployed is in progress now and will be available for the 2024.1 release.

Explore further by clicking the video below. If it does not play as expected, it is also available in the Customer / Partner Communities and may also be accessible within Stibo Systems Service Portal.

 

IBM WebSphere deprecations and replacements

Being deprecated is everything that uses IBM WebSphere MQ directly from STEP. Specifically, that affects the following areas:

  • IBM MQ SSL Receiver and IBM MQ SSL Delivery Method

  • The WMQInitialContextFactory and FileInitialContextFactory JMS Connection Factory Class within the JMS Receiver and JMS Delivery Method

The ActiveMQInitialConnectionFactory JMS Connection Factory Class will remain available within the JMS Receiver and JMS Delivery Method. Customers that use any of the above for integration with IBM WebSphere MQ should use the Dynamic JMS Receiver and Dynamic JMS Delivery Method as replacements.

This information is also outlined in the 2023.4 Upgrade Guide.

Event Mode options are renamed for clarity

The Event Mode parameter options on outbound integration endpoints (OIEPs) and event processors (EPs) are renamed to better indicate the functionality being selected.

  • The 'Detailed' option is now renamed to 'Standard.' This option continues to handle each event as it is presented by the queue to the processor.

  • The 'Efficient' option is now renamed to 'Deduplicate.' It continues to perform the same action, namely, deduplicating events on the queue at run time.

For more information on the Event Mode parameter, refer to the EP - Event Processor Tab topic in the System Setup documentation and the OIEP - Event-Based - Manual Configuration topic in the Data Exchange documentation.

PDX improved security and asset management

The PDX integration now allows use of the OAuth 2.0 token-based authorization protocol via the 'Product Data Exchange 2' delivery method in outbound integration endpoints (OIEPs) and the 'Product Data Exchange' receiver in inbound integration endpoints (IIEPs). Configuration parameters in both OIEPs and IIEPs now provide users with the option to use either 'Token-based Authentication' (with a business function providing an Auth-Header Value Function) or the existing 'Basic Authentication' (with API user name and password) options.

Additionally, the new 'Upload Assets' parameter on the 'Product Data Exchange 2' delivery method allows asset binary content to be excluded, while still sending asset metadata.

For more information, refer to the Base Setup topic in the Data Integration documentation and the Asset Publisher topic in the Digital Assets documentation.