Data Exchange Enhancements and Changes

Summary

The following updates have been made within the Data Exchange functionality:

  • Support for MongoDB 4.2 and MongoDB 4.4 with a newly installed delivery method.
  • Support for eCl@ss Basic version 11.1.
  • Support for the latest versions of UNSPSC.
  • Support added for ETIM version 8.
  • Users can now export / import image conversion configurations and asset push configurations using the STEPXML format.
  • Standard mandatory settings can now be respected during Excel Smartsheet exports.
  • The file name template parameter for several delivery methods has been added or updated to default with a readable date / time stamp and to include the file name template text in the background process name.
  • Large Excel sheet import improvements.
  • New Kafka Receiver and Kafka Delivery Method options added to endpoint integrations as part of out-of-the-box functionality available with the platform.
  • STEPXML changes include new ReplacementRules and a new element added to the XSD.
  • Chaining several post processors for OIEPs can be done with the new Chained Post Processor.
  • Imports executed via inbound integration endpoints (IIEPs) or the Import Manager now enable admin users to set the warning severity for objects rejected on import based on a business condition.
  • eCatalog customers who use the supported BMEcat formats now have the ability to configure exports of commercial terms through the Export Manager and the outbound integration endpoint (OIEP) editor.
  • Appropriately configured outbound and inbound integration endpoints (OIEPs and IIEPs) that fail with errors will now auto-send an email regardless of the error report attachment's size.

Details

New support for MongoDB 4.2, MongoDB 4.4, and driver 4.1

The MongoDB Java driver used by Stibo Systems Enterprise Platform (STEP) has been upgraded to version 4.1, which adds support for MongoDB versions 4.2 and 4.4. Before this release, STEP used MongoDB driver 3.8.12, which supported MongoDB versions through 4.0; versions through MongoDB 4.0 are still supported.

A new delivery method, ‘Mongo Delivery (from release 4.2),’ has been introduced for MongoDB versions 4.2 and 4.4 in the Edit Delivery Configuration wizard.

The extension API for accessing the MongoDB has been extended with support for the new Mongo driver API introduced in MongoDB driver 4.1. The new extended support can only be used for the new MongoDB delivery method 'Mongo Delivery (from release 4.2),' while the MongoDB delivery method option introduced prior to 10.1 will use the MongoDB driver API defined for Mongo DB driver 3.8.12. For detailed information, refer to the STEP API Documentation, which is accessible from the Start Page.

For more information on the MongoDB delivery method, refer to the Mongo Delivery Method topic in the Data Exchange documentation here.

MongoDB functionality is accessible with MongoDB Adapter commercial license activation.

Support added for latest version of eCl@ss Basic

To continue our commitment to serve the needs of users that utilize eCl@ss as a classification system tool for their products and services, STEP now supports eCl@ss Basic version 11.1. As with eCl@ss Basic's last release (version 11), version 11.1 will continue to be published in XML format 3.0.

For more information on the eCl@ss format, refer to the eCl@ss Format topic in the Data Exchange documentation here.

To access eCl@ss, the eCl@ss Hierarchy Import commercial license activation is needed.

Industry standard updates for UNSPSC

STEP now supports UNSPSC versions 22 and 23. With this update, UNSPSC users can import and have access to the most efficient, accurate classification system UNSPSC has to offer. For more detailed information, refer to the UNSPSC Format topic in the Data Formats documentation here.

To access UNSPSC, the UNSPSC Hierarchy Import commercial license needs to be enabled on your system.

Support added for latest version of ETIM format

Continuing our service to customers that use the ETIM classification model to share and exchange product data, STEP now supports ETIM version 8. With this update, ETIM users will have the latest version of ETIM, allowing them to comply with the latest industry standards.

For more information on the ETIM format, refer to the ETIM Format topic in the Data Exchange documentation here.

ETIM functionality is accessible with ETIM Hierarchy Import commercial license activation.

Exporting and importing image conversion configurations and asset push configurations

Users can now export / import image conversion configurations and asset push configurations. This new feature is useful when migrating these configurations across multiple environments. Within the Export Manager, there are two new parameters that allow users to export asset push configurations and image conversion configurations when STEPXML has been selected as the format type.

For more information, refer to the STEPXML Outbound Parameters topic in the STEPXML Format documentation here.

Two new STEPXML tags, <ImageConversionConfigurations> and <AssetPushConfigurations>, will be used to define and contain the values for these configurations within the STEPXML format.

For more information on STEPXML tags, refer to the STEPXML Tags and Examples topic in the STEPXML format documentation here.

In addition to the ability to export and import image conversion configurations, users can now either set the image conversion configuration object type to create auto-IDs for image conversion configurations or manually input an ID for image conversion configuration within the 'Create Image Conversion Configuration' wizard.

For more information, refer to the Image Conversion Configuration topic in the Digital Assets documentation here.

Smartsheets respect the standard mandatory attribute settings

Previously, to indicate a value was mandatory in a Smartsheet, an administrator either identified a metadata attribute that stored the setting, mapped the setting in the export configuration, or used the workflow State Editor. These actions were necessary because the standard methods for mandatory (setting on the attribute itself, or on an attribute reference / link) were not supported by Smartsheets. Now, the new 'Use attribute / reference mandatory setting' parameter on the Excel Smartsheet export format can take the standard mandatory settings for all mapped attributes and references / links. To provide backward compatibility, this new parameter is set to 'No' by default. When the parameter is set to 'Yes,' reduced setup is necessary to ensure that an attribute or reference / link value is required when validating a Smartsheet.

Additionally, the metadata attribute (identified by the 'Metadata mandatory attribute' parameter), the mapping option in the export, and the workflow mandatory settings continue to work without any change in existing functionality.

For more information, refer to the Mandatory Attributes and References in Smartsheets topic in the Data Exchange documentation here.

Smartsheet functionality is accessed via the activation of the STEP Supplier Item Web UI commercial license or the STEP Smart Sheets commercial license. Contact your account manager or partner manager to begin the process of enabling a license or licenses for your system.

New output default file name template improvements

Previously, the file name template parameter was not available for all export delivery methods which resulted in the same file name being used repeatedly. Using the default file name template available included a unique timestamp but was not easily recognizable as a date. Now, when creating a new configuration with an export delivery method that includes the file name template parameter, by default the timestamp variable includes a human-readable date / time. Additionally, the text generated by the 'File name template' is included in the background process name, allowing users to easily match output with the background process that generated the file. Existing saved configurations that used the default file name template also now automatically include the new date / time stamp and potentially different text dependent on the format selected. This constitutes a change in functionality. If a downstream system that consumes the exported data requires the previous default file name, manual updates must be made to the Export Manager configuration or the OIEP configuration to overwrite the new default and restore the expected file name.

This new default creates a unique file name such as 'excel-2020-07-31_12.24.45.xlsx' where the date and time for the output file are obvious.

Modifying the template to include descriptive static text like 'UpdatesRequired' (shown below) provides a one-to-one relationship between the exported output and the background process that generated it.

Additionally, the default file name template is now available for email, file, FTP, or SFTP in the structured translation wizard.

For more information about existing configurations, refer to the Maintaining a Saved Export Configuration topic in the Data Exchange documentation here. For details about exporting from Web UI, refer to the Custom Export Action topic (here) and the Export Action section of the Action Buttons topic (here) in the Web User Interfaces. For information about exporting translations, refer to the Structured Translation topic in the Translations documentation here.

Large Excel sheet import improvements

Import processing of large Excel sheets (.xlsx files) via the Import Manager is now a better experience for end users. This performance improvement is limited to '.xlsx' files and does not impact Excel sheets saved in an older binary format.

For more information, refer to the Import Manager - Select Format topic of the Import Manager documentation here.

New out-of-the-box Kafka inbound and outbound connectivity options

Customers that integrate STEP with Apache Kafka, an open-source distributed event-streaming data platform, can now take advantage of built-in options for both inbound processing from and outbound delivery to a Kafka topic allowing for the exchange of data with upstream systems (i.e., ERP and CRM) and downstream systems (i.e., web, sales points, and distribution).

With this release, a new Kafka Receiver option is available when configuring an inbound integration endpoint.

When configuring an outbound integration endpoint, a Kafka Delivery Method is now available.

Considerations and Limitations

  • Customers who use Audit Messaging Framework and want to use the same Kafka broker(s) for inbound / outbound connectivity will need to make sure they are on the same version of Kafka that is being used for the connector before using the new receiver and delivery methods. Information regarding supported versions can be found in Online Help.
  • Kafka imposes a default configured maximum of 1 MB message size. To overcome this limitation, users can:
  • Configure an endpoint post processor to split up STEPXML messages to only events for a single node per message using the new 'Generic STEPXML Splitter' option.
  • Use the 'Flattened' Split mode option to further reduce the size of the individual event messages.
  • Configure compression of the event content as well using LZ4 as a compression format.

Before using this functionality, it is important that users familiarize themselves with the prerequisites described in the Kafka Delivery Method topic in the Outbound Integration Endpoints section (here) and Kafka Receiver topic in the Inbound Integration Endpoints section (here) of the Data Exchange documentation.

STEPXML import improvements

The STEPXML import functionality has been improved with new options for updating the STEP configuration.

Two new replacement rules options for STEPXML imports

Two new replacement rules options for STEPXML imports have been added:

  • ReplacementRules > LinkTypes > ReplaceAttributeGroupLinks allows replacement of link type (reference type and classification product link type) to attribute group links.
  • ReplacementRules > LinkTypes > ReplaceAttributeLinks allows replacement of links between link types (reference types and classification product link types) and attributes, i.e., the links that make an attribute valid for a link type.

For information about the new tags, refer to the ReplacementRules Tag in STEPXML topic in the Data Formats section of the Data Exchange documentation here.

New option for deleting attributes via STEPXML import

A new 'DeleteAttribute' element valid under the existing 'DeleteAttributes' element has been added. The element has a required 'ID' attribute and an optional 'Force' attribute that when set to 'true' allows deleting attributes with values.

For more information, refer to the STEPXML XSD link to download the PIM.XSD in the STEP API documentation available at [system]/sdk or accessible from the system Start Page.

Updated ability to move Setup Groups via import

Setup groups can now be moved via a STEPXML import both by importing nested setup group structures and by using the existing ParentID attribute.

Chaining post processors for Outbound Integration Endpoints

A new post processor (Chained Post Processor) has been created, which allows the user to combine several other post processors, like the XSLT and the ContextSplitter post processors.

Import rejection message severity on imports now configurable

Imports executed via inbound integration endpoints (IIEPs) or the Import Manager now enable admin users to set the warning severity for objects rejected on import based on a business condition. Previously, a business condition set to prevent import of specific objects would display in the Execution Report of the Background Process (BGP) as an error, and the BGP would finish with 'Completed with Errors.' Because rejection of imported objects based on a configured business condition is, in most cases, the desired scenario, this update ensures the system does not incorrectly assign a 'Completed with Errors' status to an error-free BGP. By enabling configurability on the warning severity of blocked import objects, a clearer, more accurate process resolution status and report is now accessible for all users.

On the 'Select Business Rules' step in either the Import Manager or Inbound Integration Endpoint wizards, there is a new column called 'Condition Handling Options.' In this column, there is a dropdown of options from which the user may select for each added business condition.

Pictured below is the new import rejection severity level option dropdown for the Import Manager wizard:

The option displayed, 'Reject and Include Error,' is the default setting. When selected, rejected import objects will be listed on the execution report as errors, and the import's BGP may finish with a status of 'Completed with Errors.' This mirrors how business condition import errors were handled prior to this update.

Descriptions of the other options can be found in online help.

For more information on importing via the Import Manager, refer to the Import Manager topic in the Data Exchange documentation here. For more information on importing with inbound integration endpoints, refer to the Inbound Integration Endpoints topic, also in the Data Exchange documentation here.

New support for commercial terms when mapping a BMEcat export in OIEPs and Export Manager

eCatalog customers who use the supported BMEcat formats (BMEcat and BMEcat 2005) now have the ability to configure exports of price lists and commercial terms through the Export Manager and the outbound integration endpoint (OIEP) editor. As shown in the screenshot below, users have access to the configured commercial terms hierarchy when mapping a BMEcat export.

Previously, price lists and commercial terms could not be mapped for eCatalog exports using the BMEcat format.

To access BMEcat and BMEcat 2005, the BMEcat commercial license must be activated for your system.

For more information on the BMEcat format, refer to the BMEcat Format topic (here) and BMEcat 2005 Format topic (here) in the Data Exchange documentation.

Notification email sent without error report attachment when report is too large

Appropriately configured outbound and inbound integration endpoints (OIEPs and IIEPs) that fail with errors will now auto-send an email regardless of the error report attachment's size: if the error report is too large, it will not be attached; if the error report is within the configured size limits, the email will be sent with the report attached. The file type of the attached report will be an .html file. If the .html file is larger than the configured maximum, the file will be zipped and attached as a .zip file. If the .zip file is larger than the configured maximum, the email will be sent with no attachment.

Previously, if integration endpoints ended with a fatal exception and the error report was larger than 10 megabytes (MB), no email would be sent at all. Now, when the error report is larger than the configured maximum, the email that is sent does not include the attachment, but notifies the recipient that the error report was not attached because it was too large.

Additionally, admin users who want to change the maximum file size for error report attachments can now add a new property and set the desired file size maximum in MB. This can be done by adding the property shown below to the sharedconfig.properties file. The number shown below, 10, is the default maximum allowed email attachment file size in MB. Admins can set this number to whatever maximum best fits their requirements.

Integration.Endpoint.ErrorFileSizeLimit=10

This update helps provide data stewards with the information they need to ensure endpoints are operating at maximal performance standards, while also enabling admin users to customize email attachment standards that best align with their system parameters.

For more information on configuring error reports on inbound integration endpoints, refer to the IIEP - Configure Error Reporter topic in the Data Exchange documentation here. For more information on configuring error reports on outbound integration endpoints, refer to the OIEP - Configuration Flipper topic in the Data Exchange documentation here.