Data Exchange Enhancements and Changes

Summary

The following updates have been made within the Data Exchange functionality:

  • An out-of-the-box JSON export option is now available for OIEPs and Export Manager giving users a more straightforward way to generate JSON files for object data in STEP.

  • When a Unit ID is available for MongoDB, it will now include information about the Default Unit (DefaultUnitID) in the output.

  • When exporting via Excel, CSV, STEPXML, or Advanced STEPXML, users have the option to include empty fields in the exported data.

  • A new Efficient mode (renamed 'Deduplicate' in release 2023.4) for processing events is available for OIEPs and event processors.

  • The use of multiple partitions for OIEPs using Kafka Delivery Method is now supported, allowing for parallelization of processes and fault tolerance.

  • All AssetPushLocation items will be exported inside an asset item for STEPXML exports.

Details

New Generic JSON out-of-the-box export option

A new Generic JSON export format plugin has been added to outbound integration endpoints (OIEPs) and Export Manager. JSON is a widely used format for data exchange. This new functionality significantly simplifies the work for STEP administrators and enterprise architects as they can now send data in JSON format without requiring scripting, extensions, or other complex configurations.

Existing methods of sending data in JSON format will still be available to support non-standard use cases, but customers should attempt to use the out-of-box format whenever possible to reduce extensions, experience smoother upgrades, and optimize outbound performance.

For more information, refer to the Generic JSON Format topic in the Data Exchange documentation here.

Attribute information for MongoDB now includes DefaultUnitID

When an outbound integration endpoint sends the attribute information by MongoDB delivery, and a Unit ID is available, it will now include information about the Default Unit (DefaultUnitID) in the output in addition to other units linked to the attribute. This helps downstream systems to understand the data and organize it better for user consumption.

For more information on MongoDB, refer to the Mongo Delivery Method topic in the Outbound Integration Endpoints section of the Data Exchange documentation here.

Export empty values available for Excel, CSV, STEPXML, and Advanced STEPXML

Now, users can export empty data (attributes, references, metadata, data container values, etc.) in Excel, CSV, STEPXML, and Advanced STEPXML via a new option on the export manager. When used, the option will include any valid and linked data types for the selected objects so that empty values will be exported. When exporting empty values is chosen for mappable formats (Excel and CSV), empty values will be included in the export regardless of whether the data is mapped using attribute groups or individual attribute selection. When the option to export empty values is set to 'no,' the empty values will be excluded from the export.

When the option to include empty values is selected for STEPXML or Advanced STEPXML, all linked and valid attributes, references, metadata, and data containers are included in the export. For Excel and CSV exports, where empty values are included, and the user maps an attribute group using the 'All Attributes' or 'Select Attribute' options instead of / in addition to mapping individual attributes, all attributes are included in the export from the attribute group(s) for attributes that are valid and linked attributes in the product or classification hierarchies related to the exported objects through inheritance or links. In any case, exports are limited by the view permissions of the exporting user.

For more information, refer to the Data Formats section of the Data Exchange documentation.

New Efficient processing mode for OIEPs and event processors

The new Event Mode parameter can provide more efficient event handling on outbound integration endpoints (OIEPs) and event processors (EPs).

It is advised that customers verify that existing OIEPs and EPs use the most beneficial mode based on the configuration scenarios outlined below:

  • Efficient mode (renamed 'Deduplicate' in release 2023.4) improves performance when an OIEP or EP routinely has a long backlog of events to process with multiple events for the same object in its event queue and/or long running exports and/or business rules and/or it uses small batch sizes and/or when downstream systems have trouble keeping up with message processing. Efficient mode requires processing time to analyze and remove redundant events.

  • Detailed mode (renamed 'Standard' in release 2023.4) does not include processing time to analyze and remove redundant events. Event queues where the queue length is routinely short, or all of the objects on the queue are already unique, or the exports and/or business rules are simple and complete quickly, or a large batch size is configured will perform better using Detailed mode.

The Detailed mode is the default for all newly created OIEPs and EPs. For backward compatibility, existing OIEPs and EPs will continue to run in the Detailed mode until changed.

Online help updates for this functionality will be included in documentation released with 11.0-MP1.

For more information on creating an event-based OIEP, refer to the Manual Configuration step of the Creating an Event-Based Outbound Integration Endpoint topic in the Outbound Integration Endpoints section of the Data Exchange documentation here. For more information on modifying OIEPs already created, refer to the OIEP - Event-Based - Manual Configuration topic in the Outbound Integration Endpoints section of the Data Exchange documentation here.

For more information on configuring an EP, refer to the EP - Event Processor Tab topic in the Event Processor section of the System Setup documentation here.

Multiple partitions now supported in OIEPs using the Kafka Delivery Method

Now, STEP supports the use of multiple partitions for OIEPs using the Kafka Delivery Method. This allows for parallelization of processes, in addition to providing fault tolerance.

In order to support the use of multiple partitions, there is an option to send the Node ID as the event key when using Kafka integration. Events and messages with the same ID are put onto the same partition, ensuring correct sequence in processing the messages. The event ID will default to the endpoint ID unless configured otherwise.

This affects OIEPs only. IIEPs are still read from a single partition.

The available options for the event key are ${endpointId}, ${nodeType}, and ${nodeId}.

For more information, refer to the Considerations for Setting up Kafka Receiver (here) or Considerations for Setting up Kafka Delivery topic (here), both in the Data Exchange documentation.

AssetPushLocation items export update

For STEPXML exports, all AssetPushLocation items will be exported inside an Asset item. Previously, STEP would export AssetPushLocation items only for the selected workspace in the Export Configuration.