The following list includes export elements known to minimize impact on system performance. This list can be used to troubleshoot existing exports, and can also be reviewed prior to creating new exports to prevent performance problems.
To ensure maximum export performance, the usual strategy is to limit the amount of information exported, or limit the number of times the same node is exported with the same data.
For general information on exports, refer to the Data Exchange documentation.
Optimize Object Type Triggering Definitions
Ensure that only required objects are exported using the Triggering Object Types section on the outbound integration endpoint (OIEP) Triggering Definitions tab. For example, when exporting to an e-commerce system, only the product itself is potentially relevant. Triggering on other object types risks spending time exporting objects which would be irrelevant downstream.
For more information, refer to the OIEP - Event-Based - Event Triggering Definitions Tab topic in the Data Exchange documentation.
Optimize Attribute and Reference Triggering Definitions
Applying triggering definitions can ensure that objects are only exported when necessary (based on the attributes changed) and minimize performance issues. On the outbound integration endpoint (OIEP) Event Triggering Definitions tab, set the object types in the Triggering Object Types section. For example, an e-commerce system is probably only interested when certain attributes, like the description, of a product object changes. So the Triggering Object Type is set to 'product' and the Triggering Attributes is set to 'description.'
Additionally, set the OIEP triggering definitions specifically for the export recipient. Create attribute groups specifically for export channels, and then trigger attributes specified by attribute group. For example, the attribute 'WebsiteAttribute' contains only the attribute relevant for the website export channel.
For more information, refer to the OIEP - Event-Based - Event Triggering Definitions Tab in the Data Exchange documentation.
Optimize Event Filter and Event Generator Triggering Definitions
Event filters and event generators are business rules that are executed during export.
For example, consider that inheritance affects changes to attributes residing higher up the hierarchy, and can result in a significant number of products if all child products are exported. In this case, replacing a single high-level attribute change event with multiple child-level events is recommended. Using the event filter / event generator options means the significant volume of events can be handled in multiple batches, which enables output files to be kept to a reasonable size. It also allows exporting multiple batches in parallel reducing overall export times.
Typically, the 'approval' mechanism should not determine if data is suitable for publishing since approval is much earlier within the product lifecycle. Often, it is desirable to limit the data exported to just 'published' objects. The event filter and pre-processor functionality can meet both of these requirements, although they operate in slightly different ways. Alternatively, all data can be exported downstream (related to the underlying events) and a middleware-based solution can determine which data to route to which target system.
For more information, refer to the OIEP - Event-Based - Event Triggering Definitions Tab in the Data Exchange documentation..
Use Multiple Dedicated OIEPs
In many cases, a single OIEP is used to monitor all product changes, i.e., both attribute and reference changes.
In this case, it is easy to understand how the assignment of many products to a classification, e.g., 'Christmas Gifts' can cause a significant volume of data to be exported, delivered, and processed by the receiving system, even though no actual product data has changed.
This issue can be avoided through the use of multiple OIEPs:
- One OIEP listens for traditional product (attribute) changes and outputs all data for the product.
- Another OIEP listens for reference changes and outputs minimal data, e.g., product ID and target ID.
Note: Generally speaking, the performance gain can be degraded if using more than 10 OIEPs.
For more information, refer to the Creating an Event-Based Outbound Integration Endpoint in the Data Exchange documentation.
Use Multithreading
Increasing the number of threads for a given OIEP can increase export performance for event-based OIEPs.
Multithreading is effective when a large amount of data goes to a downstream system on a regular basis and the downstream system can handle the load.
Consider the following points before increasing the number of threads to more than one:
• STEP system hardware requires enough resources to perform multithreading.
• The downstream (receiving) system must be able to handle parallel events.
Since the batch-fetching of the events runs serially, increasing the thread number results in each batch size being divided by the thread number so that the contents of a batch can be processed in parallel. The data is distributed as evenly as possible to each thread. From fetching the batch through the delivery stage, the pre-processing, main processing, and post-processing take place in parallel. When all threads are complete, the batch yields a single message as is consistent with the 'Strict' transactional setting required by event-based OIEPs. This allows events to be delivered as if everything was executed serially.
Thread size is typically increased for critical information where the speed with which the message is produced is key. However, multithreading can have a negative impact on the overall system performance when a large amount of data is involved.
The recommendation is to consider using multithreading to improve production of messages to a downstream system on a regular basis to improve export performance. Run the settings in a test environment, starting with a small amount of data and then increasing it, before implementing it in the production system.
For more information, refer to the Event-Based OIEP Multithreading Support in the Data Exchange documentation.
Note: Setting the 'Maximum Number of Threads' parameter on the Configuration tab has no impact on a Select Object OIEP (which uses a static set of data, not events).
Optimize the Batch Size
The appropriate batch size is typically based on the size of messages and downstream system processing capabilities:
- Use a smaller batch size for larger messages.
- Use a larger batch size for smaller messages.
For individual OIEPs, the batch size is set using the 'Number of events to batch' parameter on the Event Queue Configuration section. For more information, refer to the OIEP - Event-Based - Event Queue Configuration Section in the Data Exchange documentation.
Note: Use the system-wide case-sensitive OutboundExportService.BatchEventsMaxSize property, set in the sharedconfig.properties file, to limit the number of events included within any batch. When the batch size in the workbench is greater than this setting, this setting overrides the workbench to limit data exported in a single export invocation.
Small Batch Size Considerations
A small batch-size, or no batching at all, invokes the export engine and surrounding framework many times (once per batch), incurring a significant overhead. A small batch-size can also cause the same product to be exported multiple times in cases where a product has been approved multiple times in quick succession, or there have been changes to externally maintained data (in this case one event will be created for each external change).
Note: While setting the batch size = 1 may appear to deliver the required results, in practice, there are problems associated with this approach as defined above. Generally, small batch sizes should be avoided.
To publish one STEPXML file per item (product or entity) while using a batch size larger than 1, use the 'Generic XML Splitter' or the 'STEPXML Splitter' post-processor. For more information, refer to the Configure the Pre-processor and Post-processor section of the OIEP - Select Objects - Output Templates Section topic or the OIEP - Event-Based - Output Templates Section topic in the Data Exchange documentation.
Large Batch Size Considerations
A large batch-size (e.g., 1,000 - 10,000 events) invokes the export engine and associated framework fewer times incurring less overhead and reduces the amount of duplicate product data exported.
The database Oracle KODO cache default limit is 10,000. For example, a hierarchy with more than 10,000 children, a product with more than 10,000 references, a list of value with more than 10,000 values, an OIEP with a batch size of more than 10,000 events, etc. Storing very large data relations can have great negative performance impacts, since there is a high likelihood that a large number of the related objects are no longer in the cache.
Recommendations
-
Avoid a batch-size of 1 and if required, use a post-processor to publish separate XML files per item.
-
Avoid a batch-size of more than 10,000. If there is a reason for a larger batch size, contact Stibo Systems Support to discuss raising the Install.DataCache.MaxRelationSize setting to allow for a larger cache size.
Use Cross-Context Exports
One way to limit how much data is being exported, is to export cross-context STEPXML on an endpoint rather than having multiple endpoints, each exporting in their own context. This is because having multiple endpoints also means exporting the non-context sensitive data multiple times.
Do not configure multiple OIEPs each outputting product details for a single context. Instead, for significantly faster overall export time, use a single OIEP configured with multiple contexts. Subsequently, the context splitter post-processor can be used if separate context output files are required.
For more information, refer to the 'Contexts' parameter section on the OIEP - Configuration Section in the Data Exchange documentation.
Use Event-Based Exports Over Static Exports
While static exports can be used, it is usually much more efficient to use event messaging via event-based exports.
In reality, for a larger STEP system with several million products, an event-based integration is the only viable approach. Not only from a STEP perspective (the time taken from the STEP side to export all data), but also from the receiving system perspective (the time to import all data).
It is recommended to use event-based exports over selection-based exports whenever possible.
For more information, refer to the Outbound Integration Endpoints in the Data Exchange documentation.
Legacy - Use Separate Queues for Important OIEPs
Legacy background process (BGP) functionality (BGP Legacy Multiple Queues) uses specified queues, while the recommended BGP execution mechanism runs BGPs based on the priority of the BGP and the created time. Refer to the BGP One Queue topic in the System Setup documentation.
The legacy outbound integration endpoint (OIEP) initiates a background process which handles the actual export.
-
The legacy Queue for endpoint parameter stores the queue that is used by the background process to poll the outbound integration endpoint. The default value is OutboundQueue.
-
The legacy Queue for endpoint processes parameter stores the queue that is used by the background processes to handle the actual export. The default value is Out.
STEP allows you to define separate queues for the endpoint and the endpoint process of an OIEP in the Configuration section of the OIEP editor. In this example, the queue for endpoint processes for the outbound integration endpoint is renamed to OutOIEP1.
The first time you activate the endpoint, a queue with the specified name is created if it does not already exist. Events are not lost if a separate queue for endpoint process is defined.
Changing the legacy Queue for the endpoint process for OIEPs means each OIEP background process uses this named queue for the actual export, causing OIEPs to run simultaneously and not wait for other OIEPs to finish processing. Do not use this when OIEPs require sequential export processing.
Recommendations
Configure the simplified BGP execution mechanism (One Queue) to run BGPs based on the priority of the BGP and the created time. Refer to the BGP One Queue topic in the System Setup documentation.
Configure separate Queue for endpoint processes (to handle the actual export) for all high-priority or long-running OIEPs that do not require sequential processing (exports).
Refer to the Background Processes and Queues topic in the System Setup documentation for examples of parallel and multithreading properties.
Consider In-Memory for Exports
In-Memory can improve export performance because In-Memory provides faster read operations. This is beneficial for complex data models, such as export configurations with data aggregations and export configurations with data that navigates references.
For more information, refer to the In-Memory Database Component for STEP topic in the Resource Materials online help documentation.
Optimize STEP Setup for Export Performance
Use the following setup recommendations to optimize export performance.
- Evaluate the export schedules of all exports and schedule long running exports sequentially (one after the other) to balance the STEP system load.
- Remove inactive and unused OIEPs (which includes queues, background processes, and export files, etc.) on operational production environments. When an OIEP is removed, the corresponding background processes are also removed.
- Set OIEPs to limit the number of background processes kept after export using the 'Maximum number of old processes' and 'Maximum age of old processes in hours' parameters, as defined in the IEP Auto Delete Settings section of the Create a Background Processes Maintenance Plan.