Storing source data takes up disk space in the database. To keep the extra storage impact stable over time, the historical source data values should eventually be purged. The source data does not use the In-Memory functionality and so does not require additional memory. When importing new records, individual records can require an additional 20-30 percent of space to accommodate for the source records. However, with minor updates to existing golden records, source data can require 250-300 percent of additional disk space for the individual records.
The ‘Source Record Data Management – Historical Values Cleanup’ event processor incrementally purges the oldest historical source data. This includes simultaneously cleaning historical values and optimizing storage to keep disk usage as low as possible. Depending on the average amount of changing values at each import, the extra storage impact for updates could be reduced to 50-80 percent.
The purging logic is individual per source record and per attribute / reference / data container, so frequently changing values does not cause other historical values to be purged prematurely.
Deleted objects (objects in the recycle bin) are ignored during processing.
The Parameters and Event Triggers sections below contain important information on settings that should be considered when creating an event processor using this processing plugin.
Prerequisites
This section of documentation describes configuration steps for this specific processor, but that is only one part of configuring an event processor. For the full set of instructions on configuring an event processor, refer to the Event Processors topic.
Parameters
Each of the relevant parameters for the Event Processor Wizard 'Configure Processing Plugin' step are described below. Any additional wizard parameters with importance for this plugin are also included in this topic.
- On the 'Configure Event Processor' step, for the Select Processor parameter, choose the Source Record Data Management – Historical Values Cleanup processing plugin to display the expected parameters on the 'Configure Processing Plugin' step.
- Click the Next button to display the wizard step 'Configure Processing Plugin' and set the following parameters:
- Days to keep historical values – The number of days historical values are kept before purging
- Max number of historical values to keep – The number of values, changing over time, to keep
Note: The data purge occurs if just one of the two configuration criteria are met. To disable the purging of historical values and keep the disk usage optimization functionality, set both configuration values to a very high number.
- Click the Next button to display the wizard step 'Schedule Event Processor' and schedule as required.
-
Click the Next button to display the wizard step 'Error Handling and Reporting' (here) and configure as required.
-
Click the Finish button.
Event Triggers
For the event processor to function as scheduled, the Event Triggering Definitions tab must be configured correctly.
By default, events are triggered on the Approved workspace. Derived event functionality is available for triggering events prior to approval, as defined in the Derived Events topic in the System Setup documentation.
- Triggering Object Types - set at least one valid Golden Record object types.
Event Triggering Definitions are the same in Event Processors as they are in an OIEP. For details about the triggers, refer to the OIEP - Event-Based - Event Triggering Definitions Tab in the Data Exchange documentation here.