Storing Source Records for Golden Records

This functionality is used by a Match and Merge solution. For more information, refer to the Match and Merge topic (here) and the Configuring Match and Merge topic (here).

During a Match and Merge operation, the imported data is often merged directly into golden records. Without configuring the 'Keep Source Records' option, this automated process discards data from different source systems. Once data is discarded during merge, unmerging is impossible because the new records created by the process are missing data.

Storage of source data is only supported on object types identified by the 'Matching - Merge Golden Record' component model.

The Matching - Merge Golden Record component model uses the following aspects to store source records imported with the source record ID and provide the unmerge functionality:

  • Keep Source Records for Golden Record Object Types

  • Source Record ID Attribute

  • Source System ID Attribute

Removing a golden record object type from the component model does not delete source data that is already stored in the system. When an object type is deleted from the component model the system stops storing source data.

Source data includes a revision history and provides data lineage functionality. For more information, refer to the Match and Merge Traceability topic here.

Considerations

Review the following when planning your configuration:

  • Ensure all data container keys are defined satisfactorily. Changing key definitions later impacts the validity of the existing stored source data and creates issues because modified data container keys are incomplete or data container instances are duplicates. There is no method to identify data container source data as there is with golden records.

  • After enabling the storage of source data, perform a full import of the source data either via IIEP or web service. Otherwise the source data in the system is incomplete and future partial updates will complicate the unmerge process. Without a full import, the system does not have a full dataset from each source.

  • Storing source data increases the disk space used by the underlying storage system. The extent of the increase depends on the frequency of source record updates. The 'Source Record Data Management – Historical Values Cleanup’ event processor (discussed below) works to limit the space used.

Storage Functionality

Source data storage include the following functionality:

  • Source data is stored persistently in the system database and therefore it is included in standard backup procedures.

  • Source data is excluded from In-Memory implementations.

  • Source data storage is accumulative, meaning a source record can be updated by only sending part of the complete source dataset.

  • Send an empty tag in STEPXML to delete an attribute value.

  • No two records of the same object type in STEP should ever share the same Source Record ID for the same Source System.

  • Source Systems may have several IDs on a single record in STEP.

  • Different source systems are expected to assign different IDs to the same customer.

The following sections describe storage functionality upon import for multi-valued data containers and multi-valued references.

Multi-Valued Data Container without a defined Data Container Key

  • All instances must be imported every time because existing instances are always replaced.
  • Existing instances that are not part of the update are deleted.
  • Applies only if that data container type is part of the import. If not, the existing instances are left unchanged.

Multi-Valued Data Containers with a defined Data Container Key

  • Only instances with a matching key are updated.
  • If no matching keys are found, a new instance is created.
  • Existing data container instances cannot be deleted.

Multi-Valued References

  • Instances of the reference are updated with respect to reference target.
  • If no matching target is found, a new instance is created.
  • Existing reference instances cannot be deleted.

Configuration

Complete the following workbench configuration:

  1. In System Setup, open the Component Model node and select the ‘Matching – Merge Golden Record’ component model.

  2. Verify an object type is selected on the 'Keep Source Records for Golden Record Object Types' aspect. If needed, modify the component model as defined in the Configuring the Matching - Merge Golden Record Component Model topic here.

  3. Create and configure event processing plugin 'Source Record Data Management – Historical Values Cleanup’ as defined in the Event Processors topic of the System Setup documentation here.

  4. Perform a full import of the source data either via IIEP or web service so the full dataset from each source is available for the unmerge process.

Maintenance

Once data has been stored, if needed, purge source data via Bulk Update as defined in the Merge Golden Records: Purge Source Data Operation topic of the Bulk Updates documentation here.