Configuring the Match Data Exchange Method

As defined in the following sections, a match and merge solution communicates with external systems using either an asynchronous IIEP or a synchronous web service setup.

For a detailed explanation of how inbound records are identified as either updates to existing records or creation of new records, refer to the Inbound Record Flow section of the Match and Merge Flow Details topic here.

This functionality is used by a Match and Merge solution. For more information, refer to the Match and Merge topic (here) and the Configuring Match and Merge topic (here).

Asynchronous Merge Inbound Integration Endpoint

Data can flow into STEP via an asynchronous inbound integration endpoint (IIEP). The IIEP is designed to receive large batches of source records from any of a number of Receiver plugins.

The incoming source data is translated into STEPXML import files. These input files are typically handled one at a time in sequence, according to the parallel settings of the IIEP queue, as defined in the IIEP - Configure Endpoint topic in the Data Exchange documentation. The result of the import operation is logged in workbench on the IIEP configuration's Background Processes tab and on the background process execution log.

Any failed records are stored on the BGP in a separate error file which allows the failed updates to be reattempted when errors have been corrected.

For configuration details, refer to the Match and Merge IIEP Configuration section of the IIEP - Configure Match and Merge Importer Processing Engine topic in the Data Exchange documentation here.

Synchronous Match and Merge Web Service Endpoint

The match and merge synchronous web service endpoint is an alternative to the asynchronous IIEP. It delivers an answer to each request, alerting the external system to the result of the match and merge operation.

The request sent to this service includes the following information:

  • User name and password for access validation.
  • A reference to a STEP context.
  • A reference to a STEP Match and Merge web service endpoint.
  • Entity representations of each record to be imported. Non-duplicates can be declared via the non-duplicate reference types, as defined by the matching component model.

The web service receives a request and completes the following process on incoming data:

  1. Validation - Ensures minimum data requirements are satisfied (e.g., record has an address or a last name). Records that are not successfully validated are rejected and not stored in STEP.
  2. Standardization - Standardizes data based on the configuration (e.g., address standardization).
  3. Matching - Identifies existing record matches and potential record matches. The outcome is one of the following:
  • new or updated golden records in STEP
  • rejection from the web service

In all cases the web service response includes if:

  • the incoming record was validated.
  • any potential duplicates were found.
  • there is new / updated information on the record itself.
  • the record will be handled manually in a clerical review workflow.

The following topics include more information on:

  • Web service endpoints - refer to the Web Service Endpoints topic in the Data Exchange documentation here.

  • Web service merging configuration details - refer to the Web Service Endpoint - Match and Merge topic in the Data Exchange documentation here.

Parallel Constraints

STEP imports use several users to import records in parallel. STEP tries to avoid two users updating the same golden record simultaneously, which will slow down imports. Often, input files contain a series of records where the sequences of records are updates of the same contact. STEP use parallel constraints to avoid running these in parallel. Avoiding a series of updates for the same records in the input data can sometimes improve performance.

There are two kinds of parallel constraints: strict parallel constraints and relaxed parallel constraints.

  • Strict: If a golden record or source system record ID exists in the input, STEP adds it as a strict parallel constraint. The STEP import cannot benefit from multiple if too many consecutive records in the input and/or all have the same source record ID for the same source system.

  • Relaxed: When you initiate two or more golden records through a match and merge matching algorithm, the algorithm calculates match codes for those golden records. STEP then adds parallel constraints based on those match codes, preventing entities with overlapping match codes from importing simultaneously. However, STEP adds the calculated match codes as a relaxed parallel constraint. When checking constraints, STEP will start ignoring relaxed constraint values that have occurred too frequently (e.g., thousands of contacts all use the same reception phone number, so STEP ignores this repeated value).