Attribute Value Migration

STEP 8.1 introduced the Compact Value Storage (CVS), a new storage model for non-LOV attribute values. The benefits of this storage model are that less space is required for storage and that the reading and writing of values can be performed faster. The CVS model is automatically used for attributes created in STEP 8.1 and newer systems.

To take advantage of CVS and migrate attribute values that were created on pre-8.1 STEP systems, the workbench includes the Migrate Attribute Values option for users to configure and schedule the migration. Refer to the Technical Migration Details section below for a detailed explanation of what the migration process entails.

Note: You cannot perform schema maintenance actions like changing the validation base type or attribute dimension dependencies while values for an attribute are being migrated.

Prerequisites

The attribute value migration should be configured and run by an admin or super user who has knowledge about the system utilization. The user configuring and running the migration must have the setup action privilege 'Modify definition of product attribute (domain and default unit)' for all attributes on the system. For more information on this privilege, refer to the Attributes section of the Setup Actions and Error Descriptions topic in the System Setup documentation here.

Important: It is advised to perform the migration, at least partially, on a test system with the same hardware setup to validate the settings before starting the migration on a production system.

Technical Migration Details

The migration functionality deletes rows from the 'valuemap', 'valuemodel' and 'value7' database tables, and inserts rows in the 'softvalue' table. Bulk deleting rows from Oracle tables, however, cause at least two issues:

  • As data is deleted, blocks and segments become sparsely populated. This may lead to Oracle selecting inefficient execution plans, again causing an observable performance decrease.

  • Tablespace is not released for 'reuse' automatically. This means that if nothing is done to prevent it, the migration requires as much additional tablespace as is required for the 'softvalue' table inserts.

To counteract these issues, the migration functionality not only deletes and inserts rows, but it also compacts tables and indexes. The following types of compactions are used:

  1. 'valuemap', 'valuemodel' and 'value7' index compaction – This type of compaction can be performed while the migration is running and does not require any restrictions to be imposed on the system usage.

  2. Table compaction of 'valuemap' and 'valuemodel' – This type of compaction cannot be performed while the 'database mode importer' is running and therefore, when configuring the migration schedule, it is possible to specify windows where the database mode importer can be disabled. The 'database mode importer' is only available to super users with all data node dependent privileges. It allows import processes to batch insert unrevised data directly in the database and may provide performance benefits for very specific use cases. The database mode importer is not available on systems running In-Memory.

  3. Shrinking 'valuemap' and 'valuemodel' – This operation requires a short-lived Single-Update Mode (SUM) lock. Therefore, when configuring the migration schedule, it is possible to specify windows where a SUM lock can be taken. Refer to the Single-Update Mode section of the System Setup documentation here.

  4. Shrinking 'value7' – Due to the 'value7' table having a function-based index, the shrink space approach used for 'valuemap' and 'valuemodel' cannot be used while STEP is running. Thus, when all values have been migrated, it is possible to schedule the required 'value7' operations for next system restart. The function-based index is dropped, the table is shrunk, and the index is rebuilt. Refer to the Completing the Migration section of the Running the Attribute Value Migration topic here.

Migration Preparation

Important: Ideally, perform attribute value migration during off peak hours and for as long as possible in one session.

After any error, restart the attribute value migration process to test if the migration has failed. Restarting the migration process often fixes issues, such as 'undo segment' errors.

To avoid common issues, perform the following steps before running the attribute value migration to avoid common issues:

  1. If possible, upgrade to the latest version of STEP.

    For environments running STEP 9.0-mp5, search the release notes for 'ISSUE-513717 - Compact Soft Migration tablespace fix' and contact Stibo Systems Support to apply the hotfix.

  2. Restart STEP and suspend all processes, including:

    • Background Processes

    • Integration Endpoints

    • Bulk Update Processes

    • Business Rules

  3. Review the Full Text Search setting and disable it during the migration since an enabled full text index ('value_ctxix') has a negative impact on the migration performance. For details about this feature, refer to the 'Important' information in the Full Text Indexable Attributes topic here.

    In the workbench, determine if the full text index is enabled, navigate to an attribute in System Setup.

    • Enabled - If the 'Full text indexable' parameter is set to Yes, then full text index is enabled.

      It is recommended to drop the index before starting the migration. It can be enabled again once the migration is complete.

      To drop the 'value_ctxix' index, delete the dropfulltextsearchindex.sql file from the STEP_HOME/admin/sqlhome folder.

    • Disabled - If the 'Full text indexable' parameter is set to No, double-click the field to display the Change Full Text Indexable dialog. If the 'Enable advanced full text search' option is disabled in the dialog, the index is disabled for your system.

  4. Run a Full Analyze maintenance job through the Stibo Systems database toolbox or manually on the database via exec adminpck.analyze(true);

  5. Verify system requirements:

    • JAVA Heap Size is no less than 8-12 GB in production environments.

    • Allocate enough memory for both the OS (24 GB free) and, if applicable, In-Memory hugepages.

      Note: Carefully monitor system RAM and hugepage usage during the attribute value migration process as out of memory (OOM) issues are common.

    • Increase STEPSYSDATA (or any tablespaces that were renamed but serve the purpose of STEPSYSDATA) as necessary. While this varies by database, generally, 20 percent free space is enough.

      Note: Carefully monitor your file system size to avoid running out of space, which can cause STEP to crash and/or not function properly.

      • Frequent crashes can generate enough heap dumps to max out STEP Home file system.

      • Heap dumps can generate from different issues but a common one is crashing due to out of memory (OOM).

      • Archive logs can increase significantly.

    Begin the migration process as defined in the Running the Attribute Value Migration topic here.