Integration Endpoint Options for VCS Integration

The suite for Version Control System Integration (VCSI) consists of these options:

• STEPXML Splitter Post-processor in Outbound Integration Endpoint (OIEP)

• Git Delivery Method or Change Package Git Delivery Method in Outbound Integration Endpoint (OIEP)

• STEPXML Joiner Pre-processor in Inbound Integration Endpoint (IIEP)

• Invoke OIEP Post-processor Inbound Integration Endpoint (IIEP)

Each is described in their own section below.

Additionally, within this topic, is information regarding an editable business rule format. JavaScript-based business rules can be created, maintained, and unit tested outside of STEP. This allows customers and partners to govern the lifecycle of business rules in a standard source code control system such as Git, and from there, deploy appropriate versions of the business rules to the various STEP systems that are part of a Development, Testing, Acceptance and Production (DTAP) environment.

Grouping Changes

Options available for grouping configuration objects include select objects, typically using a collection or event-driven with change packages upon sealing. Initial setup requirements are different for each option.

Change packages are recommended when working on projects in an iterative development process, where smaller changes are grouped separately for the complete configuration. However, a change package can also be used to group a more complete configuration definition.

STEPXML Splitter Post-processor in OIEP

Use of the STEPXML Splitter is determined by the grouping option selected:

• For the Change Package Git Delivery method, the STEPXML Splitter is integrated into the delivery method, so the STEPXML Splitter post-processor must not be added to the OIEP.

  For change packages, all exports use the Flattened split mode and export definitions as comments. The user can decide to also export business rules converted to an editable format on the delivery method dialog, which replaces the STEPXML files that include comments with editable JSON files for business rules.

• For a collection used to group changes and an Advanced STEPXML template configured for various configurations that do not include the Change Package object, the STEPXML Splitter is required on the OIEP.

The STEPXML Splitter post-processor can take any STEPXML file produced by the STEP Exporter processing engine as input and splits the file into multiple valid STEPXML files and/or editable business rule format files that are then passed to the configured delivery method. Generally, the splitter produces one file per STEP object and further normalizes the content so that elements for which the sequence has no significance in STEP are output in the same order every time. Non-object configurations (e.g., derived events) and system settings are output in a single file.

Splitting and normalizing makes it easier to compare configurations outside of STEP in a VCS like Git. Further, it makes it easy to selectively choose specific configuration items to be imported on another system.

The STEPXML Splitter post processor has the configuration options explained below.

Split mode

Split mode defaults to ‘Flattened’ but also allows 'Hierarchical'. The parameter affects how to represent STEP objects in the produced split files when they are typically exported in a nested structure.

To illustrate the difference, in the example shown below, the following classification hierarchy is being exported:

In both modes, one file will be created per classification object.

In Flattened mode, upper levels are omitted, and each file contains exactly one 'Classification' element with parent identifier information, as shown below.

(…)
<Classifications>
  <Classification ID=‘stibo.IMConfigFolder’ UserTypeID=‘ImportConfigurationsRoot’ ParentID=‘ConfigurationsRoot’>
    <Name>Import Configurations</Name>
    <MetaData>
      <Value AttributeID=‘Purpose’ QualifierID=‘en-US’>Storage for import configurations</Value>
    </MetaData>
  </Classification>
</Classifications>
(…)

In Hierarchical mode, each of the leaf classifications (e.g., Import Configurations) is nested inside the element representing the 'Configurations' classification, which is stripped of all but ID, object type ID, and parent ID information, as shown below.

(…)
<Classifications>
  <Classification ID=‘ConfigurationsRoot’ UserTypeID=‘ConfigurationsRoot’ ParentID=‘Classification 1 root’>
    <Classification ID=‘stibo.IMConfigFolder’ UserTypeID=‘ImportConfigurationsRoot’>
      <Name>Import Configurations</Name>
      <MetaData>
        <Value AttributeID=‘Purpose’ QualifierID=‘en-US’>Storage for import configurations</Value>
      </MetaData>
    </Classification>
  </Classification>
</Classifications>
(…)

Generally, it is recommended to use the default 'Flattened' mode while the 'Hierarchical' mode should only be used if the full hierarchy path must be present in each file.

Note: The 'Hierarchical' example from above can be imported on a system where the classification with ID 'ConfigurationsRoot' is not present as it will be created during import. Importing the 'Flattened' example on such a system, however, results in an error.

Convert business rules to editable format

This option determines how business rules (conditions, actions, functions, and libraries) are exported.

• If set to No, the business rules are exported as STEPXML files.

• If set to Yes, the rules are exported in the editable *.js format described in this topic.

  When exporting editable business rules, set this option to 'Yes.' The business rules in the STEPXML that are fed to the post-processor are converted to the editable format and represented in a single *.js file instead of being represented in a STEPXML file. For details, refer to the VCSI: Editable Business Rules Format topic.

Change Package Git Delivery Method in OIEP

The Change Package Git Delivery method allows integration with popular repositories, supporting the HTTPS (token-based) or the SSH (file-based) access methods for GitHub, GitLab, and Bitbucket.

The Change Package Git Delivery method delivers files produced by the OIEP processing engine using an integrated STEPXML Splitter to deliver multiple files to a branch in a remote Git repository. Refer to https://git-scm.com for more information about Git.

The change package is represented below the specified branch within a configurable directory structure. At the end of the directory structure, change packages grouping files by the Primary and Secondary flippers, then by type of object with XML or JSON files named by object type and the ID of the object.

Note: The remote repository cannot be empty. At least one branch with one file must exist (e.g., the 'main' branch and the README.md file).

Prerequisites

Important: For on-premises systems, this feature requires the configuration-management component.

Changes to shared configuration properties on the STEP application server are required to populate the dropdown list delivery options. Changing these properties does not require a system to be restarted.

Prior to configuring the Change Package Git Delivery method, add the desired case-sensitive properties described below.

Note: In the properties mentioned below, use a sequential set of variable integers in the property names, for example, 1, 2, 3. Property values are not read correctly if the sequence is not sequential, for example 1, 3, 4.

1. For both on-prem and Stibo System SaaS systems, the values for the Remote Git Repository URI parameter defines the URI for the remote repository and are read from the case-sensitive ChangePackageGitDelivery.RemoteRepoUri.[integer] property. For example:

   ChangePackageGitDelivery.RemoteRepoUri.1=https://gitlab.com/john-smith/step-conf.git 
   ChangePackageGitDelivery.RemoteRepoUri.2=git@bitbucket.org:john-smith/smithrepo.git

   Note: For internally hosted Git setups, ensure that any firewall access rules, IP white-listing, or ports (tcp/22 or similar) are configured to allow the STEP server(s) to access the needed Git repository.

2. The values for the Git Branch parameter define the name of the branch to which the delivery is published, and are read from the case-sensitive ChangePackageGitDelivery.Branch.[integer] property. For example:

   ChangePackageGitDelivery.Branch.1=step-dev-1 
   ChangePackageGitDelivery.Branch.2=step-q

3. The values for the Repository User Name parameter defines the remote repository username and supports delivery via HTTPS and PAT or SSH with a key file accessible to the application server (discussed later). Values for this parameter are read from the case-sensitive ChangePackageGitDelivery.RemoteRepoUsername.[integer] property. For example:

   ChangePackageGitDelivery.RemoteRepoUsername.1=john.smith 
   ChangePackageGitDelivery.RemoteRepoUsername.2=julie.baker 

Using SSH to Connect

To connect via SSH, the Path to Private Key When Using SSH parameter defines the remote repository RSA SSH private key generated using the old OpenSSH format and ed25519. The generated key file must be accessible to the application server through shared storage or SFTP.

Important: PuTTYgen (.ppk) SSH keys are not supported by STEP. Using an unsupported key type results in an 'invalid private key' error.

Follow these steps to generate a valid private SSH key:

1. Use ssh -V to check the version on your system and then generate a key.

   • With OpenSSH versions prior to 7.8, use the following command to generate the new OpenSSH format key:

     ssh-keygen -t rsa -b 4096 -C <comment> -f <keyfile_name>

     For example:

     ssh-keygen -t rsa -b 4096 -C john.smith@acme.com -f git_rsa4096_key

   • With OpenSSH versions 7.8+, generated keys default to the new OpenSSH format. To generate keys in the old format, use the following command:

     ssh-keygen -t rsa -b 4096 -C john.smith@acme.com -m PEM -f git_rsa4096_key

     Important: While the general approach and commands are the same with later versions of OpenSHH, the -m PEM argument shown above is needed with versions 7.8+.

2. Add the public version of the SSH key (e.g., git_rsa4096_key.pub) to your GitLab or Bitbucket account. Failure to add the public key to your account results in a 'Not authorized' error.

   • For GitHub accounts, refer to the GitHub Limitations and Configuration section below.

   • For Bitbucket accounts, refer to the Bitbucket Limitations and Configuration section below.

   • For a general overview of supported authentication methods, refer to the Supported Authentication Methods (per Git Service) section below.

     Values for the Path to Private Key When Using SSH parameter are read from the case-sensitive GetDelivery.SshPrivateKeyUri.[integer] property. These entries must contain the full path to the private SSH key file. For example:

     GitDelivery.SshPrivateKeyUri.1=/home/stibosw/.ssh/git_rsa4096_key
     GitDelivery.SshPrivateKeyUri.2=/workarea/.ssh/gitlab_rsa4096_enc_key
     GitDelivery.SshPrivateKeyUri.3=/upload/.ssh/bitbucket_rsa4096_key							

Configuration

For information on a parameter, hover over the parameter field to display help text.

1. On the Configuration tab, in the Delivery Method area, click Edit Delivery.

2. In Select Delivery Method, choose Change Package Git Delivery.

   

3. In Remote Git Repository URI, select a URI from the dropdown for the relevant remote repository.

4. In Git Branch, select relevant branch in the associated remote repository.

5. In Git Access Method, select HTTPS or SSH and provide additional values as required.

   • For HTTPS:

     • For the Repository User Name parameter, select an option from the dropdown.

     • For the Repository User Password parameter, add the personal access token generated from your repository developer tools.

   • For SSH:

     • For the Path to Private Key When Using SSH parameter, select the file path to the private key from the dropdown.

     • For the Repository SSH Passphrase parameter, add the passphrase.

       Note: Author name and email for the Git commit are sourced automatically from the user object associated with the sealing event on the change package and does not require explicit configuration in the delivery method. However, to support this functionality, email address values must exist for users that seal change packages that are integrated with Git.

6. In Directory Template, indicate where to store files on the repository using a standardized format. While editing the Directory Template, hover over the field to display the available macros.

   Use static text, macros, and the forward slash character as a separator to automatically create directories and organize change packages by:

   • $systemname$/$changepackageid$ - default, creates a directory under the branch for each system name where a change package is sent from and sub directories with the ID of the change package.

   • $systemname$ - system name, where the change package is sealed

   • $changepackageorigin$ - origin, where the change package was initially created

   • $changepackageuniqueid$ - system-defined unique ID

   • $changepackageid$ - user-defined change package ID

   • $attribute:attributeid$ - For additional flexibility, custom description attributes that are externally maintained and not dimension dependent, can be added to the change package object and referenced in the template to create subfolders based on values you define. When a custom attribute is used in the directory template, values must be present and LOV validation base type is recommended.

     Note: a-z, A-Z, 0-9,-, and _ are supported characters for static text. Other characters are replaced with an underscore.

7. In Convert Business Rules to Editable Format, select an option:

   • Yes - default, sends editable JSON files of business rules (actions, conditions, functions, and libraries) to the repository. The JSON files can be imported manually or with other files combined in a ZIP file that is sent to an IIEP using the STEPXML Joiner preprocessor.

   • No - exports STEPXML files that do not provide editable JavaScript access but can be imported as-is after export and include definitions as comments.

8. On the Edit Delivery Configuration dialog, click the OK button to save the delivery method.

Git Delivery Method in OIEP

The Git Delivery method delivers files produced by the outbound integration endpoint (OIEP) processing engine or a configured post-processor to a branch in a remote Git repository. Refer to https://git-scm.com for more information about Git.

When using this delivery method on an on-premises STEP system, the delivery to Git can be performed via a configured local directory accessible from all application servers in the cluster (refer to the Local git repository URI section below). With this setup, the delivery method first checks if the local delivery directory is Git enabled.

If the local delivery directory is Git enabled, the following operations are performed:

1. Git fetch

2. Git checkout (of configured branch)

3. Git hard reset

4. Git pull (if branch exists in remote)

If the local delivery directory is not Git enabled, the following operations are performed:

1. Git clone

2. Git checkout (of configured branch)

On cloud-based STEP systems, a temporary local directory is always used and the second approach (clone and checkout) is used.

Note: The remote repository cannot be empty. At least one branch with one file must exist (e.g., the 'main' branch and the README.md file).

The locally checked-out branch is now in sync with the remote branch, and the following operations are performed:

1. Files produced by the OIEP are written to the local directory

2. Files present in the local directory but not in the delivery are deleted

3. Git stage

4. Git commit

5. Git push

Configure the Git Delivery option using the following parameters:

Note: In the properties mentioned below, use a sequential set of variable integers in the property names, for example, 1, 2, 3. Property values are not read correctly if the sequence is not sequential, for example 1, 3, 4.

• Local git repository URI - URI for the local directory via which configuration files are synchronized.

  For on-prem systems, values for this parameter are read from the case-sensitive GitDelivery.LocalRepoUri.[integer] sharedconfig.properties file entries.

  Example:

  GitDelivery.LocalRepoUri.1=/workarea/conf-attributes
  .LocalRepoGitDeliveryUri.2=/workarea/conf-all
  

  allows selection between the values '/workarea/conf-attributes' and '/workarea/conf-all' for the parameter in the workbench.

  For cloud STEP systems, do not set the sharedconfig.properties file entry and leave this parameter blank.

• Remote git repository URI - URI for the remote repository.

  For both on-prem and cloud STEP systems, values for this parameter are read from the case-sensitive GitDelivery.RemoteRepoUri.[integer] sharedconfig.properties file entries.

  Examples:

  GitDelivery.RemoteRepoUri.1=https://gitlab.com/john-smith/step-conf.git
  GitDelivery.RemoteRepoUri.2=git@bitbucket.org:john-smith/smithrepo.git

  Note: For internally hosted Git setups, you must ensure that any firewall access rules, IP white-listing, or ports (tcp/22 or similar) have been configured to allow the STEP server(s) to access the needed Git repository.

• Git branch - Name of the branch to which the delivery is published.

  Values for this parameter are read from the case-sensitive GitDelivery.Branch.[integer] sharedconfig.properties file entries.

  Examples:

  GitDelivery.Branch.1=step-dev-1
  GitDelivery.Branch.2=step-qa
  

• Author name - Author name for the Git commit.

  Values for this parameter are read from the case-sensitive GitDelivery.AuthorName.[integer] sharedconfig.properties file entries.

  Examples:

  GitDelivery.AuthorName.1=John Smith

• Author email - Author email for the Git commit.

  Values for this parameter are read from the case-sensitive GitDelivery.AuthorEmail.[integer] sharedconfig.properties file entries.

  Examples:

  GitDelivery.AuthorEmail.1=john.smith@acme.com

• Repository username - The remote repository username supports delivery via HTTPS and PAT (discussed later).

  Values for this parameter are read from the case-sensitive GitDelivery. RemoteRepoUsername.[integer] sharedconfig.properties entries.

  Example:

  GitDelivery.RemoteRepoUsername.1=john.smith

  Note: The 'Repository username' and 'Repository user password' parameters and properties are only used for HTTPS-based and PAT-based (discussed later) connections. For SSH-based connections, leave these properties blank since they are ignored. SSH uses only the 'Path to private key when using ssh' and 'Repository ssh passphrase' (if applicable) parameters and properties.

• Repository user password - The remote repository password supports delivery via HTTPS and PAT (discussed later.) Password must be entered directly when configuring the delivery method for these authentication methods.

• Path to private key when using ssh - The remote repository RSA SSH private key generated using the old OpenSSH format.

  Important: PuTTYgen (.ppk) SSH keys are not supported by STEP. Using an unsupported key type results in an 'invalid private key' error.

  1. Use ssh -V to check the version on your system and then generate a key.

     • With OpenSSH versions prior to 7.8, use the following command to generate the new OpenSSH format key:

       ssh-keygen -t rsa -b 4096 -C <comment> -f <keyfile_name>

       Example:

       ssh-keygen -t rsa -b 4096 -C john.smith@acme.com -f git_rsa4096_key

     • With OpenSSH versions 7.8+, generated keys default to the new OpenSSH format. To generate keys in the old format, use the following command:

       ssh-keygen -t rsa -b 4096 -C john.smith@acme.com -m PEM -f git_rsa4096_key

       Important: While the general approach and commands are the same with later versions of OpenSHH, the -m PEM argument shown above is needed with versions 7.8+.

  2. Add the public version of the SSH key (e.g., git_rsa4096_key.pub) to your GitLab or Bitbucket account. Failure to add the public key to your account results in a 'Not authorized' error.

     • For GitHub accounts, refer to the GitHub Limitations and Configuration section below.

     • For Bitbucket accounts, refer to the Bitbucket Limitations and Configuration section below.

     • For a general overview of supported authentication methods, refer to the Supported Authentication Methods (per Git Service) section below.

  Values for this parameter are read from the case-sensitive GetDelivery.SshPrivateKeyUri.[integer] sharedconfig.properties entries. These entries must contain the full path to the private SSH key file.

  Example:

  GitDelivery.SshPrivateKeyUri.1=/home/stibosw/.ssh/git_rsa4096_key
  GitDelivery.SshPrivateKeyUri.2=/workarea/.ssh/gitlab_rsa4096_enc_key
  GitDelivery.SshPrivateKeyUri.3=/upload/.ssh/bitbucket_rsa4096_key

• Repository ssh passphrase - The remote repository SSH passphrase. The passphrase must be entered directly when configuring the delivery method.

Supported Authentication Methods (per Git Service)

	Protocols
Git Services	HTTPS	SSH	PAT*
GitHub
GitLab
Bitbucket

*PAT = Personal Access Token / App Password

GitHub Limitations and Configuration

The following limitations apply when using GitHub with STEP:

Unlike other Git services, GitHub is more restrictive in the level of security placed on accessing its repositories, specifically:

• Basic authentication for HTTPS connections is not supported.

• RSA SSH keys that use the sha-1 signature algorithm are not supported.

Therefore, the available authentication methods for using the Change Package Git Delivery method or Git Delivery Method with a GitHub account are with a Personal Access Token (PAT) or SSH with a private key accessible to the application server.

Bitbucket Limitations and Configuration

Bitbucket does not support using Basic authentication for HTTPS connections. This means that apart from SSH keys, the only other usable authentication method is app passwords.

Remote Setup Example

Below is an example of how a remote Git repository using Bitbucket (https://bitbucket.org) can be configured to work with the Git delivery method.

1. Create a new repository with a 'README' file via the Bitbucket web interface.

   

2. Create a system specific branch in the repository.

   

3. Get the 'Remote git repository URI' from, for example, the Bitbucket 'Clone' option.

   

STEPXML Joiner Pre Processor in IIEP

The STEPXML Joiner inbound pre-processor is designed to be used to import configurations and settings previously exported via the outbound functionality (described above). The outbound functionality produces STEPXML files and potentially *.js files representing business rules, and change package metadata when using the delivery method Change Package Git Delivery. The STEPXML Joiner option can take a .ZIP file containing any number of such files as input and then combines objects and settings in a single STEPXML and passes the combined file to the IIEP processing engine.

Various processing instructions for the combined STEPXML file can be included on the pre-processor. Processing instructions can be added via the UI (shown below) or can be included in a template STEPXML file within the .ZIP file to be processed by the endpoint.

Note: If a template file is provided in the .ZIP, the settings from this file override any UI configurations made in the UI. The template file must be named ProcessingInstructions.xml and only the ReplacementRules element should exist inside the STEP-ProductInformation element.

To configure the IIEP:

1. For Configure PreProcessor, select STEPXML Joiner.

2. For Run in single update mode, set to Yes or No as appropriate.

3. For Import context, select the context that will receive the imported data. When using the delivery method Change Package Git Delivery do not use option to ‘Derive from input files’ because they are not included when the change package is exported.

4. For Replace instructions, add replacement rules as needed.

   For 'list properties' (multiple instances of the same XML element at the same level) such as 'Value' elements inside the 'Values' element for a product or 'TargetUserTypeLink' elements for a reference type definition, special processing instructions (replacement rules) are used to express that the properties that are not present in the import file are to be removed from the system as part of the import. Click the 'Edit Replace Instructions' link.

   If you are using replacement rules across contexts with suppressions, include all contexts in an export for best results. This ensures complete results are communicated between systems.

   For more information regarding ReplacementRules, refer to the ReplacementRules Tag in STEPXML topic in the Data Exchange documentation.

5. For the Edit Replace Instructions dialog, build the rules by selecting options on the left and using the arrow button to move the rule over to the right. Click OK to save changes before moving to the next step of the Inbound Integration Endpoint Wizard.

Invoke OIEP Post Processor in IIEP

The Invoke OIEP post processor allows for an OIEP to be invoked once the IIEP process has completed. This allows updating the representation of the system configuration in a remote Git branch immediately after the configuration has been imported, if desired.

As shown below, the option requires the ID of the OIEP to be invoked.

For more information, refer to these topics in the Version Control System Integration section:

• VCSI: Editable Business Rules Format

• VCSI: Example Setups

• VCSI: Considerations and Limitations