This section describes the data gathering methodologies from a technical viewpoint for the clean-up of folders for hotfolder import and background processing.
Clean Up Import Files
When the STEP application server is overloaded, it can be caused by the number of import files in hotfolders. Furthermore, large import files can also cause the system to be slow.
Analyze the import files situation as follows:
- Set up an SFTP user in the STEP SaaS Self-Service UI and pay attention to the SFTP URL, which will typically be: <environment-name>-sftp.mdm.stibosystems.com.
- Use your preferred SFTP client application to connect via the SSH key associated with the previously created SFTP user.
- When connected, you will have folders for 'upload' and 'workarea' (as shown below):
- Look through the subfolders to find the number and size of import files in the hotfolder location under 'upload.'
Identify the Causes
If the number of import files in the hotfolder is in the hundreds of thousands (e.g., 250k files), or if the size of the import files is too large, continue with the following activities to determine the cause.
-
IIEP 'Keep file after load'
Review the active hotfolder and REST Receiver IIEPs and verify that they are configured to automatically clean up the import files. Set the 'Keep file after load' parameter to 'No' to remove import files after success import. For more information, refer to the Hotfolder Receiver topic of the Data Exchange documentation here.
-
IIEP failed or completed with errors
Review the IIEPs for those that regularly complete with errors, or fail. In these cases, the import process is saved along with the imported file for error tracing. When the import process errors, the accompanying background processes are not removed. This means neither the import files nor the background processes are being removed. When an IIEP successfully completes without errors, both the background processes and the import files can be deleted.
IIEPs that encounter Optimistic Locking errors can result in background processes that fail or 'complete with errors.' In this case, consider setting the 'Reference Target Lock Policy' to 'Relaxed' on the object types for which the long transaction applies. For more information, refer to the Reference Target Lock Policy on Object Types topic in the System Setup documentation here.
-
IIEP non-standard configuration
IIEPs are created with a standard import directory structure as illustrated below. Modifying that structure can prevent the removal of the imported files after successful import and processing.
|
Folder |
Example of Standard Config |
Description |
|---|---|---|
|
root |
/shared/upload/hotfolders/products/ |
|
|
in |
/shared/upload/hotfolders/products/in |
import files reside, and are removed after processing |
|
save |
/shared/upload/hotfolders/products/save |
import files remain when they are imported and when configured to keep the import files after successful processing |
|
error |
/shared/upload/hotfolders/products/error |
import files remain when the import process completed with errors |
|
failed |
/shared/upload/hotfolders/products/failed |
import files will remain when the import process failed |
Remove Import Files
Based on the reason for the excess import files, follow the steps below to remove them manually:
- If the import files were not removed because the 'Keep file after load' was previously set to 'Yes,' delete those files from your SFTP client.
- If the import files were not removed for a reason other than the 'Keep file after load' setting, this indicates the import files are tied to background processes.
Remove the background processes of the IIEP, which also removes the corresponding import files.
Important: Removing these import files by deleting them directly from the file system will not delete the corresponding background processes.
Create a Background Processes Maintenance Plan
Settings on IEPs and on background process allow configuration of automatic clean-up activities.
IEP Auto Delete Settings
Use the following parameters to configure automatic deletion of background processes on IIEPs and OIEPs:
- Maximum number of old processes: Specify the number of ended processes the system will keep. Succeeded and ended processes are deleted when the number exceeds the specified limit. The oldest processes are deleted first. Setting this number too high may eventually degrade performance.
For example, if 'Maximum number of old processes' is set to '1000,' a maximum of 1000 succeeded and ended background processes will be retained. The oldest background processes are deleted automatically when the number exceeds the specified limit of 1000.
- Maximum age of old processes: Specify the maximum age of ended processes that the system will keep. Ended processes are deleted when the maximum age is exceeded. Setting this number too high may eventually degrade performance.
For example, if 'Maximum age of old processes' is set to '1 y,' the background processes older than 1 year are deleted automatically.
Ensure the configurations represent a realistic situation as demonstrated below:
- In an environment with a large number of small imports and exports, use '1000' as a maximum number of old processes and '1 w' as a maximum age of old processes.
- In an environment with small number of large imports and exports, use '50' as a maximum number of old processes and '1 m' as a maximum age of old processes.
Background Process Auto Delete Settings
Automatic deletion of background processes is also managed through a configuration. The configuration defines the number of hours after the background processes ends that it will be deleted and is based on the specified background process template.
- Set up an SFTP user in the STEP SaaS Self-Service UI, if you have not done so already, and pay attention to the SFTP URL. It will typically be: <environment-name>-sftp.mdm.stibosystems.com.
- Use your preferred SFTP client application to connect via the SSH key associated with the previously created SFTP user.
- When connected, you will have folders for 'upload' and 'workarea' (as shown below):
- List the subfolder names in the 'background-processarea' under 'workarea' to view the available process templates.
- Configure the auto-delete hours of the background process templates by using the System Management Swagger UI as follows:
- Access the System Management Swagger UI, typically at the URL: https://<system name>.mdm.stibosystems.com/system-management/swagger-ui
- Authenticate yourself with the 'Authorize' button using the credentials of a user in STEP with administrative privileges:
- Check to determine if any of the case-sensitive parameter(s) have been set already using the GET /step/configuration endpoint:
- Add or edit the templates to define the number of hours that should elapse prior to the automatic deletion of the template's background process.
For example, the most important templates are generally Exporter, Importer, Outbound, Inbound, and Web Publisher. The parameter for Exporter can be configured to auto delete after 168 hours as follows:
