This is one of the technical infrastructure data gathering methodologies and performance recommendations for server hygiene. The full list is defined in the Server Hygiene topic.
When the STEP application server is overloaded, it can be caused by the number of import files in hotfolders. Furthermore, large import files can also cause the system to be slow.
Analyze the import files situation as follows:
- Determine where the import files are stored. On the Start Page, click the System Administration button and supply the login credentials.
- On the Configuration tab, click the Fetch data button to load the data.
- Search for 'Install.HotfolderRoot' and note the hotfolder location.
- Find the number of import files in the hotfolder location.
- For a Linux application server, use the following command:
cd /upload find hotfolders/ -type f | wc -l
- For a Windows application server, open the Windows File Explorer, and go to the '/upload' location. Right-click the directory and click Properties.
- Find the size of import files in the hotfolder location.
- For a Linux application server, use the following command:
cd /upload du -hsx * | sort -rh | head -10
- For a Windows application server, open the Windows File Explorer, and go to the '/upload' location and view the size as shown in the following image.
Identify the Causes
If the number of import files in the hotfolder is in the hundreds of thousands (e.g., 250k files), or if the size of the import files is too large, continue with the following activities to determine the cause.
IIEP 'Keep file after load'
Review the active hotfolder and REST Receiver IIEPs and verify that they are configured to automatically clean up the import files. Set the 'Keep file after load' parameter to 'No' to remove import files after success import. For more information, refer to the Hotfolder Receiver topic of the Data Exchange documentation here.
IIEP failed or completed with errors
Review the IIEPs for those that regularly complete with errors, or fail. In these cases, the import process is saved along with the imported file for error tracing. When the import process errors, the accompanying background processes are not removed. This means neither the import files nor the background processes are not being removed. When an IIEP successfully completes without errors, both the background processes and the import files can be deleted.
IIEPs that encounter Optimistic Locking errors can result in background processes that fail or 'complete with errors.' In this case, consider setting the 'Reference Target Lock Policy' to 'Relaxed' on the object types for which the long transaction applies. For more information, refer to the Reference Target Lock Policy on Object Types topic in the System Setup documentation here.
IIEP non-standard configuration
IIEPs are created with a standard import directory structure as illustrated below. Modifying that structure can prevent the removal of the imported files after successful import and processing.
|
Folder |
Example of Standard Config |
Description |
|---|---|---|
|
root |
/upload/hotfolders/products/ |
|
|
in |
/upload/hotfolders/products/in |
import files reside, and are removed after processing |
|
save |
/upload/hotfolders/products/save |
import files remain when they are imported and when configured to keep the import files after successful processing |
|
error |
/upload/hotfolders/products/error |
import files remain when the import process completed with errors |
|
failed |
/upload/hotfolders/products/failed |
import files will remain when the import process failed |
Remove Import Files
Based on the reason for the excess import files, follow the steps below to remove them manually:
- If the import files were not removed because the 'Keep file after load' was previously set to 'Yes,' use the following directions to remove the files from the 'save' folder:
- For a Linux server, use the command in the save folder holding the excess files:
rm -rf *
- For a Windows server, open Windows File Explorer and delete the files from the 'save' folder.
- If the import files were not removed for a reason other than the 'Keep file after load' setting, this indicates the import files are tied to background processes.
Remove the background processes of the IIEP, which also removes the corresponding import files. Refer to the Clean Up Background Processes topic for details here.
Important: Removing these import files by deleting them directly from the file system will not delete the corresponding background processes.