Amazon SQS Receiver

Using the Amazon SQS Receiver allows data to be imported into STEP from an Amazon SQS queue on AWS. For Amazon S3 storage, choose FIFO or standard queues based on your integration needs. Generally, if Amazon S3 storage is used for large messages, standard queues are recommended since a message that points to the S3 location is placed on the SQS queue and the large file is placed on S3. To determine the best option for your implementation, search the Amazon SQS Developer Guide online for details on the 'FIFO queues' and 'standard queues' options.

The Amazon SQS queues require a file size of less than 256KB. Common setup is to use file compression (zip export file) to reduce the message size to meet this limitation. Amazon S3 storage account is required to handle files 256KB or larger. For more information, search Amazon S3 online.

The Amazon SQS Receiver option can work in conjunction with the Amazon SQS Delivery Method on an OIEP to move messages into and out of STEP via AWS.

To access the Amazon SQS Receiver, the Receiver.AmazonSQS component must be activated on your system in addition to the normal update procedures. Contact Stibo Systems for more information.

Prerequisites

Changes to the properties file, outlined below, are implemented when the server is restarted.

  1. For the 'Credentials profile' parameter, create a credentials file following the instructions at http://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html. Place the credentials file in the required path on the application server. This file is used by both the OIEP Amazon SQS Delivery Method and the IIEP Amazon SQS Receiver.
  2. Prior to configuration, clicking the Server URL dropdown parameter displays the required property name. Provide a selection for the dropdown parameter via the sharedconfig.properties file on the STEP application server using the case-sensitive SQSServerUrl property. The following is an example of a property entry for a single server:
SQSServerUrl=1=https://sqs.us-east-2.amazonaws.com
  1. Prior to configuration, clicking the Credentials Path dropdown parameter displays the required property name. Provide a selection, including the name of the credentials file, for the dropdown parameter via the sharedconfig.properties file on the STEP application server using the case-sensitive SQSCredentialsPath property. If necessary, use a comma and increment the number to separate multiple paths as shown in the example below.
SQSCredentialsPath=1=C:\\Users\\stibosw\\.aws\\credentials,2=C:\\Users\\stibosw\\.aws\\credentials123
  1. For the 'Queue name' parameter, if necessary, create a new queue on the SQS server. Use the steps defined in the Amazon SQS Queue Configuration section of the Amazon SQS Delivery Method topic.
  2. If necessary, establish the Amazon S3 storage account and create the required buckets. For details, refer to Amazon S3 on the web.

Contact Stibo Systems if you need assistance with setup.

Configuration

After completing the prerequisite steps, edit the receiver of the IIEP. Use the following parameters to configure the IIEP. For information on a parameter, hover over the parameter field to display help text.

  1. For Receiver, choose Amazon SQS.

  2. In Server URL, select the URL that points to the SQS server where the data will be delivered.

  3. In Credentials path, select the path to the desired SQS credentials file.

  4. In Credentials profile, enter the profile name included in the SQS credentials profile.

  5. In Queue Name, enter the Amazon queue created for this delivery. This queue is also required when Amazon S3 is being used.

  6. For S3 Enabled, select 'Yes' if you have an Amazon S3 storage account. Setting this option to 'No' results in an error for messages that exceed the 256KB size limit.

    Note: Generally, if Amazon S3 storage is used for large messages, standard queues are recommended since a message that points to the S3 location is placed on the SQS queue and the large file is placed on S3.

  7. Click the Next button to continue with the IIEP - Configure Endpoint and subsequent steps.