User Guide > Process Steps and Components > Queue Step > AWS S3 Queue
Was this helpful?
AWS S3 Queue
Note:  For general information about Queue Steps, see Queue Step.
Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. You can use it to store and protect any amount of data for websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. For more information on Amazon's Simple Storage Service, see https://aws.amazon.com/s3/.
Actian DataConnect provides AWS S3 Queue component that can be used to connect to AWS S3. This component supports the following features:
• Specifying the AWS S3 bucket name that stores the objects being written and read.
• Specifying the Region that houses the bucket.
• Selecting common AWS authentication methods
• Uploading files or DJMessage content to S3 buckets
• Downloading objects from S3 buckets to DJMessages or files
• Removing objects from the S3 bucket when they are no longer needed in the process
• Creating a temporary bucket to be used in the current process. When the process calls the disconnect method of the component, the temporary bucket is deleted.
You can use the S3 to Avalanche template for loading data to Avalanche. It leverages the S3 component. For more information about this template, see Integration Manager documentation.
AWS S3 Queue Properties
Property
Default Value
Description
AWS Region
 
Region that hosts the specific S3 bucket. The component will be configured to use this bucket. For more information about AWS regions, availability zones, and other related topics, see
https://aws.amazon.com/about-aws/global-infrastructure/regions_az/
S3 Bucket Type
Use Existing Bucket
Select one of the following:
• Use Existing Bucket: Select this option to use an existing S3 bucket. The following additional options are displayed:
– S3 Bucket Name: Name of the AWS S3 bucket. The value must match the name of an existing AWS S3 bucket for which you have the appropriate privileges. This bucket will be used to upload and retrieve stored objects. For more information about AWS S3 buckets, see Working with Amazon S3 Buckets.
– Delete Objects When Finished: If the value is set to TRUE, then the component retains a list of all files that it uploaded during its life cycle in the process. When the component's disconnect method is called by the process, the uploaded files are deleted. The default value is FALSE.
• Create Temp Bucket: Select this option to create a temporary S3 bucket. It is used while the component instance remains active in the process. The following additional options are displayed:
– Temp S3 Bucket Prefix: Specify a custom prefix for the temporary bucket name that will be created by the component. The prefix must be consistent with AWS S3's bucket naming guidelines as defined in the Rules for Bucket Naming section in Bucket Restrictions and Limitations.
If you do not provide this name, the component uses the default prefix for all unique temporary bucket names that it creates. The default prefix is actian-dc-tempbucket-.
Authentication Type
Default Credential Provider Chain
Means by which identity is provided to AWS
• Default Credential Provider Chain: Uses the "default" credentials provider chain that is provided in the AWS SDK. For information about the standard mechanism that AWS client-side automation tools use to obtain user credentials, see New and Standardized Way to Manage Credentials in the AWS SDKs.
• Access Keys: Access keys are long-term credentials for an IAM user or the AWS account root user. Access keys are created in pairs that consist of the following (displayed as additional options):
– Access Key ID: Provide the access key ID.
– Secret Access Key: Provide the secret access key.
Note:  If the Access Keys authentication type is selected, the component will not search for additional means by which the credentials are provided.
Supported Actions
Action
Description
Connect
Creates an authenticated S3 client that will be used to perform all subsequent operations.
PutMessage
Provide a DJMessage or file name. The component will write this file or message to the S3 bucket specified in the S3 Queue properties. The component uses the authenticated S3 client to upload data to an S3 bucket and saves it using a provided "key". Since the data is represented in S3 as an object, the key is the "object name" or "file name".
GetMessage
Specify a DJMessage or a referenced file through which content can be downloaded. Uses the authenticated S3 client to download data from an S3 bucket. The data in the bucket is referenced by a key.
Disconnect
Removes the authenticated S3 client.
If the Create Temp Bucket option is enabled, then the temporary bucket (along with all uploaded content) is removed.
If Delete Objects When Finished option is selected, all uploaded content will be deleted.
 
Supported Action Parameters
Action
Parameter
Description
GetMessage
Message
Specify a DJMessage variable or file through which content can be downloaded.
PutMessage
Message
Specify a DJMessage variable or file through which content can be uploaded.
Supported Action Properties
The following action properties are applicable.
Action
Property
Description
GetMessage
Object Key
Object key that must be downloaded from the S3 bucket. The key is unique within the S3 bucket and may include separator characters, such as slash (/) to emulate folders. This option is mandatory.
Local File Name
Local file name, where the component writes the data retrieved from the S3 bucket. If you do not specify this file name, then the data read from the bucket is inserted into the DJMessage parameter.
Bytes Limit
Limit the number of bytes that are downloaded from the object in the S3 bucket. For example, the value 10000 means that only the first 10000 bytes are downloaded from the source object.
Delete Object After Retrieval
If this value is set to TRUE, then the object is deleted from the S3 bucket after it is successfully retrieved. The object will not be deleted if an error occurs during download. The default value is FALSE.
Note:  This option is displayed only if you have selected Use Existing Bucket for the component properties and have set Delete Object When Finished to FALSE.
PutMessage
ObjectKey
Key that is assigned to the object after it is uploaded to the S3 bucket. The key is unique within the S3 bucket and may include separator characters, such as slash (/) to emulate folders. If this value is not provided, then the component expects to find an override property called ObjectKey within the DJMessage.
Local File Name
Local file name from which the component reads the data that it uploads to the S3 bucket. The value of this option (whether it is blank or populated) is overridden by the LocalFileName DJMessage property. If neither value is set, then the data that will be uploaded is read from the DJMessage body.
Override Properties
You can override the values provided for the object key and local file name in the PutMessage action with DJMessage properties:
• ObjectKey DJMessage property: This header property can be set in the DJMessage argument to the PutMessage action. If provided, it overrides any value that is set in the Object Key option.
• LocalFileName DJMessage property: This header property can be set in the DJMessage argument to the PutMessage action. If provided, it overrides the value set in the Local File Name option. If both are missing, then the uploaded data is read from the DJMessage body.
Errors
The following table describes error codes generated for AWS S3 Queue component.
Action
Code
Description
Connect
ERR_OK
Connect completed successfully with no errors
ERR_OPENERR
Connect failed. Typically associated with LT_ERROR log level. Associated error message describes the specific issue.
Disconnect
ERR_OK
Disconnect completed successfully
ERR_CLOSERR
Associated with warning-level log message. Disconnect is called again after it is already disconnected.
ERR_DELETERR
• Attempt to delete non-temporary bucket. Associated with warning-level log message.
• Attempt to delete one or more objects resulted in error. Associated with warning-level log message.
GetMessage
ERR_OK
Object retrieved and saved (to file or DJMessage) with no errors
ERR_BADOPTIONVALUE
• Object key not set. Associated with error-level log message.
• Local file referenced file already exists. Associated with error-level log message.
ERR_READERR
• Error copying response content to the provided DJMessage. Associated with error-level log message.
• Wide variety of errors produced by Amazon AWS S3 API will produce this error. The error code will include a message appropriate to the error. Typically associated with error-level log entry.
ERR_DELETERR
Delete after download option is set, but the component failed to delete the object from the S3 bucket. Associated with warning-level log entry
PutMessage
ERR_OK
Object uploaded to S3 bucket with no errors
ERR_BADOPTIONVALUE
Target object key property must be set. Associated with error-level log entry.
ERR_WRITERR
• Error copying data from DJMessage. Associated with error-level log entry.
• AWS-generated error occurred while sending or writing data to the remote S3 bucket. A number of messages could be produced. All associated with error-level log entry.
Last modified date: 02/09/2024