User Guide > Process Steps and Components > Queue Step > Google Cloud Storage Queue
Was this helpful?
Google Cloud Storage Queue
Note:  For general information about Queue Steps, see Queue Step.
The Google Cloud Storage Queue component facilitates integration with Google Cloud Platform (GCP) Storage services. The key component features are:
• Specify the name of the GCP Storage bucket which will store the objects being written and read. Optionally, create a bucket if missing.
Note:  A missing bucket will be created only if the component uploads content to the bucket. A new bucket will not be created if the component is downloading content.
• Specify the storage location (region) of the bucket.
• Select authentication methods:
• Supports Service account keys contained within a file, which is provided by the GCP Storage account owner
• Supports the following schemes for finding the key file:
– Explicitly specify the file path
– Implicitly discover the file through the GOOGLE_APPLICATION_CREDENTIALS environment variable
• Upload files or DJMessage content to GCP Storage buckets
• Download "objects" from GCP Storage buckets to DJMessages or files
For GCP overview, see https://cloud.google.com/. For information about GCP Storage services, see https://cloud.google.com/storage.
 
Google Cloud Storage Queue Property Descriptions
 
Property
Default Value
Description
Bucket Name
 
GCP Storage bucket that will be used to upload and retrieve stored objects.
The value must match the name of an existing GCP Storage bucket for which the service account has appropriate privileges.
If the name does not match an existing bucket, then an error occurs unless the Create Missing Bucket property is set to TRUE. For more information about GCP Storage buckets, see https://cloud.google.com/storage/docs/key-terms.
Create Missing Bucket
FALSE
If set to TRUE, the component creates the specified bucket (if it is missing) when attempting to upload (PutMessage action) content to the specified bucket. This setting has no effect if the component is only downloading content. The default value is FALSE.
Bucket Location
 
Location for storing your object data when you create a bucket. The location types are:
• Region: Specific geographic place, such as London.
• Dual-region: Specific pair of regions, such as Finland and the Netherlands.
• Multi-region is a large geographic area, such as the United States, that contains two or more geographic places.
Objects stored in a multi-region or dual-region are geo-redundant.
Cloud Storage stores object data in the selected location in accordance with the Service Specific Terms.
For more information about locations, see https://cloud.google.com/storage/docs/locations.
This property is displayed only when the Create Missing Bucket property is set to TRUE.
Bucket Storage Class
Standard Storage
When creating a missing bucket, the component must know the GCP Storage class to be used. The options are:
• Standard Storage
• Nearline Storage
• Coldline Storage
• Archive Storage
For more information about storage classes, see https://cloud.google.com/storage/docs/storage-classes.
This property is displayed only when the Create Missing Bucket property is set to TRUE.
Authentication Type
Use path in environment variable (GOOGLE_APPLICATION_CREDENTIALS)
Select any of the following authentication type:
• Use path in environment variable (GOOGLE_APPLICATION_CREDENTIALS): Instructs the component to configure the GCP client SDK in a way, which instructs it to implicitly find the service account key file using the GOOGLE_APPLICATION_CREDENTIALS environment variable. If the environment variable is set, then its value is expected to be the path to the file. If the environment variable is not set, then an error occurs.
• Provide path to service account key file: Displays the Service Account Key File property using which you can specify the complete path of the service account key file.
• Directly provide service account key: Allows you to directly provide the service account keys. This is a non-file based authentication method.
Service Account Key File
 
Allows you to browse and select the Service Account Key file.
This property is displayed only when the Authentication Type property is set to Provide path to service account key file.
Service Account Email Address
 
(Required) Client email address of the service account from the GCP console.
This property is displayed only when the Authentication Type property is set to Directly provide service account key.
PKCS8 Formatted Private Key
 
(Required) RSA private key object for the service account in PKCS#8 format.
This property is displayed only when the Authentication Type property is set to Directly provide service account key.
Service Account Client ID
 
(Optional) Service account client ID.
This property is displayed only when the Authentication Type property is set to Directly provide service account key.
Private Key ID
 
(Optional) Private key identifier for the service account.
This property is displayed only when the Authentication Type property is set to Directly provide service account key.
Delete New Objects When Finished
FALSE
Automatically deletes all uploaded objects when process completes.
Supported Actions
Action
Description
Connect
Creates an authenticated GCP Storage client, which will be used to perform all subsequent operations. The client is stateless, so it is not actually "connected."
Disconnect
Disconnects from the authenticated client.
GetMessage
Uses the authenticated client to download data from a GCP Storage bucket. The data in the bucket is referenced using an object name. The downloaded content can be written to a file or injected into a DJMessage object which is provided to the GetMessage action.
PutMessage
Uses the authenticated client to upload data to a GCP Storage bucket, saving it using the provided object name. This object name is equivalent to an S3 "key" or local "file name". The content to be uploaded can be provided to the PutMessage action via DJMessage or a referenced file.
Supported Action Parameters
Action
Parameter
Description
GetMessage
Message
Name of a DJMessage to which the downloaded file will be written if a local file was not specified.
PutMessage
Message
Name of a DJMessage from which data will be uploaded if a local file was not specified.
 
Supported Action Properties
Action
Property
Description
GetMessage
Object Name
Name of the object to retrieve from GCP Storage bucket.
The name is unique within the bucket and may include separator characters, such as slash ("/") to emulate folders. This option is required.
Local File Name
Click Browse and specify a local file, to which the component will write the data retrieved from the bucket.
Bytes Limit
Number of bytes to download from the object in the bucket. This is useful for sampling. Leave this blank if you want to download the entire object.
PutMessage
Object Name
Name of the GCP Storage object to be written.
The name is unique within the bucket and may include separator characters, such as slash ("/" ) for emulating folders.
If left blank, then the component expects to find an override property called "ObjectName" within the DJMessage. It is an error condition if the option is left blank and the override property is missing from the DJMessage.
Local File Name
Click Browse and specify the local file from where the component will read the data that it uploads to the bucket.
The value of this option (whether it is blank or populated) is overridden by the LocalFile DJMessage property. If neither value is set, then the data that will be uploaded is read from the DJMessage body.
MIME Type
MIME type that will be assigned to the object after it is uploaded to the bucket.
This is a useful option if the content is being uploaded from the body of a DJMessage. It can be overridden through the DJMessage property MimeType.
Note:  MIME Type will be automatically determined when uploading from a local file.
A limited list of MIME types is available for selection. However, you can specify any MIME type supported by GCP Storage. The available MIME Types are:
• text/xml
• text/html
• text/plain
• application/json
• application/xml
• application/octet-stream
Replace Existing Object
If set to TRUE, the existing object is replaced on GCP Storage. The default value is FALSE.
Override Queue Properties
You can override the values provided for the object name and local file name in the PutMessage action using the following DJMessage properties:
• ObjectName: This header property can be set in the DJMessage argument for the PutMessage action. If provided, it overrides any value that is set in the Object Name option.
• LocalFile: If the DJMessage argument for the PutMessage action has a header property with this name, then its value will override any value that is set in the Local File Name option. If both are missing, then the upload data will be read from the DJMessage body.
• MimeType: This header property can be set in the DJMessage argument for the PutMessage action. If provided, it overrides the value set in the MIME Type option.
• ReplaceFile: If the DJMessage argument for the PutMessage action has a header property with this name, then its value will override any value that is set in the Replace Existing Object option. If neither value is set, then the default value is FALSE.
Errors
The following table describes the error codes generated for the Google Cloud Storage Queue component.
 
Action
Code
Description
Connect
ERR_OK
Connect completed successfully without any errors
ERR_OPENERR
Connect failed. Typically associated with LT_ERROR log level.
Disconnect
ERR_OK
Disconnect completed successfully.
 
ERR_CLOSERR
Associated with warning-level log message. Disconnect action is called after the component is already disconnected.
GetMessage
ERR_OK
File retrieved and saved (to file or DJMessage) without any errors.
 
ERR_BADOPTIONVALUE
GCP storage file name is not set. This is associated with error-level log message.
 
ERR_READERR
Local file or referenced file already exists. This is associated with error-level log message.
PutMessage
ERR_OK
File uploaded to the GCP Storage bucket without any errors.
ERR_BADOPTIONVALUE
Object name property must be set. This is associated with error-level log entry
ERR_WRITERR
Error copying data from DJMessage.
Google-generated error occurred when sending or writing data to the remote GCP Storage account. A number of messages can be generated and all these messages are associated with error-level log entry.
Last modified date: 02/09/2024