Was this helpful?
Azure Storage Queue
Note:  For general information about Queue Steps, see Queue Step.
‎Azure Data Lake Storage (ADLS) Gen2 is a set of capabilities built on Azure Blob storage. Data Lake Storage Gen2 is the result of converging the capabilities of the two existing storage services—Azure Blob storage and Azure Data Lake Storage Gen1. Features from Azure Data Lake Storage Gen1, such as file system semantics, directory, and file level security and scale are combined with low-cost, tiered storage, high availability or disaster recovery capabilities from Azure Blob storage. For more information on Azure Data Lake Storage, see Introduction to Azure Data Lake Storage Gen2.
Actian DataConnect provides the Azure Storage Queue component that can be used to connect to ADLS Gen2. This component supports the following features:
Storage account’s name and shared access key (for authentication)
Container’s specification within the storage account that hosts the data
Selecting Data Lake Storage Service (Blob Storage Service or File Storage Service)
Uploading files or DJMessage content
Downloading objects to DJMessages or files
You can use the ABFS to Actian Data Platform Azure template for loading data to Actian Data Platform. It leverages the Azure Storage Queue component. For more information about this template, see Integration Manager documentation.
 
Azure Storage Queue Properties
 
Property
Default Value
Description
Azure Storage Service Type
 
Select any of the following options for interacting with the selected container:
Blob Storage Service: Component uses Microsoft Azure's Blob-specific APIs to interact with the selected storage account and container. For more information, see Introduction to Azure Blob Storage.
ADLS File System Service: Component uses Microsoft Azure's Data Lake Storage Gen2 hierarchical namespace-specific APIs to interact with the selected storage account and container. For more information, see Azure Data Lake Storage Gen2 hierarchical namespace.
Storage Account Name
 
Name of a pre-existing storage account that is created within a Microsoft Azure account. For more information, see Storage account overview.
Container Name
 
Name of a pre-existing container within the selected storage account.
Delete Objects When Finished
FALSE
If set to TRUE, all files that are uploaded by the component during the process execution are deleted when the disconnect action is called.
Note:  The process engine automatically calls the disconnect action when the process is completed.
Azure Storage Authentication Type
 
Select the authentication scheme that the component will use:
Storage Shared Access Keys: Azure generates two 512-bit storage account access keys when you create a storage account. These keys can be used to authorize access to data in your storage account through Shared Key authorization. The following additional option is displayed:
Access Key: Provide one of the two 512-bit storage account access keys that was automatically generated by Microsoft Azure when the storage account was created.
Azure Active Directory Service Principal With Client Secret: Customer applications, which embed the Azure Storage Component, can be integrated with the Microsoft identity platform by registering it with an Azure Active Directory tenant. After this, a client secret can be generated from the registered application and it can be used for authentication. For more information, see Microsoft’s How to page. Actian Data Platform COPY VWLOAD directly supports this type of registration and authentication. The following additional options are displayed:
Application (Client) ID: Identity assigned to the application after registration. It is the service principal assigned to the application.
Directory (tenant) ID: Provide the tenant ID. A tenant is a representation of an organization in Azure Active Directory. Applications are registered relative to a tenant within Azure Active Directory.
Client Secret: Specify the client secret value. After an application or service principal is created, you can upload a certificate or create a client secret to be used for authentication.
Supported Actions
Action
Description
Connect
Creates an authenticated Azure storage client that is used to perform all subsequent operations.
PutMessage
Uses the authenticated Azure Storage client for uploading data to an Azure Storage container and saves it using the provided blob (or file) name. The content that must be uploaded can be provided to the PutMessage action through DJMessage or a referenced file.
GetMessage
Uses the authenticated Azure storage client to download data from the specified container. The downloaded content can be written to a file or injected into a DJMessage object that is provided to the GetMessage action.
Disconnect
Removes the authenticated Azure storage client. If you have chosen the option to delete uploaded objects, then all the uploaded files are deleted.
Supported Action Parameters
Action
Parameter
Description
GetMessage
Message
Specify a DJMessage variable or file through which content can be downloaded.
PutMessage
Message
Specify a DJMessage variable or file through which content can be uploaded.
Supported Action Properties
The following action properties are applicable.
 
Action
Property
Description
GetMessage
Object Name
Object name that you intend to download from the configured Azure Storage container.
If the component is interacting with the container using the blob service, then the name represents the blob name.
If the component is interacting with the container through the file system service, then the name represents the full path to the file including directory elements separated by the "/" character.
Note:  This option is required within the GetMessage action.
Local File Name
Local file name, where the component writes the data retrieved from the Azure Storage container. If you do not specify this file name, then the data read from the container is inserted into the DJMessage parameter.
Bytes Limit
Limits the number of bytes downloaded from the selected blob or file to the specified value. If you want to download the entire blob or file, do not specify this value.
Delete Object After Retrieval
If set to TRUE, the specified blob or file is deleted from the Azure Storage container after the download is successful. The default value is FALSE.
Note:  This option is displayed only if the Delete Objects When Finished component option's value is FALSE.
PutMessage
Object Name
Name of the blob or file that is assigned to the object after it is uploaded to the Azure Storage container.
If the component is configured to use the File System Service for interacting with the selected Azure Storage container, then the name can include the full path (including directories) to the file. If the path includes missing directories in the container, then the component creates the missing directories.
If the object name is not specified, then the component tries to find an override property called ObjectName within the DJMessage. It is an error condition if the option is left blank and the override property is missing from the DJMessage.
Local File Name
If a value is provided, then the component reads the data that it uploads from the referenced file. An error will be generated if the file cannot be resolved or is inaccessible.
When this option is left blank or populated, it is overridden by the LocalFileName DJMessage property. If neither value is set, then the data that is uploaded is read from the DJMessage body.
Override Properties
You can override the values provided for the object name and local file name in the PutMessage action with DJMessage properties:
ObjectName DJMessage property: This header property can be set in the DJMessage argument to the PutMessage action. If provided, it overrides any value that is set in the Object Name option.
LocalFileName DJMessage property: This header property can be set in the DJMessage argument to the PutMessage action. If provided, it overrides the value set in the Local File Name option. If both are missing, then the uploaded data is read from the DJMessage body.
Errors
The following table describes error codes generated for the Azure Storage Queue component.
Action
Code
Description
Connect
ERR_OK
Connect completed successfully without any errors.
ERR_OPENERR
Connect failed. Typically associated with LT_ERROR log level.
Disconnect
ERR_OK
Disconnect completed successfully.
ERR_CLOSERR
Associated with warning-level log message. Disconnect action is called after the component is already disconnected.
ERR_DELETERR
If the component is configured to delete uploaded objects during connect, this error code is generated when one or more objects that are deleted resulted in error. This is associated with warning-level log message.
GetMessage
ERR_OK
Object retrieved and saved (to file or DJMessage) without any errors.
ERR_BADOPTIONVALUE
Object name is not set. This is associated with error-level log message.
ERR_READERR
Local file or referenced file already exists. This is associated with error-level log message.
ERR_DELETERR
Error in copying response content to the provided DJMessage. This is associated with error-level log message.
Too many errors generated by the Azure Storage SDK produces this error. The error code will include a message appropriate to the error. This is associated with error-level log message.
PutMessage
ERR_OK
Object uploaded to Azure Storage container without any errors.
ERR_BADOPTIONVALUE
Target object name property must be set. This is associated with error-level log entry.
ERR_WRITERR
Error copying data from DJMessage. This is associated with error-level log entry
Azure-generated error occurred when sending or writing data to the remote Azure Storage container. A number of messages can be generated and all these messages are associated with error-level log entry.
Last modified date: 07/26/2024