Adding an Aliyun DataWorks MaxCompute Connection¶
Prerequisites¶
-
A user with sufficient permissions is required to establish a connection with Aliyun DataWorks MaxCompute.
-
Zeenea traffic flows towards the data source must be open.
Note
You can find a link to the configuration template in Zeenea Connector Downloads.
Installing the Plugin¶
The Aliyun DataWorks MaxCompute plugin can be downloaded here: Zeenea Connector Downloads.
For more information on how to install a plugin, please refer to the following article: Installing and Configuring Connectors as a Plugin.
Declaring the Connection¶
Creating and configuring connectors is done through a dedicated configuration file located in the /connections folder of the relevant scanner.
Read more: Managing Connections
In order to establish a connection with a Aliyun DataWorks MaxCompute instance, specifying the following parameters in the dedicated file is required:
| Parameter | Expected value |
|---|---|
name | The name that will be displayed to catalog users for this connection. |
code | The unique identifier of the connection on the Zeenea platform. Once registered on the platform, this code must not be modified or the connection will be considered as new and the old one removed from the scanner. |
connector_id | The connector type to use for the connection. Here, the value must be aliyun-dataworks-maxcompute and this value must not be modified. |
connection.region_id | The region ID. The complete list is available here. |
connection.access_key_id | The user's access key id. This parameter is optional, the key can be provided through the environment variable ALIBABA_CLOUD_ACCESS_KEY_ID.If both are defined, the environment variable will be used. |
connection.access_key_secret | The secret associated to the access key. This parameter is optional, the key can be provided through the environment variable ALIBABA_CLOUD_ACCESS_KEY_SECRET.If both are defined, the environment variable will be used. Example: export ALIBABA_CLOUD_ACCESS_KEY_SECRET="abc654dqzER54C15674EZcazd" |
| Quota | |
quota.max_retry | Maximum number of retries when a request encounters a quota expiration error. Default value: 3. |
quota.timeout_minute | Maximum waiting time when waiting for the availability of a quota, in minutes. Default value: 1 minute. |
quota.call_per_min | Calls per minute quota value. Default value: 50. |
| Proxy | |
proxy.scheme | Depending on the proxy, http or https |
proxy.hostname | Proxy address |
proxy.port | Proxy port |
proxy.username | Proxy username |
proxy.password | Proxy account password |
User Permissions¶
To collect the items, the technical user must be added to each project that needs cataloging.
Data Extraction¶
To extract information, the connector runs requests on following services:
-
ListProjects: Retrieve all projects.
-
GetMetaDBTableList: Retrieve all project's tables.
-
GetMetaTableBasicInfo: Retrieve table's info.
-
GetMetaTableColumn: Retrieve all table's columns.
-
GetMetaTableLineage: Retrieve all table's lineage.
Collected Metadata¶
Inventory¶
Will collect the list of tables and views accessible by the user.
Dataset¶
A dataset can be a table or a view.
-
Name
-
Source Description
-
Technical Data:
-
Type
-
Creation Date
-
Last Modification Date
-
Field¶
Dataset field.
-
Name
-
Source Description
-
Type
-
Can be null: Not supported. Default value
FALSE. -
Multivalued: Not supported. Default value
FALSE. -
Primary Key: Depending on the field's "Primary Key" attribute.
-
Technical Data:
- Native type
Data Processes¶
To represent the data flow between tables, a Zeenea Data Process will be created for a table and its ancestors.
- Name:
import dataset_name
Unique Identification Keys¶
A key is associated with each item of the catalog. When the object comes from an external system, the key is built and provided by the connector.
More information about how it works can be found here: Identification Keys.
| Object | Identifier Key | Description |
|---|---|---|
| Dataset | code/table ID | - code: Unique identifier of the connection noted in the configuration file - table ID: Unique ID defined by the source system |
| Field | code/column ID | - code: Unique identifier of the connection noted in the configuration file - column ID: Unique ID defined by the source system |
| Data process | code/import/output table ID | - code: Unique identifier of the connection noted in the configuration file - output table ID: Unique ID of the output table |