Adding a Dataiku Connection¶
Prerequisites¶
- A user with sufficient permissions is required to establish a connection with Dataiku.
- Zeenea traffic flows towards the data source must be open.
Note
You can find a link to the configuration template in Zeenea Connector Downloads.
Supported Versions¶
The Dataiku connector is compatible with the software SaaS version.
Installing the Plugin¶
You can download the Dataiku plugin from Zeenea Connector Downloads.
For more information about how to install a plugin, see Installing and Configuring Connectors as a Plugin.
Declaring the Connection¶
Connectors are created and configured through a dedicated configuration file located in the /connections folder of the relevant scanner.
For more information about managing connections, see Managing Connections.
To establish a connection with a Dataiku instance, fill in the following parameters in the dedicated configuration file:
| Parameter | Expected value |
|---|---|
name | Specifies the display name for the connection. |
code | Specifies the unique identifier of the connection on the Zeenea platform. Once registered on the platform, this code must not be modified or the connection will be considered as new and the old one removed from the scanner. |
connector_id | Specifies the type of connector to be used for the connection. The value must be dataiku and must not be modified. |
enabled | Specifies whether to enable or disable the connection (true or false). The default value is true. |
catalog_code | Specifies the catalog code associated with the connection (default when empty). |
alias | Specifies the list of aliases used by other connectors to generate lineage link. For example, ["localhost:1234/db","https://some-url.org"] |
secret_manager.enabled | Specifies whether to enable or disable the secret manager for the connection. This configuration works only with Scanner 73 or later and requires a functional secret manager configured in the scanner configuration file. The default value is true. |
secret_manager.key | Specifies the name of the secret. |
connection.url | Specifies the Dataiku instance URL (for example, https://url.eu-west-3.app.dataiku.io). |
connection.token | Specifies the secret token generated from the Dataiku application. |
tls.truststore.path | Specifies the TLS trust store file path. This file must be provided in case TLS encryption is activated (protocol https) and when certificates of servers are delivered by a specific authority. It must contain the certification chain. |
tls.truststore.password | Specifies the password of the trust store file. |
tls.truststore.type | Specifies the type of the trust store file. Possible values are PKCS12 or JKS. The default value is discovered from the file extension. |
proxy.scheme | Specifies the proxy protocol (http or https). |
proxy.hostname | Specifies the proxy address. |
proxy.port | Specifies the proxy port. |
proxy.username | Specifies the proxy username. |
proxy.password | Specifies the proxy account password. |
User Permissions¶
To collect metadata, the running user's permissions must allow them to access and read datasets that need cataloging.
The user must have read access to Dataiku projects that need cataloging.
Data Extraction¶
To extract information, the connector runs the following REST requests on the API:
- GET:
/public/api/projects/: To get the project list. - GET:
/public/api/projects/{project name}/datasets/: To get dataset from the project. - GET:
/public/api/projects/{project name}/datasets/{dataset name}/: To get metadata dataset. - GET:
/public/api/admin/connection/{connection name}: To get metadata from a connection. Requires administrator privileges. - GET:
/public/api/{project key}/flow/graph: To get all flow metadata (lineage) for a given project. - GET:
/public/api/{project key}/recipe/{recipe name}: To get metadata from a recipe in a given project.
Collected Metadata¶
Inventory¶
The inventory collects the list of datasets accessible by the user.
Lineage¶
The connector can reconstruct the lineage of imported dataset from databases such as MySQL and PostgreSQL. The connector can also retrieve internal Dataiku lineage generated by recipes.
Note
The technical user you configure must have administrator privileges to retrieve all lineage information. For more information, see Dataiku REST API documentation.
Dataset¶
- Name
- Source Description
- Technical Data:
- Type
- Format Type
- Project Key
- Updated At
- Updated By
Field¶
Dataset field.
- Name
- Source Description
- Type
- Can be null: Not supported. Default value
false. - Multivalued: Not supported. Default value
false. - Primary Key: Not supported. Default value
false. - Technical Data:
- Technical Name
- Native type
Object Identification Keys¶
Each object in the catalog is associated with a unique identifier key. When the object is imported from an external system, the key is generated and provided by the connector.
For more information about identifier keys, see Identification Keys.
| Object | Identifier Key | Description |
|---|---|---|
| Dataset | code/project/dataset name | - code: Unique identifier of the connection noted in the configuration file - project: Dataiku project name - dataset name |
| Field | code/project/dataset name/field name | - code: Unique identifier of the connection noted in the configuration file - project: Dataiku project name - dataset name - field name |