Adding an Atlas Connection¶

Prerequisites¶

In order to establish a connection to an Atlas data catalog, the user must have sufficient permissions.
Actian Data Intelligence traffic flow towards Atlas must be open.

Note

You can find a link to the configuration template in Connector Downloads.

Supported Versions¶

This connector may be used on an Atlas catalog linked through its v2 API to a Hadoop cluster.

Installing the Plugin¶

Since scanner version 26.0, the Atlas plugin can be downloaded here: Connectors: download links

For more information on how to install a plugin, refer to the following article: Installing and Configuring Connectors as a Plugin.

Declaring the Connection¶

Creating and configuring connectors is done through a dedicated configuration file located in the /connections folder of the relevant scanner.

Read more: Managing Connections

In order to establish a connection with Atlas, specifying the following parameters in the dedicated file is required:

Parameter	Expected value
`name`	The name that will be displayed to catalog users for this connection.
`code`	The unique identifier of the connection on the Actian Data Intelligence Platform. Once registered on the platform, this code must not be modified or the connection will be considered as new and the old one removed from the scanner.
`connector_id`	The type of connector to be used for the connection. Here, the value must be `Atlas` and this value must not be modified.
`connection.url`	URL used to connect to the application
`connection.username`	Atlas user used to log into Actian Data Intelligence. Must have sufficient permissions
`connection.password`	User password
`connection.authentication_scheme`	Method to be used for authentication with the Atlas server. Possible values: `Basic`, `Digest`, `Ntlm`, `Spnego`, `Kerberos`. Default value: `Basic`.
`connection.max_concurrent_request`	Maximum concurrent requests to Atlas (default: 16)
`connection.request_timeout`	Request timeout in seconds (default: 60 s)
`connection.request_page_size`	Request page size (default: 100)
`connection.item_reference_bulk_chunk_size`	To improve import performance, the connector retrieves the processes associated with an item (lineage) from Atlas in bulk by grouping multiple entity identifiers (GUIDs) into a single request instead of sending one request per entity. This parameter sets the maximum number of GUIDs sent in each bulk request. If a bulk request fails or the Atlas server does not support bulk retrieval, the connector automatically falls back to loading the entities one by one. The value must be a positive integer. Default value: 50.
`api.min_ext_info.atlas_types`	Allows adding the `minExtInfo` parameter when calling the Atlas entity retrieval service. Allowed values : `hive_table`, `hbase_table`, `hive_process`, `hdfs_path`, `spark_process`, `sqoop_process`, or `sqoop_dbdatastore`.
`cache.path`	Directory where caches are stored
`cache.inventory.enable`	Enable Inventory cache
`cache.inventory.expiration_delay`	Inventory Cache expiration Delay (format ISO-8601: PnDTnHnMn.nS)

Some additional settings, dedicated to improving performance, are available in the Atlas connector configuration file.

Finally, the connector can be configured to filter items (by type) during inventory and import. Without specific configuration, all items are taken into account. Both parameters must have as their value a list of types, separated by a comma, between double-quotes. Example: "hdfs_path, sqoop_process, sqoop_dbdatastore".

The available types are listed below:

inventoried types
hive_process
spark_process
sqoop_process
sqoop_dbdatastore

Parameter	Value
`filter.atlas_type.includes`	Inclusion list. All types included if not defined.
`filter.atlas_type.excludes`	Exclusion list. No exclusion if not defined.

Extracting Metadata¶

In order to gather metadata from Atlas, the connector first references datasets and fields, but it will also retrieve the Data Process used in Atlas to build a Dataset. This allows Actian Data Intelligence to build the entirety of the process, and the lineage.

Important

The Atlas connector behavior is very specific, and must be taken into account when deploying the connector. When a dataset is imported or updated, the connector will also import:

The selected Dataset
Any Data Process of which the selected dataset is an output
Any Data Process of which the selected Dataset is an input

Note that, in the case of updates, they are by default scheduled to be done daily.

Collected Metadata¶

Inventory¶

The inventory collects the list of data sources that the user can access.

Dataset¶

The Datasets are data sources.

Name
Source Description
Technical Data:
- GUID: Atlas technical id
- Qualified Name: Atlas technical name
- Created: creation date
- Created By
- Data Source Type: Data source type
- Owner
- Updated: date of last update
- Updated By
- Database: optional
- Table: optional
- Display Name: optional

Field¶

Name
Source Description
Type
Can be null: false (default)
Multivalued: true when the field represents a table, false otherwise
Primary Key: false (default)
Technical Metadata: none

Data Process¶

All documented lineages in Atlas.

Name
Input: Datasets in input
Output: Datasets in output

Object Identification Keys¶

An identification key is associated with each object in the catalog. In the case of the object being created by a connector, the connector builds it.

Object	Identification Key	Description
Dataset	code/type/dataset identifier	- code: Unique identifier of the connection noted in the configuration file - type: Object type, possible values: `hbase_table`, `hdfs_path`, `hive_table` - dataset identifier: Atlas attribute `qualifiedName`
Field	code/type/dataset identifier/field name	- code: Unique identifier of the connection noted in the configuration file - type: Object type, possible values: `hbase_table`, `hdfs_path`, `hive_table` - dataset identifier: Atlas attribute `qualifiedName` - field name
Data process	code/type/data process identifier	- code: Unique identifier of the connection noted in the configuration file - type: Object type, possible values: `hive_process`, `spark_process`, `sqoop_process` - data process identifier: Technical data process identifier calculated from data process outputs