Remote File Configuration
The dr.fileConfigurations() method is used to create a collection of remote file configurations that provides the cluster environment with the appropriate information and credentials for the current graph. The method returns a FileConfiguration object that can be used to define the set of required filesystem credentials. Once the required properties have been set, the dr.setFileConfigurations() method can be used to specify the fileConfiguration that must be used by the graph. The FileConfiguration object supports the following methods:
The dr.setFileConfigurations() method can be used to set the particular set of filesystem configurations that will be used during graph execution. The following example demonstrates creating and setting the file configuration. See
Supported File Systems for more information about the particular field properties that are supported by various filesystems.
Setting Remote File Configurations
var s3FileConfigs = dr.fileConfigurations()
.defaultConfiguration("hdfs://")
.configurationProperty("s3://", "access key", "XXXXXXXXXX")
.configurationProperty("s3://", "secret key", "YYYYYYYYYY")
.configurationProperty("s3a://", "access key", "XXXXXXXXXX")
.configurationProperty("s3a://", "secret key", "YYYYYYYYYY");
dr.setFileConfigurations(s3FileConfigs);
Last modified date: 06/14/2024