Was this helpful?
Remote File Configuration
The dr.fileConfigurations() method is used to create a collection of remote file configurations that provides the cluster environment with the appropriate information and credentials for the current graph. The method returns a FileConfiguration object that can be used to define the set of required filesystem credentials. Once the required properties have been set, the dr.setFileConfigurations() method can be used to specify the fileConfiguration that must be used by the graph. The FileConfiguration object supports the following methods:
Function Name
Input Parameters
Returns
Description
defaultConfiguration
String:root path
 
Creates a default configuration for the path location.
configurationProperty
String:root path, String:field, String:values
 
Set a credential field for the associated filesystem.
The dr.setFileConfigurations() method can be used to set the particular set of filesystem configurations that will be used during graph execution. The following example demonstrates creating and setting the file configuration. See Supported File Systems for more information about the particular field properties that are supported by various filesystems.
Setting Remote File Configurations
var s3FileConfigs = dr.fileConfigurations()
.defaultConfiguration("hdfs://")
.configurationProperty("s3://", "access key", "XXXXXXXXXX")
.configurationProperty("s3://", "secret key", "YYYYYYYYYY")
.configurationProperty("s3a://", "access key", "XXXXXXXXXX")
.configurationProperty("s3a://", "secret key", "YYYYYYYYYY");
dr.setFileConfigurations(s3FileConfigs);
Last modified date: 06/14/2024