Configuring and Executing DataFlow Jobs

DataFlow applications can be configured, scheduled, and executed using Integration Manager if you have an active DataFlow subscription.

For convenience, DataFlow 6.9 is embedded within Integration Manager powered by DataCloud.

For on-premises worker activation, you simply need to install DataFlow on your worker machine and add a few entries to your Integration Manager or Worker-Agent application.properties file:

dataflow.localEngineInstallPath=${sharedDataPath}/actian-dataflow-6.9.0-1/bin

Integration Manager uses the dr command line interface to execute DataFlow applications. Before creating any DataFlow configurations, you should familiarize yourself with the dr interface concepts. You can learn more about these concepts at https://docs.actian.com/dataflow/6.9/#page/TroubleshootingRef%2FUsing_dr.htm%23.

To learn more about creating DataFlow applications, see https://docs.actian.com/dataflow/6.9/.

1. Create a new Configuration.

2. Upload your RushScript as the package (must have .js extension).

3. Optionally upload additional scripts or other files to the Files section, which will be injected using the "--includedir" command line option.

4. Optionally add entries to the Macros section, which will be injected using the "--variable" command line option. Note that the macro value text will be escaped accordingly when the command is constructed.

5. You can also take advantage of 2 special Integration Manager variables within your macro values:

• $(LOCAL_JOB_SPEC_DIR) - This will resolve to the runtime working directory where your RushScript and additional files are located.

• $(LOCAL_JOB_OUT_DIR) - This will resolve to the runtime output directory if you need to retrieve output files. Any files written to this location will be available at: {api_url}/jobs/{job-id}/out after the job has completed.

1. Create a new Configuration.

2. Upload your Json Graph as the package (must have .json extension).

3. Optionally upload a properties file named "override.properties" to the Files section, which will be injected using the "--overridefile" command line option.

4. Optionally add entries to the Macros section, which will be injected using the "--override" command line option. Each macro can only override an operator within the graph, and MUST use the convention of MacroName=<operatorName.operatorProperty> and MacroValue=<overrideTextValue>. Note that the macro value text will be escaped accordingly when the command is constructed.

5. You can also take advantage of 2 special Integration Manager variables within your macro values:

• $(LOCAL_JOB_SPEC_DIR) - This will resolve to the runtime working directory where your RushScript and additional files are located.

1. Create a new Configuration.

2. Upload your Java Application .jar file as the package.

3. Select your main class within the jar as the entry point.

Note that additional Files and Macros are not supported with Java Application configurations. They must be self-contained and able to discover any resources they need.