Working with DataFlow
DataFlow applications can be configured, scheduled, and executed using Integration Manager if you have an active DataFlow license or subscription. For convenience, DataFlow 7.0 is embedded within Integration Manager powered by DataCloud.
DataFlow Properties
For on-premises worker activation, you simply need to install DataFlow on your worker machine and add a few entries to your Integration Manager, Worker, or Agent application.properties file:
# Dataflow Configuration
dataflow.enabled=true
dataflow.licensePath=${sharedDataPath}/license/df6.slc dataflow.localEngineInstallPath=${sharedDataPath}/dataflow/bin
How to Work with DataFlow
Integration Manager uses the
dr command line interface to execute DataFlow applications. Before creating any DataFlow configurations, you should familiarize yourself with dr interface concepts. You can learn more at:
https://docs.actian.com/dataflow/7.0.2/#page/TroubleshootingRef%2FUsing_dr.htm%23 Three DataFlow application types are supported:
• RushScript (.js)
• JSON Graph (.json)
• Java Application (.jar)
To learn more about creating DataFlow applications, see:
https://docs.actian.com/dataflow/7.0.2/ Configure RushScript (.js)
To configure RushScript
1. Create a new Configuration.
2. Upload your RushScript as the package (it must have a .js extension).
3. (Optional) Upload additional scripts or other files to the Files section, which will be injected using the “--includedir” command line option.
4. (Optional) Add entries to the Macros section, which will be injected using the “--variable” command line option. Note that the macro value text will be escaped accordingly when the command is constructed.
5. You may use these Integration Manager variables within your macro values:
$(LOCAL_JOB_SPEC_DIR)
Resolves to the runtime working directory where your RushScript and additional files are located.
$(LOCAL_JOB_OUT_DIR)
Resolves to the runtime output directory if you need to retrieve output files. Any files written to this location will be available at: {api_url}/jobs/{job-id}/out after the job has completed.
Configure JSON Graph (.json)
To configure JSON Graph
1. Create a new Configuration.
2. Upload your Json Graph as the package (it must have a .json extension).
3. (Optional) Upload a properties file named override.properties to the Files section, which will be injected using the “--overridefile” command line option.
4. (Optional) Add entries to the Macros section, which will be injected using the “--override” command line option. Each macro can only override an operator within the graph, and must use the convention of MacroName=<operatorName.operatorProperty> and MacroValue=<overrideTextValue>. Note that the macro value text will be escaped accordingly when the command is constructed.
5. You may use these Integration Manager variables within your macro values:
$(LOCAL_JOB_SPEC_DIR)
Resolves to the runtime working directory where your RushScript and additional files are located.
$(LOCAL_JOB_OUT_DIR)
Resolves to the runtime output directory if you need to retrieve output files. Any files written to this location will be available at: {api_url}/jobs/{job-id}/out after the job has completed.
Configure a Java Application (.jar)
To configure a Java application
1. Create a new Configuration.
2. Upload your Java application .jar file as the package.
3. Type in the fully qualified main class within the jar as the entry point, for example, com.foo.bar.MainClass, not MainClass.
4. (Optional) Add entries to the Macros section, which will be injected as an environment variable to the JVM. Note that the macro value text will be escaped accordingly when the command is constructed.
Last modified date: 08/22/2022