DF 8.2 | DataFlow Operator Library

Building DataFlow Applications > Building DataFlow Applications > Building DataFlow Applications in Java > DataFlow Operator Library

Was this helpful?

DataFlow Operator Library

The DataFlow operator library consists of prebuilt operators that come with DataFlow and can be used within the DataFlow API or RushScript. The library is organized into various categories, listed below, with each category covering the associated operators. Each operator page includes details such as usage examples, properties, and port information.

Input Output Operations - The DataFlow operator library includes several pre-built Input/Output operators. For more information, refer to the following topics:

• ByteSource and ByteSink I/O Operators

– About ByteSource Implementations

– BasicByteSource Operator -

– ConcatenatedByteSource Operator

– GlobbingByteSource Operator

– About ByteSink Implementations

– BasicByteSink Operator

– ReadParquet Operator

– ReadFromJDBC Operator

– ReadDelimitedText Operator

– ReadFixedText Operator

– ReadSource Operator

– ReadLog Operator

– ReadARFF Operator

– ReadStagingDataset Operator

– ParseTextFields Operator

– ReadJSON Operator

• Write I/O Operators

– WriteAvro Operator

– WriteToJDBC Operator

– BinaryWriter Operator

– DeleteFromJDBC Operator

– UpdateInJDBC Operator

– WriteDelimitedText Operator

– WriteFixedText Operator

– WriteSink Operator

– WriteStagingDataset Operator

– WriteARFF Operator

– ForceRecordStaging Operator

– WriteORC Operator

• Vector Operators

– LoadActianVector Operator

– ReadActianVector Operator

• HBase Operators

– DeleteHBase Operator

– ReadHBase Operator

– WriteHBase Operator

SQL-Like Operations - The DataFlow operator library offers a set of operators for performing SQL or relational operations. Using DataFlow, data can be retrieved from various sources, and joined, filtered, aggregated, or reordered as needed. This flexibility enables working with data using common relational concepts and capabilities, without the limitation of having all data in a single location.

The operators that implement relational functionality are composed and configured in the same way as other operators within DataFlow. Some allow for a more SQL-like syntax in their configuration, providing a smoother transition for users familiar with SQL. For more information, refer to the following topics:

• Data Aggregation Operator

– Group Operator

• Data Filtering and Sampling Operators

– FilterRows Operator

– FilterExistingRows Operator

– LimitRows Operator

– SampleRandomRows Operator

• Data Merging Operators

– Join Operator

– CrossJoin Operator

– UnionAll Operator

• Data Sorting Operator

– Sort Operator

• Data Processing Operator

– ProcessByGroup Operator

Data Manipulation Operations - The DataFlow operator library includes several operators for manipulating the fields within record ports. For more information, refer to the following topics:

• Data Manipulation Operators

– DeriveFields Operator

– DiscoverEnums Operator

– MergeFields Operator

– RemoveFields Operator

– RetainFields Operator

– SelectFields Operator

– RemapFields Operator

– SplitField Operator

– RowsToColumns Operator

– ColumnsToRows Operator

Data Analytics Operations - The Dataflow operator library includes several pre-built analytics operators. For more information, refer to the following topics:

• Association Rule Mining Operators

– FrequentItems Operator

– FPGrowth Operator

– ConvertARMModel Operator

• Cluster Analysis Operators

– KMeans Operator

– ClusterPredictor Operator

• Decision Tree Operators

– DecisionTreeLearner Operator

– DecisionTreePredictor Operator

– DecisionTreePruner Operator

• DiscoverDomain Operator

– DiscoverDomain Operator

• RunRScript Operator

– RunRScript Operator

• KNNClassifier Operator

– KNNClassifier Operator

• Naive Bayes Operators

– NaiveBayesLearner Operator

– NaiveBayesPredictor Operator

• PMML Operators

– ReadPMML Operator

– WritePMML Operator

• Regression Analysis

– LinearRegressionLearner Operator

– RegressionPredictor Operator

– LogisticRegressionLearner Operator

– LogisticRegressionPredictor Operator

Statistical Operations - The DataFlow operator library includes several pre-built operators for statistics and data summarization. For more information, refer to the following topics:

• Statistical Operators

– DataQualityAnalyzer Operator

– SummaryStatistics Operator

– DistinctValues Operator

– NormalizeValues OperatorRank Operator

– SumOfSquares Operator

– CountRanges Operator

– EqualRangeBinning Operator

– MostFrequentValues Operator

Support Vector Machine Operations - The DataFlow operator library includes operators for working with support vector machines (SVMs), which are supervised learning models used to analyze data and recognize patterns, often for data classification. For more information, refer to the following topics:

• Support Vector Machine Operators

– SVMLearner Operator

– SVMPredictor Operator

Text Processing Operations - The DataFlow operator library includes several operators for text processing, which involves extracting information from unstructured text. This is achieved by first structuring the text in a format suitable for analysis, followed by applying various transformations and statistical techniques. For more information, refer to the following topics:

• Text Processing Operators

– TextTokenizer Operator

– CountTokens Operator

– FilterText Operator

– DictionaryFilter Operator

– ConvertTextCase Operator

– TextStemmer Operator

– ExpandTextTokens Operator

– CalculateWordFrequency Operator

– CalculateNGramFrequency Operator

– TextFrequencyFilter Operator

– ExpandTextFrequency Operator

– GenerateBagOfWords Operator

– DrawDiagnosticsChart Operator

Data Cleansing Operations - The DataFlow operator library includes operators designed for data cleansing. For more information, refer to the following topics:

• Data Cleansing Operators

– RemoveDuplicates Operator

– ReplaceMissingValues Operator

Data Matching Operations - The DataFlow operator library includes several pre-built fuzzy matching operators, which can be used to identify duplicates or establish links between records. For more information, refer to the following topics:

• Data Matching Operators

– DiscoverLinks Operator

• Data Clustering Operators

– ClusterDuplicates Operator

– ClusterLinks Operator

• Match Analysis Operators

– AnalyzeDuplicateKeys Operator

– AnalyzeLinkKeys Operator

Assertion Operations - The DataFlow operator library includes several operators that can be used for asserting various data conditions. While these are primarily used for testing, they can also be useful for other purposes. For more information, refer to the following topics:

• Assertion Operators

– AssertEqual Operator

– AssertEqualTypes Operator

– AssertPredicate Operator

– AssertRowCount Operator

– AssertSorted Operator

– AssertEqualHash Operator

Data Capturing Operations - The DataFlow operator library includes several operators that allow the executing application to access data within the application directly. For more information, refer to the following topics:

• Data Capturing Operators

– CollectRecords Operator

– GetModel Operator

– EmitRecords Operator

– PutModel Operator

– LogRows Operator

Generating Additional Data Operations - The DataFlow operator library includes a set of operators for generating data tokens to be used within a DataFlow application. For more information, refer to the following topics:

• Generating Additional Data Operators

– GenerateConstant Operator

– GenerateRandom Operator

– GenerateRepeatingCycle Operator

– GenerateArithmeticSequence Operator

Partitioning Data Operations - The DataFlow framework allows operators to specify their data needs through metadata. For more information, refer to the following topics:

• Partitioning Data Operators

– PartitionHint Operator

– GatherHint Operator

– Randomize Operator

Script Processing Operations - The DataFlow operator library includes a set of operators that can be used to process rows using user-defined scripts. For more information, refer to the following topics:

• Script Processing Operators

– RunScript Operator

– RunJavaScript Operator

Last modified date: 03/10/2025