The Dataflow operator library contains several pre-built analytics operators. This section covers each of those operators and provides details on how to use them.

Association Rule Mining (ARM) is an unsupervised learning technique for finding relationships of items contained within transactions. ARM is typically applied to sales transactions where transactions contain line items. Each line item provides a transaction identifier and an item identifier. ARM uses this relationship (transaction to item) to find relationships between sets of items. The two main results of ARM are frequent item sets and association rules.

Frequent item sets capture sets of items that are considered frequent along with some statistics about the sets. Item and item sets are considered frequent if they meet a minimum support threshold. The minimum support is usually quantified as a percentage of transactions within a data set. For example, setting the minimum support to 5% implies that an item is only frequent if it appears in 5% of the transactions within a data set. If a data set contains 5 million transactions (unique transactions, not just records), then an item must appear in at least 250,000 transactions to be considered frequent.

Item sets can have a cardinality of 1 (meaning one item per set) up to a maximum value, usually denoted as k. Frequent item sets are useful in finding sets of items that appear together in transactions and may be opportunities for up-sell and cross-sell activities.

Association rules are another result of ARM. Association rules provide information about frequent item sets and how they interact with each other. From association rules, it can be determined how the presence of one set of items within a transaction influences the presence of another set of items. The rules consist of:

DataFlow provides operators for calculating the frequent items and the association rules. If only the frequent items are wanted, use the FrequentItems operator. For a combination of the frequent items and association rules, use the FPGrowth operator. Using the ConvertARMModel Operator to Convert Association Models from PMML can be used to convert an association PMML model into other formats.

The FrequentItems operator computes the frequency of items within the given transactions. Two fields are needed in the input flow: one to identify transactions and another to identify items.

A minimum support value must also be specified as a percentage of the transactions an item must participate in to be considered frequent. Setting this value’s threshold lower will typically return more data. An optional label field may also be specified that will be used for display purposes instead of the transaction identifiers.

The main output of this operator is the set of frequent items. Two fields are contained in the output: the item field from the input and the frequency count of the items that are above the minimum support value.

The operator also outputs a PMML port that contains an association model. The model will be partially filled in with frequent items and transaction statistics.

In this example we are using a movie ratings data set to determine which movies have been rated by at least 40% of the users.

The following example demonstrates using the operator in RushScript. Note that the operator has two output ports: the frequent items and a PMML model. Access the returned data set container variable with the port names to access a specific port.

The FrequentItems operator provides the following properties.

The FrequentItems operator provides a single input port.

The FrequentItems operator provides the following output ports.

The FPGrowth operator implements the FP-growth algorithm, creating a PMML model that contains generated item sets and association rules. The FP-growth algorithm is optimized to require only two passes over the input data. Other ARM algorithms such as apriori require many passes over the data.

The input data is required to have two fields: a transaction identifier and an item identifier. The transaction identifier discriminates transactions. The item identifier indicates items within transactions.

Transactions are assumed to be in line item order (one input record per item in a transaction). Transactions records are also assumed to be contiguous. An optional label field may also be specified that will be used for display purposes instead of the transaction identifiers.

The operator has two outputs: the frequent item sets and a PMML model. The frequent item sets are output in pivoted form with a set identifier and one item per row. The output model is a PMML-based association model. The model contains frequent items, frequent item sets, and association rules.

In this example, the BPM sample data is read and used as input into the FPGrowth operator. Downstream from the FPGrowth operator, the PMML model is persisted to a local file. The PMML model contains the frequent items, frequent item sets, association rules, and various metrics about the model.

The FPGrowth operator provides the following properties.

The FPGrowth operator provides a single input port.

The FPGrowth operator provides two output ports.

The ConvertARMModel operator converts a PMML model created by Using the FPGrowth Operator to Determine Frequent Pattern Growth into other formats. PMML is a standard format for data mining models developed by the DMG organization.

The operator currently supports the output format: GEXF. This format is a standard that is accepted by many graph visualization tools. GEXF was originally developed for the open source Gephi tool. The format is also supported by other visualization tools.

When converting into the GEXF format, the frequent item sets of the PMML model are used. Each frequent item is added as a node in the graph. Attributes are added to each node indicating the support metric for the item.

Each relationship between items is added as an edge. Edges are weighted by how many times items are found in frequent item sets together. After the GEXF file is created, the Gephi tool can be launched to visualize the results. The Gephi tool is not included as part of DataFlow. You can download it from http://gephi.org/.

A path name is required to specify where to persist the converted model. When executed, the operator applies the specified conversion of the input PMML model into the specified format. The results are written to the specified file. If the file exists already, it will be overwritten.

The ConvertARMModel operator has the following properties.

The ConvertARMModel operator provides a single input port.

The ConvertARMModel operator has no output ports.

Cluster analysis includes techniques for grouping entities based on their characteristics as well as describing the resulting groups also known as clusters. This is done by applying cluster algorithms to attributes of entities that make them similar. The goal is to create clusters such that entities that belong to the same cluster are similar with regard to the relevant attributes, but entities from two different clusters are dissimilar.

Cluster analysis is a common task in data mining as well as statistical data analysis. It helps to organize observed data and build taxonomies.

The KMeans operator performs k-means computation for the given input. All included fields in the given input must be of type double. The k-means algorithm chooses a random k data points as the initial centroids. Computation ends when one of the following conditions is true:

The following example demonstrates computing the clusters for the Iris data set using k-means. The input data contains 4 numeric fields but only 3 are used and so must be explicitly specified.

The KMeans operator provides the following properties.

The KMeans operator provides a single input port.

The KMeans operator provides the following output port.

The ClusterPredictor operator assigns input data to clusters based on the provided PMML clustering model. The explicit cluster IDs will be used for the assignment if the model provides any. Otherwise, the implicit 1-based index, indicating the position in which each cluster appears in the model will be used as ID.

The input data must contain the same fields as the training data that was used to build the model (in the PMML model: clustering fields with the attribute "isCenterField" set to "true") and these fields must be of type double, float, long, or int. The resulting assignments will be part of the output alongside with the original input data.

The ClusterPredictor operator provides the following properties.

The ClusterPredictor operator provides the following input ports.

The ClusterPredictor operator provides a single output port.

Decision tree learning is a method that uses inductive inference to approximate a target function, which will produce discrete values.

Decision trees, also referred to as classification trees or regression trees, can be used in decision analysis to visually and explicitly represent decisions and decision making. It is generally best suited to problems where the instances are represented by attribute-value pairs. Also, if each attribute has a small number of distinct values, it is easier for the decision tree to reach a useful solution. A decision tree learner will produce a model that consists of an arrangement of tests that provide an appropriate classification for an instance of data at every step in an analysis.

The decision tree implemented in DataFlow is a classification tree. A classification tree can be used to classify a particular instance by sorting them down the tree from the root node to some leaf node, which then provides the classification of the instance.

Each node in the tree specifies a test of some attribute of the instance, and each branch descending from that node corresponds to one of the possible values for this attribute.

An instance in a specific data set is classified by starting at the root node of the tree, testing the attribute specified by this node, and then moving down the branch corresponding to the value of the attribute. This process is then repeated at the node on this branch and so on until a leaf node is reached and the instance can be classified.

DataFlow provides operators to produce and use decision tree models. The learner is used to determine the classification rules for a particular data set while the predictor can apply these rules to a data set. There is also an operator that can be used to prune a decision tree when the model provided is overly complex and does not generalize the data well.

The DecisionTreeLearner is the operator responsible for constructing a decision tree PMML model. The implementation is based primarily on Ross Quinlan’s book, C4.5: Programs for Machine Learning. Quinlan’s C4.5 implementation (and this implementation) have the following key features and limitations:

The following are differences between this implementation and Quinlan’s C4.5 implementation:

Implementation, from a scaling and parallelism standpoint, is based on the following papers:

At a minimum, we require 13 bytes of RAM per row of data in order to support the row mapping data structure. This is distributed throughout the cluster, so if we have 10 nodes in the cluster and 100 million rows of data, we require 13*100 million/10 = 130 MB of RAM per node.

If the data set contains null values, this minimum memory requirement may be larger, as we require an extra n*12+12 bytes of bookkeeping for each row that must be split between children nodes where n is the number of children of the split.

If the inMemoryDataset option is used, the memory requirements are much larger as the attribute tables must be kept in memory. Attribute tables require roughly 32 bytes per row, per attribute. In addition, whenever attributes are split we require working space for the split, so the number of attributes+1 must be calculated. Finally, unknown(null) values may impact the memory sizes since splitting on an unknown value requires adding the row in question to both of the children nodes. Note that attribute tables are distributed throughout the cluster, so memory requirements for attributes do scale out in the same way as for the row mapping structure mentioned above.

The following example demonstrates using the DecisionTreeLearner operator to train a predictive model using the "class" column as the target variable. The decision tree learner produces a PMML model that can be persisted. The generated PMML model can be used with the DecisionTreePredictor to predict target values.

The DecisionTreeLearner operator provides the following properties.

The DecisionTreeLearner operator provides a single input port.

The DecisionTreeLearner operator provides a single output port.

The DecisionTreePredictor operator applies a previously built decision tree PMML model to the input data. This operator supports most of the functionality listed at http://www.dmg.org/v4-0-1/TreeModel.html. Specifically it supports all required elements and attributes as well as the following optional elements and attributes:

It ignores the following elements and attributes:

The DecisionTreePredictor operator provides the following properties.

The DecisionTreePredictor operator provides the following input ports.

The DecisionTreePredictor operator provides a single output port.

The DecisionTreePruner operator performs pruning of the provided decision tree PMML model. This is useful when the learning data creates a model that is overly complex and does not generalize the data well. The operator will examine the tree and reduce its size by removing sections of the tree that provide little power to classify the data. Therefore the overall goal of the DecisionTreePruner is to reduce the complexity of the model while improving its predictive accuracy.

The DecisionTreePruner operator provides the following properties.

The DecisionTreePruner operator provides a single input port.

The DecisionTreePruner operator provides a single output port.

The DiscoverDomain operator can be used as a utility operator for discovering the domain of String fields. Note that this is not intended to be used as a "top-level" operator; rather this is a utility for iterative operators that want to compute their domains.

The following example demonstrates configuring and using the DiscoverDomain operator within an iterative operator.

The DiscoverDomain operator provides the following properties.

The DiscoverDomain operator provides a single input port.

The DiscoverDomain operator does not provide an output port. Instead when the operator has , you can get the result of the domain discovery.

The RunRScript operator is used to execute a snippet of R code within the context of a DataFlow operator. The R script is executed for all data contained within the given input flow. The output of the R script is pushed as the output of the operator.

The R environment must be installed on the execution platform. If this operator is to be used in a distributed environment, the R environment must be installed in all worker nodes of the cluster. The installation path must be the same on all nodes. Visit the CRAN (Comprehensive R Archive Network) website for download links and installation instructions.

By default, the RunRScript operator executes in parallel. This implies that the given R script has no data dependencies and can run independently on segmented data and produce the correct results. If this is not the case, disable parallel execution for the operator.

Note that the operator gathers all of its input data to hand off to R. The R engine then loads the data into memory in preparation for executing the script. The input data must therefore fit into memory. In a parallel or distributed environment, the data is segmented, and each segment of data is processed in a separate R engine. In this case, each data segment must fit into the memory of its executing engine.

The R script is handed an R data frame in a variable named "R". The script can use the data frame as desired. The resultant data is gathered from a data frame of the same name. Two variables are set within the R environment to support parallel execution:

These variables are numeric R variables and can be accessed directly within the user-provided R script.

The sequence of operations are these:

The following code example demonstrates using the RunRScript operator. First, a set of R statements is defined. Next, the RunRScript operator is created and the required properties are set. Code to connect the input and output ports of the operator is not shown.

Note how the R script uses the data frame in the variable named "R" and sets the data to be output in a data frame of the same name.

The RunRScript operator supports the following properties.

The RunRScript operator supports the following input ports:

The RunRScript operator supports the following output ports:

The KNNClassifier operator applies the k-nearest neighbor algorithm to classify input data against an already classified set of example data. A naive implementation is used, with each input record being compared against all example records to find the set of example records closest to it, as measured by a user-specified measure. The input record is then classified according to a user-specified method of combining the classes of the neighbors.

The field containing the classification value (also referred to as the target feature) must be specified. It is not necessary to specify the fields used to calculate nearness (also referred to as the selected features). If omitted, they will be derived from the example and query schema using all eligible fields. The example and query records need not have the same schema. All that is required is that:

The output consists of the query data with the resulting classification appended to it. This value is in the field named "PREDICTED_VAL".

The implementation is designed to minimize memory usage. It is possible to specify an approximate limit on the amount of memory used by the operator; it is not necessary to have sufficient memory to hold both the example and query data in memory, although performance is best in this case.

This example uses the Iris data to classify the type of iris based on its physical characteristics.

The KNNClassifier operator provides the following properties.

The KNNClassifier operator provides the following input ports.

The KNNClassifier operator provides a single output port.

The Naive Bayes algorithm uses Bayes’ theorem to find the probability of an event occurring given the probability of another event that has already occurred.

The Naive Bayes learner produces a model that can be used with the Naive Bayes predictor as a simple probabilistic classifier based on applying Bayes’ theorem with strong, or naive, independence assumptions.

One of the advantages of a naive Bayes classifier is that it only requires a relatively small amount of training data to estimate the parameters necessary for classification.

DataFlow provides operators to produce and use naive Bayes classifiers. The learner is used to determine the classification rules for a particular data set while the predictor can apply these rules to a data set.

The NaiveBayesLearner operator is responsible for building a Naive Bayes PMML model from input data. The base algorithm used is specified at http://www.dmg.org/v4-0-1/NaiveBayes.html, with the following differences:

This example uses Naive Bayes to create a predictive classification model based on the Iris data set. It uses the field "class" within the iris data as the target column. This example produces a PMML model that is persisted. This PMML model can then be used with the NaiveBayesPredictor operator to predict target values.

The NaiveBayesLearner operator provides the following properties.

The NaiveBayesLearner operator provides a single input port.

The NaiveBayesLearner operator provides a single output port.

The NaiveBayesPredictor operator applies a previously built Naive Bayes model to the input data. The base algorithm used is specified at http://www.dmg.org/v4-0-1/NaiveBayes.html, with the following differences:

The NaiveBayesPredictor operator provides the following properties.

The NaiveBayesPredictor operator provides the following input ports.

The NaiveBayesPredictor operator provides a single output port.

The Predictive Modeling Markup Language (PMML) is an XML-based markup language used extensively by DataFlow. PMML provides a vendor-independent method of defining mining models that can be used by many of the DataFlow analytics operators. Any operator that consumes or produces a PMML model will do so through the use of a PMMLPort.

Most operators have a method called getModel() that exposes access to the PMMLPort. There are also various utility methods and classes that can be used to manipulate PMML documents in the com.pervasive.dataflow.analytics.pmml package.

The ReadPMML operator reads a PMML object model from a PMML file.

The ReadPMML operator provides one property.

The ReadPMML operator provides a single output port.

The WritePMML operator is used to write a PMML object model to a PMML file.

The WritePMML operator provides the following properties.

The WritePMML operator provides a single input port.

Regression analysis includes many techniques for modelling and analyzing several variables, when the focus is on the relationship between a dependent variable and one or more independent variables.

Regression analysis is applied when you want to understand how the typical value of the dependent variable changes as the independent variables are varied. It is therefore useful in finding the relationship between variables. This information can then be used to predict the expected values of a variable given the current values of the other variables.

The LinearRegressionLearner operator performs a multivariate linear regression on the given training data. The output is a PMML model describing the resultant regression model. The model consists of the y-intercept and coefficients for each of the given independent variables.

A dependent variable must be specified. This is a field in the input that is the target of the linear regression model. One or more independent variables are also required from the input data.

This operator supports numeric as well as categorical data as input. The linear regression is performed using an Ordinary Least Squares (OLS) fit. Dummy Coding is used to handle categorical variables.

This approach requires for each of the categorical variables one value from its domain to be chosen that serves as reference for all other values in that domain during the computation of the model. Specifying reference values using operator's API is optional. If for a certain categorical variable no reference value is specified by the user, it will be randomly chosen.

The output is an estimate of coefficients for the model:

Y = a + (b1*x1 + ... + bn*xn) + (0*w1 ref + c1,1*w1,1 + ... + c1,k 1 *w1,k1 + ... + 0*w m ref + c m,1 *w m,1 + ... + c m,km *w m,km )

where

The following assumptions are made about the nature of input data:

This example uses linear regression to create a predictive model based on a simple data set. It uses the field "y" as the dependent variable and the fields "x1" and "x2" as the independent variables. This example produces a PMML model that is persisted. This model can then be used with the RegressionPredictor operator to predict data values.

The LinearRegressionLearner operator provides the following properties.

The LinearRegressionLearner operator provides a single input port.

The LinearRegressionLearner operator provides a single output port.

The RegressionPredictor operator applies a previously built regression model to the input data. The model defines the independent variables used to create the model. The input data must contain the same dependent and independent variable fields as the training data that was used to build the model. The predicted value is added to the output dataflow as an additional field. All input fields are also transferred to the output.

The RegressionPredictor operator provides the following properties.

The RegressionPredictor operator provides the following input ports.

The RegressionPredictor operator provides a single output port.

The LogisticRegressionLearner operator performs a stochastic gradient descent, a probabilistic algorithm for very large data sets, on the given training data. The output is a PMML model describing the resultant classification model.

A single dependent variable must be specified. This is a field in the input data that is the target of the logistic regression model. One or more independent variables are also required from the input data.

This operator supports numerical and categorical data as input.

Take the following considerations into account when composing the operator:

This example uses logistic regression to create a predictive model based on the Iris data set. It uses the "class" field as the dependent variable and the remaining fields as the independent variables. This operator produces a PMML model that may be persisted. The resultant model can be used with the RegressionClassifier operator to predict data values.

The LogisticRegressionLearner operator provides the following properties.

The LogisticRegressionLearner provides a single input port.

The LogisticRegressionLearner operator provides a single output port.

The LogisticRegressionPredictor operator implements a maximum-likelihood classifier using regression models to predict the relative likelihoods of the various categories. The operator is used to apply a previously built classification model to the input data.

The model defines the independent variables used to create the model. The input data must contain the same dependent and independent variable fields as the training data that was used to build the model.

The predicted categories are added to the output dataflow as an additional field. All input fields are also transferred to the output. The output also includes a confidence score for each category of the dependent variable.

The LogisticRegressionPredictor operator provides the following properties.

The LogisticRegressionPredictor operator provides the following input ports.

The LogisticRegressionPredictor operator provides a single output port.