The following are the DataFlow input and output nodes available in KNIME.

The following Read and Write nodes are available:

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the ReadARFF Operator to Read Sparse Data.

The ARFF Reader node supports the Attribute-Relation File Format (ARFF). See the ARFF Weka page for more information. ARFF supports both sparse and dense data formats. The format is detected automatically. ARFF contains a metadata section that defines the schema of the ARFF file. Hence there are very few options needed to configure the reader.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the ReadAvro Operator to Read Apache Avro Data.

The Avro reader supports the Apache Avro format. Avro files contain metadata defining the schema of the file. Therefore, very few options are needed to configure the reader.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the ReadFromJDBC Operator to Read Databases.

Reads from a relational database table, storing the results in a staging dataset.

Click Fetch Metadata to get the information about all the visible tables. This will query the database to get all the schema information that is visible. It is useful while constructing the query.

Configuration consists of the following steps:

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the ReadDelimitedText Operator to Read Delimited Text.

This node reads a text file of delimited records. Records are separated by new lines, represented either Windows-style (CRLF) or UNIX-style (LF). By default, the style is automatically detected.

Delimited text supports the following separators used to identify field boundaries:

The field separator cannot be empty. The field delimiter can be empty. It is not expected that all fields start and end with a delimiter, though if one starts with a delimiter it must end with one.

Fields containing significant characters, such as white space and the record and field separators, must be delimited to avoid parsing errors. Should a delimited field need to contain the field delimiter, it is escaped from its normal interpretation by duplicating it. For instance, the value "ab""c" represents a delimited field value of ab"c.

Delimited files may or may not have a header row. The header row is delimited as usual but contains the names of the fields in the data portion of the record. The reader must be told whether a header row exists or not. If it does, the reader will skip the header row; otherwise the first row is treated as a record and will appear in the output. Additionally, when discovering the schema of a delimited file with a header row, the field names will be taken from the header.

Data from the input file may be browsed within the configuration dialog. Once the field separator and other options are set, the fields within the input data should be parsed correctly and displayed in the data table within the configuration dialog.

An attempt will be made to identify the names and data types of each field. However, the names and data types of each field may be modified by right clicking on the column header of the field and following the menu items on the context menu. Select the option to configure the schema to modify the names and data types of fields.

The reader can tolerate some degree of error in the input file. How it reacts to errors is configurable. Typically, one of five actions can be taken when an error is discovered.

Additionally, a limit can be placed on warnings, causing the read to fail if a certain threshold is reached.

When setting a field’s data type to enumerated, you have the option to discover the values of the enumeration from the input data. Click Start Discovery to begin the discovery process. The input data will be read, and each unique value of the selected field will become a category within the enumeration. You can either wait for the discovery to complete reading the whole files or click Stop Discovery to complete the operation. You should only stop the discovery process if you know that all of the categorical values have been discovered.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the ReadHBase Operator to Read from HBase.

This node allows reading from Apache HBase.

The first step in configuring the operator is to specify an HBase Cluster and mapping method. You may select any cluster that you have configured in the HBase Clusters preferences page.

After you have specified a cluster and mapping method, click Connect to connect to the HBase cluster. Proceed to either the field mapping or the HCatalog table selection.

After a connection to the cluster is established, a list of available tables will be loaded. Select the table you want to read and click Get Schema. This will load available schema information, so you can map HBase qualifiers to output fields.

If the table was previously written by the DataFlow HBase Writer, the reader will be able to retrieve schema information about that table, including qualifiers and their types. This information will be automatically loaded into the schema mapping list, where you can assign an output field to each qualifier. If there are any additional qualifiers you have added to the table outside of DataFlow, you may manually add them using the Add Qualifier button.

If the table has not been written by the DataFlow HBase Writer, the only information available will be the table’s families. Select a family and click Add Qualifier to add a new qualifier to read. After a qualifier is added, you can specify an output field name. These fields will have a binary type, which may be converted downstream using another operator.

After a connection to the cluster is established, a list of available HCatalog databases will be loaded. Select the database, and a list of tables in that database is loaded. Select the HCatalog table from which to read.

You may limit the records returned to a certain timestamp range by selecting the Limit by Time Range option. When this option is selected, you may enter a start time and an end time in the corresponding fields.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the ReadJSON Operator to Read JSON Text.

This node reads a text file of JSON lines records. For more information about the JSON lines format, see http://jsonlines.org/. Records are stored in JSON text format with one record per line, separated by a newline separator character.

Data from the input file may be browsed within the configuration dialog. Once the various JSON parsing options are set, the fields within the input data should be parsed correctly and displayed in the data table within the configuration dialog. An attempt will be made to identify the names and data types of each field. However, the data types of each field may be modified by right-clicking on the column header of the field and following the menu items on the context menu. Select the option to configure the schema to modify data types of fields.

The reader can tolerate some degree of error in the input file. How it reacts to errors is configurable. Typically, one of five actions can be taken when an error is discovered:

Additionally, a limit can be placed on warnings, causing the read to fail if a certain threshold is reached.

When setting a fields data type to enumerated you will have the option to discover the values of the enumeration from the input data. Click on Start Discovery to begin the discovery process. The input data will be read and each unique value of the selected field will become a category within the enumeration. You can either wait for the discovery to complete reading the whole file(s) or click on Stop Discovery to complete the operation. You should only stop the discovery process if you know that all of the categorical values have been discovered.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the ReadLog Operator to Read Log Data.

The Log Reader specifies a node for reading log data files. The various log types can accept a format specifier string in the same format as would be accepted by the particular log producer unless otherwise specified. The output is determined by the format and the logs being read.

The minimum configuration required is setting the log type property and the source, although some formats may additionally require the pattern property be set to produce meaningful output. The node will by default attempt to determine the newline character used by the log files, however this can be specified if known in advance.

Log entries that cannot be parsed will not be present in the output while fields that cannot be parsed will be null valued.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the ReadMDF Operator to Read MDF Files.

The MDF File Reader node enables reading records from MDF format files. It can read data previously written using MDF format. Currently only MDF version 4.0 or greater is supported. For more information, see https://www.asam.net/standards/detail/mdf/wiki. This node will read the selected records with the specified ID from a single data group within the file.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the ReadORC Operator to Read Apache ORC Files.

A node for reading records from ORC format files.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the ReadParquet Operator to Read Apache Parquet Files.

This node enables the reading of records from Parquet format files.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the ReadPMML Operator to Read PMML.

Reads a Predictive Model Markup Language (PMML) model from a file. If you are using KNIME version 2.10 or later, then PMML version 4.2.x is supported. If you are using KNIME 2.9.x, then PMML version 4.0 or 4.1 is supported. For PMML definition, visit the website Data Mining Group.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the ReadStagingDataset Operator to Read Staging Datasets.

Imports a DataFlow staging dataset. This allows you to exchange datasets between KNIME workflows. In addition, the input format is the standard produced by the DataFlow library operator WriteStagingDataset, so it can easily integrate with other applications built on the DataFlow library.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the LoadActianVector Operator to Load Actian Vector.

The Load Actian Vector node enables loading bulk data to an Actian Vector database instance. The loader supports the following methods for loading the data:

For more information about methods for loading the data, see Using the LoadActianVector Operator to Load Actian Vector.

For information about using Actian Vector Writer node to load data to Avalanche, see Loading Delimited Text Data to Avalanche Using Actian Vector Writer Node.

This node can be used to load the Actian Vector database version 4.2 and later.Connections to Actian Vector must be set up using the File > Preferences > Actian Vector Connections page in DataFlow. After a connection is configured, you may load the data by selecting it from the connection list in the configuration window. To configure a Vector connection, see Actian Vector Connections.

By default, the loader maps the source fields to the target database fields by position. If few input fields are available, then the unmapped database fields are loaded with null values. If any of the unmapped fields cannot have a null value, then a loader error occurs. To manually map the source fields to the target table fields, click Map Fields to open a mapper window. You can click and drag source fields to a target database column or click any of the map options to map by name or position. You also can select the source field by using the list available on each target field.

When the loader is executed, the input data is transformed into loader format in a temporary data space. After the transformation is complete, the staged loader files are loaded into the database. When the load finishes, the temporary files are removed and the output of the load command is logged at the INFO level. Log entries can be viewed in the console or by browsing the log file.

Actian Avalanche is a cloud-based version of Actian Vector. You can use Actian Vector Writer node to load data into Avalanche.

To load data into a table in the managed Avalanche data warehouse instance, perform the following steps:

You can configure one of the following as the staging location:

To configure the remote staging location:

To configure the Actian Vector connection to access Avalanche:

To create a DataFlow workflow file, see Creating an Actian DataFlow Workflow.

Use the Delimited Text Reader operator for reading the flat file and the Actian Vector Writer operator for writing the data to a table in the Avalanche instance.

To run the workflow, click the Execute all executable nodes icon.

You can see that the first node is reading the data and the second node is loading the data to the Avalanche cluster.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the WriteARFF Operator to Write Sparse Data.

This writer supports the Attribute-Relation File Format or ARFF. ARFF files are convenient in that they embed metadata within the data stream. The metadata defines the schema of the data including the field names, types, and positions.

When writing an ARFF file you can specify the data format: either sparse or dense. Sparse mode is very compact when used with sparse data. However, when used with dense data the overhead can cause waste of file space. Dense mode is best used when the data is not sparse. Dense format closely resembles comma-separated values (CSV) format.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the WriteAvro Operator to Write Apache Avro Data.

This writer supports the Apache Avro format. The Avro format is convenient because it embeds metadata within the data stream. The metadata defines the schema of the data, including the field names, types, and positions. Additionally, it is designed for parallelized writing and performs a level of data compression on its encoded data.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the DeleteFromJDBC Operator to Write Database Deletes.

This node deletes data rows in the database based on the selected columns from the input. This node allows setting the fields used as keys to find matching rows for the delete. When configuring the schema used to map the input fields to the database, any field that is mapped is used as a key field. It is important to note that it is not required to map every field if only a subset of the input fields are required. These key fields are used to find matching rows to delete in the database. Additionally, the node produces an output table containing the data used for the delete and an additional column with the number of rows in the database affected by each row used in a delete statement.

Configuration consists of the following steps:

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the UpdateInJDBC Operator to Write Database Updates.

This node will update data rows in the database with the data values in the selected columns from the input. This node allows setting the fields that will be updated in addition to the fields that will be used as keys to find matching rows for the update.

When configuring the schema that will be used to map the input fields to the database, any field that is mapped may be used as a key field or an update field. It is important to note that it is not required to map every field if only a subset of the input fields is required; however, a minimum of at least one key and update field must be specified when configuring the mapping. These key fields will be used to find matching rows that will then have the specified update fields updated.

Additionally the node will produce an output table containing the data used for the update and an additional column with the number of rows affected by each row used in an update statement.

Configuration consists of the following steps:

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the WriteToJDBC Operator to Write to Databases.

Writes a dataset to a relational database table.

Configuration consists of the following steps:

By default, the loader maps source fields to the target database fields by position. If too few input fields are present, the unmapped database fields will be loaded with null values. If any of the unmapped fields are nonnullable, a loader error will occur.

To manually map the source fields to the target table fields, click the Map Fields button. A mapper dialog will appear. The source fields can be dragged and dropped onto a target database column. Alternatively one of the map buttons can be used to map by name or position. The source field can also be selected using the dropdown list available on each target field. The SQL type can also be customized if a more specific type than the default is desired.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the WriteDelimitedText Operator to Write Delimited Text.

Specifies a node for writing a text file of delimited records. Records are separated by a known character sequence. Typically, new lines are used, either Windows-style (CRLF) or UNIX-style (LF). Unless otherwise specified, the system-appropriate style is used.

Delimited text supports the following separators used to identify field boundaries:

The field separator cannot be empty. The field delimiter can be empty. It is not expected that all fields start and end with a delimiter, although if you start with a delimiter, you must end with one. Fields containing significant characters, such as whitespace and the record and field separators, must be delimited to avoid parsing errors. Should a delimited field need to contain the field delimiter, it must be escaped from its normal interpretation by duplicating it. For instance, the value "ab""c" represents a delimited field value of ab"c.

Delimited files may or may not have a header row. The header row is delimited as usual but contains the names of the fields in the data portion of the record. These names come from the field names of the input data.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the WriteHBase Operator to Write to HBase.

This node allows writing to Apache HBase.

The first step in configuring the node is to specify an HBase Cluster and at least one mapping method. You may select any cluster that you have configured in the HBase Clusters preferences page.

A mapping may be provided by hand by specifying an HCatalog database and table, or both.

After you have specified a cluster and mapping method, click Connect to connect to the HBase cluster. Proceed to the field mapping or the HCatalog table selection.

After a connection to the cluster is established, a list of available tables will be loaded. Select the table you would like to write, or enter the name of a new table, and click Get Schema. This will load available schema information so you can map HBase qualifiers to input fields.

If the table was previously written by the DataFlow HBase Writer, the reader will be able to retrieve schema information about that table, including qualifiers and their types. This information will be automatically loaded into the schema mapping list, where you can assign an input field to each qualifier. You may also manually add new qualifiers using the Add Qualifier button.

If the table has not been written by the DataFlow HBase Writer, the only information available will be the table’s families. Select a family and click Add Qualifier to add a new qualifier to write. After a qualifier is added, you can specify an input field name.

If the table does not exist, you will have to manually add the families and qualifiers you want to write. Click Add Family to add a new family. Within each family, click Add Qualifier to add new qualifiers to that family. After you have added families and qualifiers, you can specify input fields for each qualifier.

In the DataFlow HBase Writer, new qualifiers can be created as either a scalar type or a record type. Scalar types store a single value in a cell, which can be read by any HBase client. Record types allow you to store multiple fields in a single HBase cell. Using record types increases the space and time efficiency, as only a single lookup is needed to read or write an entire set of fields; however, they are stored in a format unique to DataFlow, and the schema for these records remains fixed once they are created.

To create a record-type qualifier, click Add Qualifier, and when prompted for the type, select Record. After you have added the record-type qualifier to the schema, select it and click Add Field to add subfields to the record. You can then specify input fields for each subfield.

In addition to families and qualifiers, you can also map the row ID and row timestamp to an input field. In most cases, both of these are optional: if the row ID is not specified, a unique ID will be generated for each row; if the row timestamp is not specified, the actual insert time will be used. However, when using HCatalog, you must specify a row ID.

After a connection to the cluster is established, a list of available HCatalog databases are loaded. Select the database, and a list of tables in that database is loaded. Select the HCatalog table you want to write to , or enter the name of a new table to create.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the LoadMatrix Operator to Load Actian Matrix.

The Load Actian Matrix node bulk loads data into a Matrix database instance. The loader directly loads data into Matrix using an ODI connector. Loading supports parallel and distributed execution (on a cluster).

Connections to Actian Matrix must be preconfigured using the Actian Matrix Connections preference page in the Actian preferences section. After a connection is configured, it can be used to load data by selecting it from the connection list in the configuration dialog. To configure a Matrix connection, see Actian Matrix Connections.

By default, the loader maps source fields to the target database fields by position. If too few input fields are present, the unmapped database fields are loaded with null values. If any of the unmapped fields are nonnullable, a loader error occurs.

To manually map the source fields to the target table fields, click the Map Fields button to open a mapper dialog. You can click and drag source fields to a target database column, or click one of the map buttons to map by name or position. The source field can also be selected using the list available on each target field.

When the loader is executed, the input data is transformed into loader format in a temporary data space. After the transformation is complete, the staged loader files are loaded into the database. When the load finishes, the temporary files are removed.

When the load finishes, the output of the load command is logged at the INFO level. Log entries can be viewed in the console or by browsing the log file.

Network communication between DataFlow and the Matrix database is accomplished using TCP/IP sockets. The properties startPortRange and endPortRange can be used to limit the range of TCP/IP ports utilized for this communication. The ports within this range need to be available for two-way communication. In a firewall environment, the ports must be opened within the firewall configuration. The default port range is from 40000 to 41000 (exclusive).

When you run multiple workflows concurrently or run workflows with multiple Load Matrix operators, ensure the port range is large enough to allow at least one open port per slice per operator. Also, enure Matrix database workload management is configured to allow enough concurrent queries to leave at least one available for each active Load Matrix operator. This availability is configured using the max_concurrent_queries property in ~paraccel/padb/rel/etc/padb_wlm.xml on the Matrix server.

You may configure a DataFlow service class with specific properties for DataFlow jobs, which will allow the loader to check for available resources before launching a job and fail quickly if there are insufficient resources on the Matrix server; without this service class, a job may hang when there are insufficient resources. For more information about configuring this service class, see Installing and Configuring DataFlow for Matrix On-Demand Integration (ODI).

Data is loaded into Matrix in parallel. A data connection into Matrix is created for each parallel stream within DataFlow. The number of data connections can be limited by setting the max data connections property (see below). Setting this value may cause a redistribution of the data before loading. Consider setting the overall parallelism for the loader job to the desired number of connections to prevent data redistribution.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the MatrixSink Operator to Load Actian Matrix from SQL.

The Matrix Sink node loads large data into a Matrix database instance. This node is similar to Load Actian Matrix node. Refer to this node for more information about the process for loading large data.

The purpose of the Matrix Sink node is to embed in workflows that are executed from Matrix SQL. Workflows having the Matrix Sink node can be directly exported to a Matrix instance using the Exporting an Actian DataFlow Workflow. Execution of workflows from Matrix SQL requires the invoke_dataflow user-defined function (UDF) to be installed on the Matrix instance. For more information, see Installing and Configuring DataFlow for Matrix On-Demand Integration (ODI).

There is no defined configuration for the Matrix Sink node. All the required settings are provided to the node by the invoke_dataflow UDF before executing the workflow.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the WriteORC Operator to Write Apache ORC Files.

This node can write records in Apache Optimized Row Columnar (ORC) format files.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the WritePMML Operator to Write PMML.

Writes a PMML model to a file. If you are using KNIME version 2.10 or later, then PMML version 4.2.x is supported. If you are using KNIME 2.9.x, then PMML version 4.0 or 4.1 is supported. For PMML definition, visit the website Data Mining Group.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the WriteStagingDataset Operator to Write Staging Data Sets.

Exports a DataFlow staging dataset. This allows you to exchange datasets between KNIME workflows. In addition, the output format is the standard consumed by the DataFlow library operator ReadStagingDataset, so it can be easily and efficiently consumed by other applications built on the DataFlow library.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the ForceRecordStaging Operator to Explicitly Stage Data.

Useful when creating a workflow intended for the DataFlow Executor to force staging to occur. This node forces partitioning of DataFlow graphs into smaller graphs at its placement (cut points). This is useful for preventing queue expansion that would otherwise occur.

 KNIME  This topic describes a KNIME node. It is useful for testing the behavior of other nodes during workflow development.

Logs information about the input data from a flow. The record type of the flow is logged. A log frequency is used to determine which data rows are logged. A data row is logged by outputting the value of each input field within the row. A format can be provided that specifies output format for each row log. The final row count is also logged.

 KNIME  This topic describes a KNIME node. For the DataFlow operator it is based on, see Using the ParseTextFields Operator to Parse Text Records.

Specifies a node for parsing records consisting of text fields. These records typically represent data which has been read from a file and broken into individual fields based on some criteria, but with no additional interpretation performed. This node can be used to interpret the text fields as configured data types, making the result suitable for use with other operators.

The provided schema is used to determine the output type. Input fields are parsed according to the format specified for the field of the same name in the output schema. Fields present in the input but not named in the output schema are dropped. Fields named in the output need not exist in the input; their value will be NULL in the output.