Generating Data
Generating Additional Data Fields
DataFlow provides a set of operators that can be used to generate data tokens for use within a DataFlow application.
Covered Operations
Using the GenerateConstant Operator to Generate Constants
The
GenerateConstant operator can be used to generate copies of a constant value.
Code Example
This code fragment shows how to initialize a
GenerateConstant operator for your graph. The output port will provide 100 records with a single field called "afield" that contains an integer value of 1.
Using the GenerateConstant operator in Java
ScalarTokenType type = TokenTypeConstant.INT;
ScalarToken value = TokenUtils.parse(type, "1");
RecordToken rconst = new RecordToken(record(field(type, "afield")), value);
GenerateConstant constants = graph.add(new GenerateConstant(rconst, 100));
Using the GenerateConstant operator in RushScript
var token = RecordToken(IntToken(1));
var data = dr.generateConstant({rowCount:100, constant:token});
Properties
The
GenerateConstant operator has the following properties.
Ports
The
GenerateConstant operator provides a single output port.
Using the GenerateRandom Operator to Generate Random Data
The
GenerateRandom operator can be used to generate random data. All field types except generic and object are supported.
The generated data for each data type field does not generally cover the full range that is supported by the DataFlow system, but it does cover a range that any operator claiming to support that type should be able to handle.
• boolean: Either true or false.
• binary: Between 1 and 2048 random bytes, with a uniform distribution of the number of bytes.
• char: ASCII characters 32-126 ("Valid Unicode" is not well defined).
• date: The range of days representable by a
Date, +- 2^64 milliseconds from 1970-01-01, corresponding to +- about 292 million years.
• double: The range of Double excluding NaN and +-Inf. (NaN values can be generated by setting nullFraction > 0.0).
• float: The range of Float, excluding NaN and +-Inf. (NaN values can be generated by setting nullFraction > 0.0).
• int: The full range of Integer.
• long: The full range of a Java Long.
• numeric: The integer part is formed from 1 to 100 binary digits, corresponding to up to 31 decimal digits (2^100 - 1 = 1,267,650,600,228,229,401,496,703,205,375). This value is then divided by 10^scale, where the scale is 0 to 29, and made negative with a probability of 50%.
• string: Zero or more random ASCII characters (see also char data type). The string length is unlimited, but the probability decays exponentially, as 0.9^min_length: 10% are empty strings, 9% are 1 character long, 8.1% are two characters long, 7.29% are 3 characters long, etc.
• timestamp: The range of seconds is the range representable by a
Timestamp, +- 2^64 milliseconds from 1970-01-01, corresponding to +- about 292 million years. The nanoseconds range is 0 to 999999999. The time zone offset is a whole number of minutes in the range -12:00 to +12:59 (the extra hour is for daylight savings).
Code Example
This code fragment shows how to initialize a
GenerateRandom operator for your graph. The output port will provide 100 records with two fields called "name" and "age", which will be randomly filled with String and int values respectively.
Example Usage of the GenerateRandom operator in Java
RecordTokenType type = record(STRING("name"), INT("age"));
GenerateRandom randoms = graph.add(new GenerateRandom(type, 100);
The following example demonstrates using the operator in RushScript.
Using the GenerateRandom operator in RushScript
var type = dr.schema().STRING('name').INT('age');
var data = dr.generateRandom({rowCount:100, outputType:type});
Properties
The
GenerateRandom operator has the following properties.
Ports
The
GenerateRandom operator provides a single output port.
Using the GenerateRepeatingCycle Operator to Generate Repeating Cycles
The
GenerateRepeatingCycle operator is used to generate a cycle of repeating values. First all the rows of the input are copied to the output, then all the rows of the input are copied to the output again, and so on. Output stops when the desired row count is reached.
Code Example
This code fragment shows how to initialize a
GenerateRepeatingCycle operator for your graph. It will generate 10 random integers, which it will then repeatedly push to the output until 100 records have been generated.
Using the GenerateRepeatingCycle operator in Java
GenerateRandom rand = graph.add(new GenerateRandom(record(INT("number")), 10));
GenerateRepeatingCycle repeat = graph.add(new GenerateRepeatingCycle(100));
graph.connect(rand.getOutput(), repeat.getInput();
Using the GenerateRepeatingCycle operator in RushScript
var repeatData = dr.generateRepeatingCycle(data, {rowCount:100});
Properties
The
GenerateRepeatingCycle has one property.
Ports
The
GenerateRepeatingCycle operator provides a single input port.
The
GenerateRepeatingCycle operator provides a single output port.
Using the GenerateArithmeticSequence Operator to Generate Sequences
The
GenerateArithmeticSequence operator is used to generate a sequence of numerical values, with a constant difference between consecutive values.
Code Example
This code fragment shows how to initialize a
GenerateArithmeticSequence operator for your graph. The output port will provide 100 records with a single field of type long with the value starting at 1 and incrementing by 1 on each successive record.
Using the GenerateArithmeticSequence operator in Java
GenerateArithmeticSequence seq = graph.add(new GenerateArithmeticSequence(100));
seq.setStartValue(1);
seq.setStepValue(1);
Using the GenerateArithmeticSequence operator in RushScript
var data = dr.generateArithmeticSequence({rowCount:100, startValue:1, stepSize:1});
Properties
The
GenerateArithmeticSequence operator has the following properties.
Ports
The
GenerateArithmeticSequence operator provides a single output port.