Defining Custom Operators

The operators shipped with the DataFlow library are automatically imported into the RushScript environment. If you develop a custom DataFlow operator, your new operator will need to be imported also. You can do this in one of two ways outlined below.

The dr.defineOperator() function can be used to explicitly register a new operator within RushScript. An example of invoking the function follows. Note that the call to the defineOperator() function provides a unique name of the operator and the fully qualified class name of the Java class that implements the operator. The Packages expression in JavaScript must be used to start the fully qualified class name. This is a JavaScript construct that helps bridge the gap between JavaScript and Java.

After an operator is registered using the defineOperator() function, the operator is ready to be used within RushScript. A new function will be added to the dr variable using the given operator name. Using this function will create a new operator instance and add it into the current graph as expected.

Note the use of the newly created dr.customEx() method in the code example below. As long as the custom operator follows the conventions for building a DataFlow operator then the operator will be usable within RushScript. Reference the section Writing an Operator for more information on operator development conventions and best practices.

The class file (compiled Java) for the operator and any of its dependencies must be in the class path of the RushScript environment at execution time. The most common way of doing this is to create a .jar file of the needed class files and add the .jar file to the class path. See Command Line Usage: dr for more information about how to set the class path on the command line when executing a RushScript script.

Within an IDE such as Eclipse, the class path for the project containing your script may need to be modified to add the custom operator classes. Setting up a project class path varies by IDE. Follow the instructions for adding projects or .jar files to a project within your IDE of choice.

// Define a custom operator
dr.defineOperator("customEx", Packages.dr.training.customops.CustomExecutable);

// Define ratings schema
var ratingschema = dr.schema()
    .nullable(true)
    .trimmed(true)
    .INT("userID")
    .INT("movieID")
    .DOUBLE("rating")
    .INT("timestamp");

// Read the ratings
var ratings = dr.readDelimitedText({source:'data/ratings.txt', schema:ratingschema, fieldSeparator:"::", header:true});

// Apply the custom operator to the input data
var results = dr.customEx(ratings, {inputFieldName:"rating", outputFieldName:"sqr"});

// Log the results of the custom operator
dr.logRows(results, {logFrequency:1});

// Execute the graph with the custom operator
dr.execute("customOp");

Customizing Operator Serialization discusses how to use JSON annotations and the operator registry to register new operators. Using the operator registry is not required, but if used it makes importing custom operators into the RushScript environment easier.

An operator that uses the operator registry bypasses the requirement to use the dr.defineOperator() function within RushScript. As long as the jar file containing the operator is available on the RushScript class path at execution time, the operator will be directly accessible with no further work within RushScript.