Writes all the distinct (unique) values found for the filed into an output file.

String, Date, Timestamp, Boolean, Numeric, Double, Float, Long, Integer

The results file name must not contain the following characters:

Writes all duplicate values found in a data set into a file.

String, Date, Timestamp, Boolean, Numeric, Double, Float, Long, Integer

The results file name must not contain the following characters:

Also, this rule is included as a part of Single Statistics rule. Using Single statics rule includes Min, Max, Mean, Median, Mode, Standard Deviation, Sum, and Variance.

Duplicate clusters provides you the ability to identify potentially duplicate records by evaluating multiple columns at the same time by using the fuzzy matching rules and match score filter.

Real-world data is often dirty and contains variations and errors. Most common examples can be errors in person names, cities, organization names, etc. This could be caused by a spelling mistake. Finding “duplicate” data in this case does not mean finding an exact match but means finding an approximately similar record. Fuzzy Matching (also called Approximate String Matching) is a technique that can be used to filter these similar looking fuzzy duplicates by finding an approximate match.

There are many situations where Fuzzy Matching techniques can be used. Let’s look at some real-world examples:

Let us also understand how the cluster technique works. The main idea in clustering is to group similar records together, into clusters or blocks, using the Key Field value from the records. After this all pairs of records are compared only within the individual blocks. Because real-world data are often dirty and contain variations and errors, an important criteria for a good clustering key is its capability to group similar records into the same cluster or group. Thus identifying the key is critical to the success of the cluster method.

It is also possible to use a combination of Fields to define a Key Field. For example, assuming that most people enter their first name correctly, we can use a combination key of First Name and City to create a cluster or group of all customer records that have same First Name and City entry. After this a measure such as the Levenshtein distance can be used to calculate the scores between pairs of string fields such as Address to identify duplicate customer records.

Grouping Tab - Allows you to define which rows to consider for comparison with each other.

Matching Tab - Allows you to configure string comparison rules and set the match score cutoff. Click to add a comparison rule and to delete it. Specify the following rule details:

 

 

On the Statistics tab, the results for the Duplicate Clusters rule will include:

Also, there will be option to browse the data written to the Results File.

Keys can be of any datatype but Rules must be string fields.

Supported Fuzzy Matching algorithms:

Calculates the following statistics for a numeric field.

Min, Max, Mean, Median, Mode, Standard Deviation, Sum, Variance.

Numeric data types include all the above rule. However, the string data types include only maximum, minimum, median, and mode.

String, Date, Timestamp, Boolean, Numeric, Double, Float, Long, Integer

This rule allows you to include several summary rule at once instead of adding one rule at a time.

Constructs a histogram and defines an equal range for each bin.

Top How Many: Calculates equal width ranges across the field values and counts the number of values in each range.

RangeCount: Specify the number of ranges to create. The default is 10.

 

The Equal Range Binning rule allows you to set certain parameters to control the histogram output. See the following for examples of the histogram output for each parameter.

In this parameter, you set no minimum or maximum values. Therefore, the graphic output may include one bin that contains nearly all the values while the other bins include only a few values.

All numeric types supported as input:

This rule constructs a histogram and defines an equal range for each bin. The rule algorithm works as follows:

The generated histogram has a Range Count and one bar for nulls.

Exceptions: If min and max values are identical, then bar is generated. If data type is integer or long and the minimum and maximum are close together, then it generates the number of bars that fits in the graph.

Example: If minimum is 1 and maximum is 4, then rangeCount > = 4 results in one bar for each possible nonnull whole value.

The behaviour for interpolated breaks between ranges depends upon the input data type:

Calculates the maximum value of a field and supports String types as a separate rule. However, Numeric, Double, Float, Long, and Integer data types are included as part of the Statistics rule.

None

String, Date, Timestamp, Boolean, Numeric, Double, Float, Long, Integer

None

Calculates the median (middle of a sorted list) value of the field.

None

String, Date, Timestamp, Boolean, Numeric, Double, Float, Long, Integer

None

Calculates the mode (most frequently occurring value) of the field.

None

String, Date, Timestamp, Boolean, Numeric, Double, Float, Long, Integer

None

Calculates the most frequent values of a field.

Type a value in the Top How Many text box for the number of frequent values.

The default is 25, which returns the top 25 most frequent values.

String, Date, Timestamp, Boolean, Numeric, Double, Float, Long, Integer

This rule replaces some of the functionality of the Distinct Values rule.

Estimates standard deviation, a measure of how widely values are dispersed from the average value (the mean).

None

String, Date, Timestamp, Boolean, Numeric, Double, Float, Long, Integer

Standard deviation equals the square root of the variance.

Calculate the sum of a field.

None

All numeric types supported as input:

None

Indicates how possible values are spread around the expected value.

None

All numeric types supported as input:

The square root of the variance equals the standard deviation.