Remediation rules can help you to identify and correct errors, inconsistencies, and inaccuracies in source data. This can include tasks such as removing duplicate records, standardizing format and data types, and filling in missing values.

During profile execution, the Remediation rules are processed first followed by the Profiling Rules. The processing order ensures that the changes specified in the Remediation rules are applied to your data before the Profiling rules are evaluated.

Rules can be configured in any order. While you can reorder the Remediation rules by moving them up or down, the Profiling rules cannot be reordered.

The Remediation rules are executed in the order you create them or reorder them on the Rules tab and the results may vary as per their order, however, the RemoveDuplicates and RemoveDuplicatesFuzzyMatching rules will always be processed last.

See Add New Rule for information about adding rules.

The following is a list of remediation rules:

 

This rule changes the format of a source string field from one format to another (applies to string fields which can be parsed as string/numbers/boolean types).

This rule has the following properties.

 

This rule changes the case of a string field to specified case (for example upper, lower, and title).

This rule has the following properties.

 

This rule changes data via masking or encryption so that it cannot be linked to an individual. The masked or encrypted data can then be shared without risk of exposing sensitive personal information.

This rule has the following properties.

To decrypt values encrypted with this operation, you will need the following parameters. Please ensure these details are stored securely, as they are crucial for data retrieval.

 

This rule calculates and modifies Date and Timestamp data types. It is useful for tasks such as:

This rule only appears in the Add New Rule menu for Date and Timestamp fields.

This rule has the following properties.

Input: Date, Timestamp

Output: Date, Timestamp

Use Case: Display times and timestamp transactions in the local time zone.

To change a value from Pacific Standard Time (2026-01-07 05:00:00) to Eastern Standard Time, add 3 hours to get the output 2026-01-07 08:00:00.

Use Case: Monitor invoices and track accounts receivable.

To calculate a due date, add 30 days to the order date (2026-01-07). The output is 2026-02-06.

Use Case: Monitor customer service response times, track SLA compliance, escalate tickets approaching breach time.

To set a response deadline, add 4 hours (the required response time) to the ticket's open time (2026-01-07 09:00:00). The output is 2026-01-07 13:00:00 (SLA deadline).

 

This rule removes specified characters or words from a string field.

This rule has the following properties.

 

This rule writes unique records to pass target and duplicate records to fail target, using exact match.

This rule supports all data types.

This rule involves the use of clustering. Clustering refers to grouping similar records together, into clusters, based on the comparison of key fields. When the profile is executed, the values in the specified key fields are compared to each other to identify duplicates. If a duplicate value is found, then a cluster containing the duplicate records is created. The cluster contains duplicate records based on the key fields and the cluster is assigned a cluster ID.

After a cluster is created, users can configure a Cluster Matching Rule and a Record Matching Rule to specify criteria for finding unique records within a cluster. For information about creating clusters, see Cluster Data Tab.

Only one Remove Duplicates rule is permitted within a profile. However, multiple fields can be designated as key fields. Once the Remove Duplicates rule has been added to a profile, it will be categorized under MultipleFields on the Rules tab, instead of by the name of the key field.

See Also: RemoveDuplicatesFuzzyMatching.

This rule has the following properties.

Additional properties for this rule are available in the following tabs:

Select a field from the list of Input Fields as a key field. The values in the key field will be evaluated to identify duplicates or exact matches. If a duplicate value is found, a cluster is created. The cluster will contain the duplicate records based on the key fields and will be assigned a Cluster ID (visible when browsing the cluster file).

The Sorting tab allows you to arrange the records within each cluster in a specific order. You can sort on multiple fields. Click to add a row and to delete it. After adding a row, specify the following values:

The Unique Record Rules tab provides a way to specify one or more clusters via cluster matching rules, then identify one or more records as unique via the record matching rules.

You can perform the following actions:

The Add Unique Record Rule dialog allows you to add rules of the following type:

When the profile is executed, the focus is shifted to the Statistics tab from which you can view the graphical representation of the pass/fail data. On the Results Summary pane, click on the required field name:

The Cluster Data tab is only visible after generating clusters.

On the Rules tab, select the Remove Duplicates rule and click from the Rule Definition pane to generate clusters and browse the cluster file in the Cluster Data tab.

The following additional fields are displayed:

All data types.

This rule writes unique records to pass target and duplicate records to fail target based on fuzzy matching rules and match score filter.

This rule supports all data types.

Finding “duplicate” data in this case does not mean finding an exact match but means finding an approximately similar record. Fuzzy Matching (also called Approximate String Matching) is a technique that can be used to filter these similar looking fuzzy duplicates by finding an approximate match.

There are many situations where fuzzy matching techniques can be used. For instance, using fuzzy matching, you can find multiple variations of a word, term, or phrase. For example, if your word is “DataConnect,” fuzzy matching returns DataConnect, Data Connect, D Connect, Data C, DattaConecct, and so on.

This rule uses clusters. Let us now understand how the cluster technique works. The main idea in clustering is to group similar records together, into clusters, based on comparison using key fields. When the profile is executed, the values in the specified key field are compared to each other to identify duplicates. If a duplicate value is found, then a cluster is created. The cluster contains the duplicate records, based on the key field and is assigned a cluster ID.

After a cluster is created, you can configure a Match Score Filter, a Cluster Matching Rule, and a Record Matching Rule to specify criteria for finding approximately similar records within a cluster.

This rule has the following properties.

Additional properties for this rule are available in the following tabs:

Select a field from the list of Input Fields as a key field. The values in the key field will be evaluated to identify duplicates or exact matches. If a duplicate value is found, a cluster is created. The cluster will contain the duplicate records based on the key fields and will be assigned a Cluster ID (visible when browsing the cluster file).

Allows you to configure string comparison rules and set the match score cutoff. Click to add a comparison rule and to delete it. Specify the following rule details:

The Sorting tab allows you to arrange the records within each cluster in a specific order. You can sort on multiple fields. Click to add a row and to delete it. After adding a row, specify the following values:

The Unique Record Rules tab provides a way to specify one or more clusters via cluster matching rules, then identify one or more records as unique via the record matching rules. You can perform the following actions:

 

The Add Unique Record Rule dialog allows you to add a rules of the following type:

 

When the profile is executed, the focus is shifted to the Statistics tab from which you can view the graphical representation of the pass/fail data. On the Results Summary pane, click on the required field name:

See Also: Viewing Pass, Fail, and Drill Down Output.

On the Rules tab, select the Remove Duplicates rule and click from the Rule Definition pane to generate clusters and browse the cluster file in the Cluster Data tab. The following additional fields are displayed:

All data types.

 

This example demonstrates how to configure and execute the RemoveDuplicatesFuzzyMatching rule to identify and remove potential duplicate records based on exact and fuzzy matching logic.

Assume your dataset contains customer records with the following fields:

You suspect there may be duplicate records where:

Create a rule that uses first_name as the matching key. This means a separate cluster will be generated for each exact match of the first name.

On the Rule Configuration screen, select first_name as the Key Field.

This means, a cluster will be created for each exact first_name. For example, all records with first_name = "Chuck" will be grouped into the same cluster. Clusters are evaluated independently for possible duplicates.

Within each cluster, the last_name and company fields need to be evaluated to determine whether they are sufficiently similar to indicate potential full-record duplicates that should be removed.

On the Matching tab, these fields need to be selected and a matching algorithm assigned to each one. Navigate to the Matching tab. Within each cluster (for example, all “Chuck” records), configure fuzzy matching on the last_name and company fields:

Select an appropriate fuzzy matching algorithm for each field based on the available options (e.g., DAMERAU_LEVENSHTEIN algorithm).

This allows the rule to, identify records where last_name values are similar (e.g., “Smith” vs. “Smyth”) and identify records where company values are close matches (e.g., “ABC Corp” vs. “ABC Corporation”)

Go to the Sorting tab. Choose zip as the sort field within each cluster. Sorting helps organize records logically when reviewing potential duplicates.

Click Run to execute the profile. When processing is complete, you are navigated to the Results page. Click on RemoveDuplicatesFuzzyMatching rule. On the right side of the screen, you’ll see a pass/fail pie chart summarizing the results.

Below the pie chart, click Browse Cluster File to view the generated clusters. You will now see grouped clusters.

For example: Cluster 30 contains all records where first_name = "Chuck". Within that cluster, you will observe, similar last_name values and similar company values. This indicates likely duplicate records.

You may determine that the record with Id: 569000 appears to be the most accurate or complete version and therefore is the best candidate to keep.

To retain specific records:

The process can be repeated. This allows controlled and precise duplicate resolution.

This configuration only handles exact matches on first_name. If your data includes variations such as, “Chuck”, “Chuk”, “Chuc”, you should, first clean up exact first_name duplicates using the rule described above. Then create a new RemoveDuplicatesFuzzyMatching rule where:

This staged approach improves matching accuracy and reduces false positives.

 

This rule replaces field values based on a specified constant, field, function result, regular expression result, or a lookup result. This rule also enables users the option to replace values only if a specified condition is met. The field data being replaced is the field this rule is written for.

This rule has the following properties.

This rule has the following parameters.

Select an option from the Replace Function Choices drop-down list.

This option allows you to perform a lookup (search and retrieve values from another dataset). The lookup data is stored in memory (RAM).

This rule is useful when your data contains a field that is related to another field in another dataset and you want to replace it’s value with a value out from that other dataset.

Users select or add a lookup, specify a Lookup Key Field from the lookup list of fields in the lookup dataset, specify a Matching Key Field from the source data or derived field, and the lookup Replacement Field from the lookup dataset. During execution, when the Lookup Key Field and the Matching Key Field values match, the lookup Replacement Field value replaces the data field value before writing to the target.

This rule option supports all data types. However, the Lookup Key Field and Matching Key Field values must be the same data type.

This option requires a lookup source definition. You can add a lookup source in the Source tab (see Managing Lookups in Profile Editor), or in the Rules tab when configuring this rule or the LookupValue rule.

This topic includes the following:

 

The following use case examples describe a user with a source State column that contains state codes (for example CA and TX). The user wants the values to be state names (for example California and Texas). These examples use the following “States.txt” data file as the lookup file, which contains a StateCode column and a StateName column:

To configure the rule we set the following:

This option replaces what matches the specified regex expression with the replacement value, before writing to target field.

The following parameters are available.

All data types.

When replacing values in the target field, you may encounter a truncation error, often due to a mismatch in field sizes between the target field and the replacement value. To resolve this issue, adjust the field size of target field according to the replaced/overwritten value size.

 

This rule standardizes numeric values (including numeric strings) to a defined decimal precision using a specified rounding method, such as Half Up, Down, or Ceiling.

It provides the following benefits:

Use Case: Standardize the output to two decimal places and round the value up. Typically used to ensure consistent and accurate currency values for billing, reporting, and regulatory compliance.

To change the input value 123.4567 to the output value 123.46, parameters can be set as follows:

Use Case: Standardize measurement data to one decimal place and round the value down. Typically used to present clean, uniform data in dashboards and analytics for better readability and consistent reporting standards.

To change the input value 98.765 to the output value 98.7, parameters can be set as follows:

Use Case: Round sensor readings to nearest integer for threshold checks. Typically used to enforce quality standards and trigger alerts based on rounded values to maintain operational thresholds and compliance requirements.

To change the input value 49.8 to the output value 50, parameters can be set as follows:

Use Case: Standardize scientific instrument readings to two decimal places, using a method that minimizes rounding bias (Banker's Rounding). Typically used to ensure statistical neutrality in large data sets by rounding to the nearest even number when equidistant (for example, 5.725 rounds to 5.72, while 5.735 would round to 5.74), which is critical for accurate research and analysis.

To change the input value 5.725 to the output value 5.72, parameters can be set as follows:

Use Case: Calculate the number of pallets needed to ship an order, ensuring that any fraction of a pallet is counted as a full pallet to prevent shortages. Guarantees sufficient capacity or stock by always rounding a number toward positive infinity (up), ensuring that a calculated need is never underestimated.

To change the input value 12.01 (representing 12.01 pallets) to the output value 13, parameters can be set as follows:

Rounding Mode - Ceiling

Decimal Precision - 0

Use Case: Calculate a safe, conservative estimate for an allowable cost or discount amount, ensuring the value is never inflated. This can also be used to prevent over-invoicing, over-allocating funds, or over-reporting on a conservative metric.

To change the input value 45.999 to the output value 45.99, parameters can be set as follows:

Rounding Mode - Floor

Decimal Precision - 2

Rounding modes behave differently for negative numbers. This can be important in calculations (especially those related to temperature, finance, or measurement tolerances). The following table describes how each rounding mode calculates negative input numbers.

 

This rule standardizes the format of monetary values by converting one or more source field formats into a single specified format. You can either specify the source currency formats to convert (in Currency Identifiers), or automatically detect all currency formats (by selecting Convert all currency formats). Recognized values are converted to the specified format (in To Format), while any unrecognized values are output unchanged.

This rule has the following properties.

 

This rule standardizes date values by converting values from one or more source date formats into a single specified date format. You can specify one or more source “from formats” to identify which values should be converted. Matching values are converted into the specified “to format”, while any unrecognized values are returned unchanged. It supports automatic detection of source date formats.

This rule has the following properties.

This rule has the following parameters.

This rule standardizes time values by converting values from one or more source time formats into a single specified time format. You can specify one or more source “from formats” to identify which values should be converted. Matching values are converted into the specified “to format”, while any unrecognized values are returned unchanged. Automatic detection of source time formats including 12-hour, 24-hour, and dot-separated styles is supported.

This rule has the following properties.

This rule has the following properties.

 

This rule standardizes the format of phone number values by converting one or more formats in the source into a single specified format. You can either specify the source phone number formats to convert (in From Phone Formats), or automatically detect and convert all source phone number formats (by selecting Convert all phone formats). Recognized values are converted to the specified format (in To Format), while any unrecognized values are output unchanged. Match criteria is based on the length of the value.

You can also specify whether to include a country code, add one if it’s missing from the source data, and whether to include phone number extensions. This can be useful, for example, for ensuring data consistency for systems like call centers or SMS messaging, which require specific number formats to function properly.

This rule has the following properties.

 

This rule standardizes the format of postal code values by converting one or more formats in the source into a single specified format. You can either specify the source postal code formats to convert (in From Postal Formats), or automatically detect and convert all formats (by selecting Convert all postal code formats). Recognized values are converted to the specified format (in To Postal Format), while any unrecognized values are output unchanged. Match criteria is based on the length of the value and the order of placeholders. Accepted placeholders are A, C and N. The length and number of placeholders in the target format must exactly match the length and order of placeholders in the source format. For example, if the source format is "NNA CA", the target must also have the same number of characters. For example, "NNA-CA" is a match (and "NNAA-CA" and "NNAACA" are not matches).

You can also replace missing leading zeros that may have been removed by external systems.

This rule has the following properties.

 

This rule converts US state values into a consistent format—full name (e.g., California), abbreviation (e.g., CA), or ISO code (e.g., US-CA). Accepts any of these formats as input and matches values case-insensitively. Unrecognized values are returned unchanged.

This rule has the following properties.

 

This rule trims whitespace from a String field.