Running Profile
After validating the profile, you can run the profile. Click 
, to run the profile.
, to run the profile. If the run is successful, the Results tab displays the status message for each event as it executes. This information can be used for monitoring purposes. The following information is displayed:
• Time: The time stamp for when the message was logged.
• Step Name: Name of the data flow operator for which the statistics are provided. Name of the operator is the full path to the actual operator processing the data in dataflow. One common pattern to notice in the name is the name ending with ".input" which always tells the number of records received by a particular operator for processing. Left most part of the name is the parent operator while the right most part is the last child in the hierarchy which is doing the work. Most common parent operator names which will be seen are:
– DeriveFields - Derives fields into other fields for example, StringToInteger or any internal conversions done by the engine for achieving a use case.
– DataQualityAnalyzer - Performs quality tests using the rules configured by the user. For example, consider IsNotNull - so in this case, the quality tests will check if the field is not null and calculate the percentage of pass/fail.
– PassTargetWriter - Writer instance for writing data to the pass target. This would look like PassTargetWriter.WriteSink in case of text files.
– FailTargetWriter - Writer instance for writing data to the fail target. This would look like FailTargetWriter.WriteSink in case of text files.
– DrilldownTargetWriter - Writer instance for writing data to the drilldown target. DrilldownTarget.WriteSink would tell the number of bytes written to the drilldown file.
• Status: Either executing or completed based on the state of the operator. In case of error or profiling aborted, the status would be Error.
• Error Code: In case of an error/profile abort, this column will be populated with the error code.
• Message: Message displaying statistics about the operator progress. Only five types of messages can be seen
– Execution Started - Execution has started
– <number> bytes read (in case of delimited text/fixed text on the source side)
– <number> records read - shown for all the operators configured in the dataflow graph except for source. The operator name in this case ends with ".input"
– <number> bytes written (in case of delimited text/fixed text on the target side and for drilldown)
– Execution ended in error (in case of an error or profiling aborted)
– Profile Execution Successfully Completed, ...
Note: The progress result is only available when profile is run from UI, executing from invoker and command line will not display these messages.
Note: In case of smaller datasets, when the profile is run for the second time and so on, you may just see only the start and end message.
The Statistics tab displays the profile results in the drilldown charts view and summary rule. However, if you are on another tab in the profile and if you click validate, then the Statistics tab opens by default to view the results.
1. In the Profile Run Statistics window, the Results Summary pane displays the following information for each rule:
• Field: name of the field profiled
• Rule: the type of rule used in the profile
• Description: the profile statistics, for example, the percentage of passed and failed data
2. On the Results Summary pane, click on the required field name to view the pass or fail data in the graph representation on the right pane. Also, you can click on the ***The Counts- Total*** field to view the total pass and fail data in the pie chart representation and **Rule - BarCharts** field to view all the pass and fail rule in the bar chart representation.
Note: You can click on parts of the graph to view the data in the Drill Down Data view. You can also set pass and fail data settings in profile properties to create text files that you can browse.
Viewing Pass, Fail, and Drill Down Output
To view pass, fail, or drill down data output:
1. Run the profile and rule results and a graph of the selected rule is displayed on the Statistics tab.
2. In the Project Navigator view, expand the project folder and double-click the following files to view the output data in the editor.
• Fail.txt - FAIL_TARGET output displays data for the records that did not meet the specified criteria within the applied rule.
• Pass.txt - PASS_TARGET output displays output data for the records that met the specified criteria in the applied rule.
• Stats.json - STATS_TARGET output displays a graphical representation of the rule from the profile run that is automatically generated. However, it is not suggested to integrate this file into automated business processes.
• DrillDown.txt - DRILLDOWN_TARGET displays output data for the records that does not meet the criteria specified in the applied rule. This file is primarily used by Data Profiler to enable drill down reporting. However, you can use this file to browse in the editor to illustrate the relationship between the FAIL_TARGET records and the corresponding rule.
Note: The DRILLDOWN_TARGET uses the Boolean data representation or date/time formats from the source fields. So if the source has Y/N for Boolean field, then DRILLDOWN_TARGET will also contain Y/N and if the source has True/False then DRILLDOWN_TARGET will contain True/False. The only exception is for DMS connectors (Netsuite/ServiceNow/Oracle CRM). For these connectors, DRILLDOWN_TARGET will always use, True/False for Boolean, and ISO-8601 format for date. For example, 2022-05-16T08:24:16+00:00.
Rule and Their Resulting Outputs
Some rule display their results in pie charts and some results are displayed in bar charts. However, Distinct Values or Duplicate Values do not create graphic output. There are two charts that represent overall results:
• Counts Total displays pass and fail counts in a pie chart graph.
• Rule-BarChart displays all pass and fail rule data in a single bar chart with each rule pass and fail represented in a separate bar.
• Each rule pass and fail data is shown in an individual pie chart.
The following table provides information about the outputs for each rule and rule function:
Rule Name | Graphic | Additional Output |
|---|---|---|
Compare to Constant, Compare to Field | Pie Chart | None |
Is Not Blank, Is Not Null | Pie Chart | None |
Maximum, Minimum, Mode, Sum | None | Statistical values |
Standard Deviation | None | Statistical values |
Equal Range Binning, Most Frequent Values | Bar Chart | None |
Distinct Values | None | Text file of distinct values |
Duplicate Values | None | Text file of distinct values |
Last modified date: 10/22/2024