Skip to content

Architect the Solution

When starting a new data quality initiative, first identify the data quality problems and business requirements you need to address. In conforming to the Agile Methodology, create the requirements document but only invest as much time for the team to understand the objectives and the required outcomes. Make sure to update this requirements document as you progress.

Business Requirements

Identify the data quality challenges to solve and state them in non-technical language. For example:

Objective: We need to ensure customer data quality before migrating to the new CRM system in Q1 of the next fiscal year.

Capture the data quality processes that your solution will address as a workflow diagram. Include data sources, validation checkpoints, and interactions with downstream systems.

The 'customer data validation' workflow is described as:

  • Extract customer records from legacy CRM system
  • Validate email addresses, phone numbers, and postal codes
  • Identify and flag duplicate customer records
  • Standardize address formats and state abbreviations
  • Route validated records to pass target for CRM migration
  • Route failing records to fail target for manual review and correction

Document the requirements that will determine success from a business perspective and not from a technological perspective. For example:

Success criteria: We need the data quality solution to:

  • Validate 100% of customer records before migration
  • Achieve minimum 95% pass rate for critical fields (email, phone)
  • Identify and resolve all duplicate customer records
  • Standardize formats for addresses, phone numbers, and state codes
  • Provide detailed quality reports showing pass/fail statistics by field and dimension

The Business Requirements Document is used to explain and get buy-in from business stakeholders and provide context to the implementation team for the technical requirements.

Technical Requirements

After documenting the business requirements, compile the mapping documents that bridge the gap from architecture to design. Essentially, this means defining:

  • Data sources: Source systems, file formats, database connections, and connector configurations. See source connectors and target connectors.
  • Quality dimensions: Which data quality dimensions to measure (accuracy, completeness, consistency, validity, uniqueness, timeliness)
  • Profile rules: validation rules that check data without modifying it (IsNotNull, ValidateEmail, ValidatePhoneNumber, IsNotDuplicate, and more)
  • Remediation rules: transformation rules that correct data (StandardizePhoneNumber, StandardizeUSState, DeidentifyValue, and more)
  • Target outputs: Where passing and failing records should be routed. See Set Target.
  • Quality metrics: Data Quality Index (DQI) thresholds, dimension scores, and rule weights
  • Execution options: Sampling strategies, alert thresholds, and logging preferences, configured on the Execution tab

You must document to the extent practical: rules for data governance, field mappings and semantic types, quality dimensions and scoring criteria, access for design and runtime users, and the promotion life cycle from development to test and to production.

Data Quality Design Architecture

Design approaches, rule categories, and dimensions

Your architecture builds on three DataConnect concepts that are documented in full elsewhere:

  • Design approaches — Automate Design, Design Assistance, and Manual Design. See Three Design Approaches.
  • Rule categories — profile rules (validation, read-only), remediation rules (transformation), and data prep rules (type conversion and derived fields). See Data Quality Designs.
  • Quality dimensions — Accuracy, Completeness, Consistency, Timeliness, Uniqueness, and Validity. Associate each rule with a dimension and assign a weight (1–5, where 5 is most important) so it contributes to the Data Quality Index (DQI) and per-dimension scores. See Managing Data Quality Dimensions.

The rest of this section covers what is specific to solution architecture: the execution workflow and target configuration.

Execution Workflow

The data quality execution workflow follows these steps:

  1. Connect: Configure the data source connection and connector properties.
  2. Remediate: Remediation rules run first to standardize and correct the data.
  3. Analyze: Profile rules validate the corrected data against your configured criteria.
  4. Score: Calculate quality metrics (DQI, dimension scores, pass/fail counts).
  5. Route: Separate records into pass and fail targets based on rule results.
  6. Report: Generate detailed statistics by field, rule, and dimension.

Target Configuration

Data quality designs write results to two separate targets:

  • Pass Target: Records that meet all quality criteria are routed here for immediate downstream processing
  • Fail Target: Records that fail quality checks are routed here for review, correction, and reprocessing

Both targets can be configured with their own connector type, connection properties, and output format. For details, see Understanding Pass and Fail Targets.

Data Quality Design Best Practices

Rule Organization

  • Group related rules by field or business entity
  • Use meaningful rule names that describe the validation or transformation
  • Associate rules with appropriate quality dimensions for comprehensive scoring
  • Set rule weights to reflect business importance
  • Use remediation rules to standardize formats before validation

Sampling Strategy

  • Use appropriate sampling during design phase to speed up rule testing
  • Configure different sampling strategies for different environments:
    • Development: Sample a small record range (for example, the first 1,000–10,000 records) for fast iteration
    • Test: Use a representative sample—a larger record range or every Nth record—to validate across the data
    • Production: Process all records (no sampling)

Alert Configuration

Configure alerting SLAs on the Execution tab to catch quality regressions:

  • Runtime SLA: Flag runs that exceed an expected duration.
  • Data Quality Index (DQI) threshold: Flag when the overall DQI drops below your target.
  • Dimension thresholds: Set a minimum score per dimension (Accuracy, Completeness, Consistency, Timeliness, Uniqueness, Validity) for business-critical characteristics.

Set stricter thresholds for critical fields and dimensions, and looser thresholds where some issues are expected. You can monitor the live DQI and per-dimension scores in the Data Quality Index panel on the Rules page.

Logging Configuration

  • Use Debug logging during development and testing, and a lower level in production.
  • The available log levels form a superset, each adding to the one before:
    • Errors: Log errors only (lowest overhead).
    • Warnings: Errors and warnings.
    • Information: Errors, warnings, and informational messages.
    • Debug: All messages, including debug detail.

The Technical Requirements document will be constantly updated as unknown requirements are discovered. The next sections provide best-practices and examples for managing these concepts effectively using Actian DataConnect.