Release Notes¶
What's New in DataConnect 14.0¶
| Feature | Description | Key Benefits |
|---|---|---|
| Redesigned Studio Interface | DataConnect 14.0 features a redesigned desktop application with a modern, intuitive, AI-enabled interface. • Modern look and feel • Improved data quality design editor • Customizable themes • More responsive interface • Optimized for Windows 10 and Windows 11 | Better User Experience: • Faster loading and response times • Easier navigation and workflow • Optimized memory usage • Smoother overall performance |
| Enhanced Data Quality Engine | The Data Quality Engine has been upgraded with Apache Spark technology for better performance and new capabilities. • Platform Support: Studio available for Windows; engine available for both Windows and Linux • Comprehensive rule library for profiling, data preparation, and remediation | Better Data Quality: • Process data faster with improved performance • Handle larger datasets more efficiently |
| AI-Powered Data Quality | DataConnect 14.0 introduces AI assistance across the data quality workflow. • AI rule recommendations: when you scan a source, DataConnect analyzes the data (Data Discovery and semantic-type detection, including simple PII detection) and recommends rules — applied automatically with Automate Design, or offered for review with Design Assistance • AI Assistant (Ask AI): create rules and explore your data using natural-language requests • Bring your own AI provider: DataConnect does not host an LLM — connect your own provider and model (OpenAI, Anthropic, Microsoft Azure, Google Gemini, Google Vertex, Groq, AWS Bedrock, Ollama, OpenAI-compatible) in Settings > AI Provider | Faster, guided data quality: • Identify and fix issues faster with AI guidance • Build rules without deep rule expertise through fully automated design or Design Assistance • Use the AI provider and model you prefer |
| Data Quality Target Connectors | DataConnect 14.0 writes Data Quality pass and fail results to file and database targets. This release supports five target connectors — ASCII (Delimited), ASCII (Fixed), Unicode (Delimited), Unicode (Fixed), and JDBC — with additional target connectors planned for a future update. | Write results where you need them: • Route passing and failing records to file or database targets • Land validated data via JDBC or delimited/fixed text files |
| Three Design Approaches | Build a data quality design the way you want, with three approaches that range from fully automated to fully manual. • Automate Design: Minimum user participation — DataConnect creates and chooses the rules for you • Design Assistance: Partial participation — DataConnect creates the rules and you choose which ones to apply • Manual Design: Full participation — you decide which rules to create | Flexible rule creation: • Let DataConnect do the work or stay in full control • Faster design with AI-powered recommendations • Review and approve rules before they are applied |
| Data Discovery | Automatic, per-field analysis of your source data that runs as soon as you connect. • Field insights: unique and duplicate values, string patterns, common values, null/blank counts, and numeric statistics (min, max, mean, median, standard deviation) • Summary and Table views, plus a maximized view with String pattern, Frequent values, and Statistics tabs • Click to filter: click a pattern or value to filter the Data Browser to matching records • Change/Standardize workflows turn findings into profiling and remediation rules that normalize inconsistent patterns or values | Understand data before you build rules: • Spot anomalies, outliers, and inconsistencies quickly • Turn findings directly into rules |
| Interactive Data Browser | Explore and validate results in a dynamic, color-coded data grid. • Auto-refresh: with Auto-Execute on, the design re-runs and every panel updates as you add, change, or remove rules • Color-coded records: green for passed, red for failed, purple for remediated • Click to filter: click a Pass, Fail, or Remediated count or a segment of the Data Quality Index gauge to filter the grid and shade the affected column • Per-rule drill-down: click a rule's pass/fail/remediated bar to see just that rule's records for its field | See quality at a glance: • Validate a design before setting targets • Pinpoint failing records instantly • Confirm remediation worked as expected |
| Data Prep | Shape the structure of your data before it is written. • Schema control: add, remove, rename, and reorder fields, and change field types and sizes • Derived fields: create in-memory fields — for example, combine first_name and last_name into a single name field• Data prep rules: a library of rules for joining, splitting, converting, parsing, looking up, and transforming field values • Derived fields can feed further rules or be written to the output | Get data into the right shape: • Conform output to any target schema • Build derived values for analysis and output • Reshape data without changing the source |
| SQL Expression Editor | A Spark SQL expression editor for the ExecuteExpression and Assert rules, replacing the v12.5 DataRush scripting language. • Spark SQL syntax with syntax highlighting and line numbers • Searchable function catalog: Conversion, Date and Time, Math, String, Aggregate, and Window functions, plus operators and conditional constructs • Real-time validation of your expression as you type • What changed from 12.5: expressions now use Spark SQL instead of DataRush scripting, so existing scripts must be rewritten (see the migration note below) | Build expressions with confidence: • Insert fields and functions from a catalog instead of memorizing syntax • Catch errors immediately with live validation • Standard, portable SQL |
Maps and Processes
Map and process designers are not yet available. These designers will be available in a future release. However, the standalone Runtime Engine continues to support maps and processes from previous DataConnect versions via the command line interface.
Migration Required for Spark Processing
Due to the migration to Spark-based processing, the DataRush scripting language for Execute Expression and Assert rules now uses Spark SQL syntax. These rules support Spark SQL expressions only.
Action Required:
- Existing scripts must be rewritten to comply with SQL syntax
- Some existing .dp/.dq files may not load correctly and will need to be recreated
- If a file does load, review any rules that no longer exist and remove or disable them before running
Change Log - version 14.0¶
Build 14.0.0-43¶
The following table lists the key features and enhancements delivered in the DataConnect 14.0 release.
| Issue ID | Description |
|---|---|
| DCIM-82, DCIM-83, DCIM-84, DCIM-87 | New Electron-based Design Studio built on the v14 application framework, with a modern UI. |
| DCIM-114 | New Settings page for preferences, macros, AI provider, execution, and licensing. |
| DCIM-5, DCIM-7, DCIM-8 | Source, Target, and Configuration editor pages, with supporting services and engine. |
| DCIM-12, DCIM-13 | Source scan workflow to profile a source and generate rules. |
| DCIM-33 | Data Discovery panel with per-field analysis and semantic-type detection. |
| DCIM-26 | AI Suggested Rules panel that recommends rules from source analysis. |
| DCIM-40, DCIM-68 | Data Quality Rules panel, including manual rule creation. |
| DCIM-47 | Interactive Data Browser to explore and validate results. |
| DCIM-54 | Data Prep screen to reshape the schema and create derived fields. |
| DCIM-19, DCIM-150 | DQ Assessment Report page summarizing quality metrics and issues. |
Known Issues¶
The following table lists known issues in the current release.
| Issue ID | Description | Resolution |
|---|---|---|
| DCIM-3571 | When you use Automate Design or Design Assistance on a field inferred as Timestamp, DataConnect can recommend ValidateDateFormat and StandardizeDateFormat rules configured with a timestamp pattern (for example, MM-dd-yyyy HH:mm:ss). These rules apply to Date values only, so that pattern cannot be set for them in Manual Design, and the recommended StandardizeDateFormat rule is given the same From and To format. The recommended rules are still added and execute without error. | Scheduled for patch (August 2026). |
| DCIM-3570 | Writing to a JDBC target can fail during execution when the source table contains a data type that the target database does not support. Pre-creating the target table raises a DesignSdkException — for example, against DB2: Error during JDBC table pre-creation for table [...]: DB2 SQL Error: SQLCODE=-20441 ... SQLERRMC=BOOLEAN — so the target table is not created and the run does not complete. | Scheduled for patch (August 2026). |
| DCIM-3566 | With Unicode (Fixed) as the target, a derived field converted to the Timestamp data type (for example, using a StringToConversion data prep rule) is shown with the wrong data type — Name instead of Timestamp — in the target schema. Closing and reopening the design shows the correct type. | Scheduled for patch (August 2026). As a workaround, close and reopen the data quality design to refresh the target schema. |
| DCIM-3565 | Writing to an HCL Informix database through the JDBC target fails during execution with a DesignSdkException: Error during JDBC table pre-creation for table [...]: A syntax error has occurred. The target table is not created, so the run does not complete. | Scheduled for patch (August 2026). |
| DCIM-3564 | With a JDBC source connected to an HCL Informix database, clicking Inspect Data fails with Inspection failed: An unexpected error occurred, even though the connection succeeds and the source data previews correctly. | Scheduled for patch (August 2026). |
| DCIM-3534 | With a JDBC source, using Design Assistance to re-scan only a single field or a subset of fields produces no AI rule suggestions (the rules recommended earlier for those fields are not shown, and the Data Discovery dropdown appears instead). Scanning with all fields selected works as expected, and other connector types such as flat files are not affected. | Scheduled for patch (August 2026). As a workaround, re-scan with all fields selected. |
| DCIM-3522 | Casting a Date value to a String with the ExecuteExpression rule can return a date that is off by one day. A time-zone offset is applied during the conversion instead of preserving the source's exact (wall-clock) date, so the derived string may not match the source date — for example, Oct 22, 2019 can be written as 2019-10-23. | Scheduled for patch (August 2026). |
| DCIM-3521 | The ValidateTimeFormat and StandardizeTimeFormat rules do not correctly reject invalid times when the format uses the K or k hour characters. Out-of-range values such as 24:30:00 are accepted as valid (and converted by StandardizeTimeFormat) instead of failing. Formats using H and h are not affected. | Scheduled for patch (August 2026). As a workaround, use the H or h hour characters in the format pattern instead of K or k. |
| DCIM-3492 | Running a data quality profiling job through the standalone (embedded) engine on JRE 17 fails when the source is a database with a typed column (such as a DATE column). The job errors while writing the target (for example, 10031: Error writing to Pass Target ... [EXPRESSION_DECODING_FAILED]) because the profiler's forked child process is missing a required Spark-on-JDK-17 JVM option. File-based sources are not affected (their fields are treated as strings), and the DataConnect Studio profiler and the dataprofiler.bat / dataprofiler.sh scripts are not affected. | Scheduled for patch (August 2026). As a workaround, run the profiling job from the DataConnect Studio profiler or with the dataprofiler.bat / dataprofiler.sh scripts, which are not affected. |
| DCIM-3489 | The DataTypeConversion data prep rule does not correctly convert a Float field to a numeric type (such as Decimal); the resulting derived field does not contain a valid numeric value. | Scheduled for patch (August 2026). |
| DCIM-3483 | Opening an existing data quality design after relaunching DataConnect can drop its saved source connection: clicking Edit source shows an empty Select Connector field and no connection details instead of the saved source (auto-execute also fails on open in this state). | Scheduled for patch (August 2026). As a workaround, close the design and reopen it — this may take more than one attempt. |
| DCIM-3471 | Long-running data quality jobs do not emit progress events during execution. While a job runs, no per-artifact logs or progress updates appear — only a workspace-level status entry (for example, execution status running) is recorded — so progress cannot be tracked until the job finishes. Log timestamps are recorded in UTC, so they may not match your local start time. | Scheduled for patch (August 2026). |
| DCIM-3467 | When a DeidentifyValue remediation rule and an IsNotDuplicate profiling rule are applied to the same field, IsNotDuplicate can incorrectly send records to the Fail results even when they are not duplicates. The rules work correctly when applied individually; the issue has been observed with a Salesforce source (flat-file sources such as ASCII (Delimited) are not affected). | Scheduled for patch (August 2026). As a workaround, apply the DeidentifyValue and IsNotDuplicate rules to the field individually rather than together — each works correctly on its own. |
| DCIM-3420 | On the DQ Source page, changing the source connection and clicking Connect applies the new source immediately. The Replace schema? dialog (Overwrite / Keep current) appears only after the switch, so Keep current cannot restore the previous source — it leaves the design with the old schema and rules pointing at the new source's columns. A later Inspect or Scan then fails to resolve the changed columns (for example, a column cannot be resolved). | Scheduled for patch (August 2026). As a workaround, when you change the source, choose Overwrite to apply the new schema and then re-scan to regenerate the rules; do not rely on Keep current to undo a source change. To restore the original source, reconnect to it and choose Overwrite. |
| DCIM-3403 | In the Build Connection URI form, macros are not resolved reliably for all connection schemes. Macros work for the File, GZip, and FTPS schemes, but may not resolve before the connection is attempted for the Azure, GCP, and AWS S3 schemes. | Scheduled for patch (August 2026). As a workaround, enter literal values instead of macros for the Azure, GCP, and AWS S3 schemes. |
| DCIM-3325 | When you resolve a rule conflict by clicking Merge, the merged rule does not retain its configured parameter values, so it fails during execution (for example, merging two ValidateDateFormat rules that use different date format patterns). | Scheduled for patch (August 2026). |
| DCIM-3214 | When you edit a Date field's cell on the Rules page, the value is shown in full JavaScript timestamp form (for example, Mon Dec 30 2019 05:30:00 GMT+0530 (India Standard Time)) instead of the formatted date (Dec 30, 2019). The displayed source data is not affected. | Scheduled for patch (August 2026). |
| DCIM-2695 | Two instances of DataConnect running for the same user can open the same workspace at the same time. DataConnect does not lock a workspace that is already open in another instance, so it does not prevent concurrent access or prompt the user to choose a different workspace (as it did in 12.5). | Scheduled for patch (August 2026). |
| DCIM-1740 | In the Build Connection URI form, connecting with the AWS S3 scheme using the Default Credential Provider Chain authentication type (DEFAULT_AUTH) does not connect to the target object. This scenario worked in DataConnect 12.5. | Scheduled for patch (August 2026). As a workaround, use a different AWS S3 authentication type instead of the Default Credential Provider Chain. |