Skip to content
M↓ View as Markdown ↗

Accuracy and Limitations

Actian AI Analyst is an AI-powered analytics system built on large language models (LLMs). Because its output is generated, not read from a fixed set of pre-computed reports, AI Analyst can make mistakes — including mistakes that sound confident and plausible. This page explains where answers come from, which factors influence their accuracy, what the known limitations are, and which responsibilities remain with your organisation when using the product. It is provided as a transparency statement, in the spirit of the transparency obligations for AI systems under Regulation (EU) 2024/1689 (the "EU AI Act").

Info

In short: AI Analyst is an LLM-powered system and can make mistakes. The accuracy of its results depends strongly on the quality of your semantic layer and your underlying data. Treat every answer as a well-informed draft to be verified — not as a final, audited figure.

How AI Analyst produces answers

AI Analyst does not search your raw database with an AI model. When a user asks a question, the agent:

  1. Looks up the relevant business definitions in your semantic layer — models, metrics, dimensions, filters, and glossary terms
  2. Writes a query in SemQL, which is compiled to SQL and executed read-only against your data source
  3. Sends the returned rows to an LLM, which produces a natural-language answer with full source traceability

This design has an important consequence for accuracy: the numbers come from your database via a governed, deterministic query, while the interpretation and narrative around them are generated by an LLM. Most accuracy risk lives in the interpretation step — choosing the right metric, applying the right filter, and describing the result correctly — which is why the quality of your semantic layer matters so much.

For details on this process, including what data is sent to AI providers, see How AI Analyst handles your data.

Factors that influence accuracy

📘 Semantic layer quality — The agent answers using your definitions. If metrics are unnamed or misnamed, models lack descriptions, or the glossary doesn't capture your team's vocabulary, the agent may select the wrong measure, apply the wrong filter, or misinterpret a business term. A well-maintained semantic layer is the single biggest lever for accurate answers.

🗄️ Underlying data quality — AI Analyst reports what is actually in your connected data source. Stale, incomplete, duplicated, or incorrect source data produces factually correct queries over incorrect data — and the agent cannot detect this on its own.

❓ Question clarity — Ambiguous questions ("How is revenue trending?", "Who are our active users?") are resolved against your glossary and model definitions. Where those definitions are missing or contested, the agent must make a judgement call that may not match the reader's intent.

🤖 LLM behaviour — Like all generative models, the underlying LLMs can misread intent, over-generalise, or produce phrasing that sounds more certain than the evidence warrants. Model behaviour can also change subtly as providers update their models.

Known limitations

  • AI Analyst can make mistakes. Answers may contain errors in interpretation, phrasing, or emphasis — even when the underlying query results are correct. Treat generated text with the same scepticism you would apply to any draft written by a fallible author.
  • Answers are non-deterministic. Asking the same question twice can yield slightly different queries, phrasings, or interpretations, particularly for ambiguous questions.
  • Completeness is bounded by your setup. The agent can only use data sources that are connected, models that are defined, and terms that are documented. It will not know about data you have not modelled, and it may not notice that a relevant data source is missing.
  • Generated summaries can lose nuance. The narrative, chart titles, and key takeaways are condensed by an LLM and may understate caveats present in the underlying results.
  • AI Analyst is not professional advice. It is an analytics tool. Its output is not legal, financial, tax, or audit advice, and should not be used as a substitute for qualified professional judgement.
  • Availability of specific models may change. AI Analyst uses LLMs from third-party providers; the models available and their behaviour may change over time.

Responsible use

AI Analyst is designed to keep humans in control. To use it responsibly — and in line with the human-oversight expectations of the EU AI Act — we recommend:

  • Verify before consequential decisions. Check the underlying query, result rows, and source traceability shown with an answer before relying on it for reporting, decisions with financial or legal impact, or external communication.
  • Keep meaningful human oversight. A person should review and approve AI-generated analysis before it drives significant decisions or is published. Do not fully automate consequential workflows on AI Analyst output alone.
  • Invest in your semantic layer. Define models, metrics, and glossary terms carefully, and refine them when you notice misinterpretations. Steward can help keep definitions current — see Using Steward.
  • Be precise when asking. Naming the metric, timeframe, and filter you intend makes the agent's interpretation verifiable. See Tips for Asking Questions.
  • Review usage via audit logs. A record of who asked what and when is available for your compliance needs — see Audit Logs.
  • Build AI literacy. Make sure users in your organisation understand they are interacting with an AI system, that it can make mistakes, and how to verify its output.
  • Report problems. If AI Analyst produces an incorrect or misleading answer, report it to your administrator or contact support so the underlying definitions can be corrected.

Warning

This page is provided for general information and transparency purposes. It does not constitute legal advice and does not replace the terms of your agreement with Actian. How the EU AI Act applies to your specific use of AI systems — including your obligations as a deployer — depends on your role and context, and should be assessed with your own legal counsel.

For more on how the platform processes data, see How AI Analyst handles your data and How Actian AI Analyst protects your data.