Configure inference¶

Availability

Deployment: Self-managed deployments only. On Actian SaaS, Actian provides inference.

No model, model weights or model runtime ships with Actian AI Analyst. On a self-managed deployment, you connect AI Analyst to APIs that you run:

A chat API (LLM endpoint). The AI Analyst and Steward use it to answer questions. Without one, chat doesn't work.
An embedding API (Embedding endpoint). Optional. AI Analyst uses it to turn your semantic layer into vectors for search. Without one, search matches keywords only.

You configure these endpoints once for the whole deployment. This is the deployment default. Organizations on the deployment can also add their own endpoints in the app.

Before you start¶

You need:

An OpenAI-compatible chat completions API that supports tool calling. For example, vLLM, Text Generation Inference (TGI), Ollama, LiteLLM, or Azure OpenAI in your own tenant. The AI Analyst calls tools to do its work, so a model without tool calling can't answer questions.
Optionally, an OpenAI-compatible embeddings API.
Network reach from the AI Analyst server to both APIs. See Reach endpoints on your network.

How AI Analyst chooses an endpoint¶

For each organization, AI Analyst uses the first endpoint that exists:

The organization's own endpoint, added under Settings → Inference.
The deployment default that you configure on this page.
None. Chat is blocked, and search matches keywords only.

Chat and search are chosen separately. For example, an organization can use its own LLM endpoint with the deployment's Embedding endpoint.

Administrators can see which endpoint their organization uses under Settings → Inference. The Source for each of Chat and Search is one of:

Source	Meaning
Provided by your administrator	The deployment default you configure on this page.
Your organization's own endpoint	An endpoint the organization added.
None	No endpoint.

Set the deployment default LLM endpoint¶

Set these keys in .env (Compose bundle) or in the server Secret (Helm chart):

Key	Required	Description
`LLM_PROVIDER`	To use a deployment default	Set to `openai-compatible`. If you leave it unset, the deployment has no default LLM endpoint (see If you leave inference unset).
`LLM_BASE_URL`	Yes	The API's base URL, including `/v1`, for example `http://llm.example.internal:8000/v1`. AI Analyst appends `/chat/completions`.
`LLM_MODEL`	Yes	The model id that the API serves.
`LLM_CAPABILITY_PRESET`	Yes	`generic` or `reasoning`. See the table below.
`LLM_API_KEY`	No	Sent as a bearer token. Leave it unset if the API needs no key.
`LLM_FAST_MODEL`	No	A second model for quick calls, such as conversation titles. Defaults to `LLM_MODEL`.

AI Analyst doesn't guess what a model supports from its name. You declare it with LLM_CAPABILITY_PRESET:

Preset	Use it for	Context window AI Analyst assumes
`generic`	Models with tool calling and no reasoning options.	128,000 tokens
`reasoning`	Models that accept a reasoning effort setting (`reasoning_effort`).	128,000 tokens

If LLM_PROVIDER is set and any required key is missing, or the preset is unknown, the server refuses to start and names the key.

Compose bundleHelm chart

In .env:

LLM_PROVIDER=openai-compatible
LLM_BASE_URL=http://llm.example.internal:8000/v1
LLM_API_KEY=<API_KEY>
LLM_MODEL=<MODEL_ID>
LLM_CAPABILITY_PRESET=generic

Then run docker compose up -d --wait.

Add the keys to the server Secret, for example when you create it:

kubectl -n ai-analyst create secret generic ai-analyst-secrets \
  --from-literal=LLM_PROVIDER=openai-compatible \
  --from-literal=LLM_BASE_URL=https://llm.example.internal/v1 \
  --from-literal=LLM_API_KEY=<API_KEY> \
  --from-literal=LLM_MODEL=<MODEL_ID> \
  --from-literal=LLM_CAPABILITY_PRESET=generic \
  ...

If the Secret already exists, update it and restart the server pods. See How you supply configuration.

Use only openai-compatible

LLM_PROVIDER also accepts anthropic and openai. These values select Actian Inference. They aren't supported on self-managed deployments. An Air-gapped deployment refuses to start with either value, and also refuses ANTHROPIC_API_KEY and OPENAI_API_KEY.

Set the deployment default Embedding endpoint¶

Embeddings are optional. Without them, search in the semantic layer matches keywords only, and everything else keeps working.

AI Analyst knows two Embedding presets, models with calibrated search settings. You can also use any other model as a custom model.

`EMBEDDING_MODEL`	Type	Dimensions	Notes
`text-embedding-3-small`	Preset	1536 (fixed)	AI Analyst sends the model name `text-embedding-3-small`.
`qwen3-embedding-0.6b`	Preset	1024 (fixed)	AI Analyst sends the model name `Qwen/Qwen3-Embedding-0.6B`.
Any other model id	Custom	Set `EMBEDDING_DIMENSIONS`	AI Analyst sends the id as you wrote it. Search uses a loose match threshold and scans all vectors, and the server logs a warning at startup that the model is uncalibrated.

Key	Required	Description
`EMBEDDING_MODEL`	To turn on embeddings	A preset id or your own model id.
`EMBEDDING_BASE_URL`	Yes on Air-gapped, and for `qwen3-embedding-0.6b` and custom models	The API's base URL, without `/v1`, for example `http://embeddings.example.internal:8080`. AI Analyst appends `/v1/embeddings`.
`EMBEDDING_API_KEY`	No	Sent as a bearer token.
`EMBEDDING_DIMENSIONS`	For a custom model	The length of the vectors the model returns, as a positive whole number. Don't set it for a preset.
`EMBEDDING_MAX_DISTANCE`	No, custom models only	The cosine distance above which a vector match is ignored, greater than 0 and at most 2. The default for custom models is `0.8`.
`EMBEDDING_QUERY_PREFIX`	No, custom models only	Text that AI Analyst puts in front of each search query before it embeds it, for models that expect an instruction.

Always set EMBEDDING_BASE_URL to your own endpoint

On a Connected deployment, the text-embedding-3-small preset without EMBEDDING_BASE_URL calls a vendor-hosted API. If semantic-layer text must stay inside your network, always set the base URL to your own endpoint.

Base URL paths differ

Include /v1 in LLM_BASE_URL, but leave it out of EMBEDDING_BASE_URL. An EMBEDDING_BASE_URL that ends in /v1 makes AI Analyst call /v1/v1/embeddings, which fails.

A wrong embedding setting stops the server

Leaving EMBEDDING_MODEL unset is fine. A setting that's present but wrong stops the server at startup. For example: a custom model without a valid EMBEDDING_DIMENSIONS, or a preset with EMBEDDING_MAX_DISTANCE or EMBEDDING_QUERY_PREFIX.

Example for a custom model in .env or the Secret:

EMBEDDING_MODEL=<MODEL_ID>
EMBEDDING_BASE_URL=http://embeddings.example.internal:8080
EMBEDDING_API_KEY=<API_KEY>
EMBEDDING_DIMENSIONS=1024

Vector length is checked. Every vector the endpoint returns must have the configured number of dimensions. AI Analyst refuses any other length and logs an error, because such vectors can't be compared with the stored ones.
Text Embeddings Inference servers are checked too. If your endpoint is a Hugging Face Text Embeddings Inference server, AI Analyst asks it which model it serves. The served model name must match the configured model, or AI Analyst refuses to embed. For a custom model, EMBEDDING_MODEL must equal the model id the server reports, for example BAAI/bge-m3. For a preset, the served revision must match too. The error in the server log names the model and revision it expects.
The Compose bundle passes only some embedding keys. It passes EMBEDDING_MODEL, EMBEDDING_BASE_URL, EMBEDDING_API_KEY and EMBEDDING_DIMENSIONS. It doesn't pass EMBEDDING_MAX_DISTANCE, EMBEDDING_QUERY_PREFIX or the EMBEDDING_NEXT_* keys.
Leave the Helm chart's embeddings section off. The chart contains an embeddings section that isn't for customer installs. Keep embeddings.enabled: false (the default) and set EMBEDDING_* in your Secret. If you turn the section on, it overrides your EMBEDDING_* settings.

When you set or change the embedding model, the server's background jobs embed your organizations' semantic layers. Until that finishes, search matches keywords for anything not yet embedded.

Change the embedding model¶

You can switch models in one step or without a gap in semantic search.

In one step. Change the EMBEDDING_* keys and restart. On its next start, the server deletes the vectors of the old model and embeds everything again with the new one. Until that finishes, search matches keywords only for anything not yet re-embedded.

Without a gap (Helm chart). Use the EMBEDDING_NEXT_* keys. They take the same names as the EMBEDDING_* keys, with NEXT_ added: EMBEDDING_NEXT_MODEL, EMBEDDING_NEXT_BASE_URL, EMBEDDING_NEXT_API_KEY, EMBEDDING_NEXT_DIMENSIONS, EMBEDDING_NEXT_MAX_DISTANCE and EMBEDDING_NEXT_QUERY_PREFIX.

Add the new model as EMBEDDING_NEXT_* in the Secret, and restart the server pods. AI Analyst now writes vectors for both models and fills in the new model's vectors in the background. Search keeps using the current model.
Leave the EMBEDDING_NEXT_* keys in place while the background jobs fill in the new vectors. AI Analyst doesn't report when this finishes, and search keeps using the current model meanwhile. If you move on before it finishes, search matches keywords only for anything not yet embedded.
Move the new values into the EMBEDDING_* keys, remove the EMBEDDING_NEXT_* keys, and restart the server pods. Search switches to the new model straight away. On the next start of the background jobs, AI Analyst deletes the old model's vectors.

On an Air-gapped deployment, EMBEDDING_NEXT_BASE_URL is required whenever EMBEDDING_NEXT_MODEL is set.

Note

If you remove EMBEDDING_MODEL, AI Analyst keeps the stored vectors. Search matches keywords only until you set the model again.

If you leave inference unset¶

No LLM endpoint. If you leave LLM_PROVIDER unset, the deployment has no default LLM endpoint. Every question is blocked before it reaches a model. Members see:

Chat needs an LLM endpoint

No language model is connected for your organization yet. Ask your administrator to connect an LLM endpoint, then try again.

Chat with the Steward is blocked too, and shows Chat needs an LLM endpoint. Chat stays blocked until you set the LLM_* keys, or until the organization adds its own LLM endpoint.

No Embedding endpoint. If you leave EMBEDDING_MODEL unset, search matches keywords only. Under Settings → Inference, Search shows the Source None.

Let organizations use their own endpoints¶

Availability

Who can do this: members with the Manage inference capability for the organization.
Plan: On a self-managed deployment, included unless your Licence excludes customer-hosted inference.

An organization can add its own LLM endpoint and Embedding endpoint under Settings → Inference. An organization's endpoint replaces the deployment default for that organization only. AI Analyst tests an LLM endpoint before it saves it: the model must call a tool.

If the Licence excludes customer-hosted inference, administrators see Your plan doesn't include customer-hosted inference, so your organization uses the deployment's default. Contact Actian to add it.

Organization endpoints and private addresses

AI Analyst checks the address of an organization's endpoint before every call. It refuses redirects, and it applies the same private-address rules as for warehouses. On the Helm chart, an endpoint on a private address works only when its range is listed in jdbcBridge.networkPolicy.allowPrivateEgressCidrs. Otherwise the test fails with This deployment isn't allowed to call that address. See Warehouse connectivity.

The deployment default endpoints you set on this page aren't subject to these checks.

Reach endpoints on your network¶

Compose bundleHelm chart

The server sits on a Docker network with no route off the host. To reach your LLM and Embedding APIs, do one of the following:

Run the APIs as containers on the same host, attached to the stack's network, ai-analyst_internal.
Attach the stack to your network with the override file:
```
docker compose -f docker-compose.yml -f compose.customer-network.yml up -d --wait
```
From then on, the host firewall keeps the deployment Air-gapped. Allow traffic out of that network only to your own address ranges.

The chart doesn't ship a NetworkPolicy for the server pods. Your cluster's egress policy decides what the server can reach. Allow egress from the server pods to your LLM and Embedding APIs.

If an API uses a certificate from a private certificate authority, the server must trust it. See Private certificate authorities.

Slow self-hosted models¶

AGENT_STREAM_TIMEOUT_MS limits how long one agent response may take. The default is 240000 (4 minutes). A self-hosted model that needs minutes to answer and call tools can hit this limit, and the question then fails. Raise the limit, for example:

AGENT_STREAM_TIMEOUT_MS=900000

Air-gapped rules¶

On an Air-gapped deployment, including every Compose installation, the server refuses to start when:

LLM_PROVIDER is anthropic or openai.
ANTHROPIC_API_KEY or OPENAI_API_KEY is set.
EMBEDDING_MODEL is set without EMBEDDING_BASE_URL, or EMBEDDING_NEXT_MODEL without EMBEDDING_NEXT_BASE_URL.

See Connected and Air-gapped deployments.

Check your configuration¶

Start or restart the server and confirm it starts. If it refuses, the log names the setting. See How the server reports configuration problems.
Check embeddings with GET /api/health. The embeddings field reads ok when the Embedding endpoint answers, degraded when it doesn't, and unconfigured when no model is set.
Sign in as an administrator and open Settings → Inference. Chat should show the Source Provided by your administrator, and so should Search if you set an Embedding endpoint.
Ask a question in Explorer.

Troubleshooting¶

Symptom	Likely cause	Fix
The server refuses to start and names `LLM_CAPABILITY_PRESET`, `LLM_BASE_URL` or `LLM_MODEL`.	`LLM_PROVIDER` is set but a required key is missing or unknown.	Set all the required keys.
Members see Chat needs an LLM endpoint.	No deployment default and no organization endpoint.	Set the `LLM_*` keys and restart.
Questions fail with an error after a few minutes.	The model takes longer than `AGENT_STREAM_TIMEOUT_MS`.	Raise `AGENT_STREAM_TIMEOUT_MS`.
Chat fails although the keys are set.	The server can't reach the API, or doesn't trust its certificate.	Check network reach and private certificate authorities.
Search matches keywords only, and `/api/health` shows `degraded` for `embeddings`.	The Embedding endpoint doesn't answer.	Check `EMBEDDING_BASE_URL` (without `/v1`), the key and network reach. Search works with keywords meanwhile.
The server log reports a vector of the wrong length.	`EMBEDDING_DIMENSIONS` doesn't match the model, or the endpoint serves a different model.	Correct `EMBEDDING_DIMENSIONS` or the model.
An organization's endpoint test fails with This deployment isn't allowed to call that address.	The endpoint is on a private address that the deployment doesn't allow.	Add its range to `jdbcBridge.networkPolicy.allowPrivateEgressCidrs` (Helm) or `WAREHOUSE_EGRESS_ALLOW_PRIVATE_CIDRS` (Compose).