---
description: Point a self-managed deployment of Actian AI Analyst at your own OpenAI-compatible chat and embedding APIs, change embedding models safely, and fix common inference problems.
---

# Configure inference

!!! info "Availability"
    - **Deployment:** Self-managed deployments only. On Actian SaaS, Actian provides inference.

No model, model weights or model runtime ships with Actian AI Analyst. On a self-managed deployment, you connect AI Analyst to APIs that you run:

- **A chat API (LLM endpoint).** The AI Analyst and Steward use it to answer questions. Without one, chat doesn't work.
- **An embedding API (Embedding endpoint).** Optional. AI Analyst uses it to turn your semantic layer into vectors for search. Without one, search matches keywords only.

You configure these endpoints once for the whole deployment. This is the *deployment default*. Organizations on the deployment can also add their own endpoints in the app.

## Before you start

You need:

- **An OpenAI-compatible chat completions API that supports tool calling.** For example, vLLM, Text Generation Inference (TGI), Ollama, LiteLLM, or Azure OpenAI in your own tenant. The AI Analyst calls tools to do its work, so a model without tool calling can't answer questions.
- **Optionally, an OpenAI-compatible embeddings API.**
- **Network reach** from the AI Analyst server to both APIs. See [Reach endpoints on your network](#reach-endpoints-on-your-network).

## How AI Analyst chooses an endpoint

For each organization, AI Analyst uses the first endpoint that exists:

1. The organization's own endpoint, added under **Settings** → **Inference**.
2. The deployment default that you configure on this page.
3. None. Chat is blocked, and search matches keywords only.

Chat and search are chosen separately. For example, an organization can use its own LLM endpoint with the deployment's Embedding endpoint.

Administrators can see which endpoint their organization uses under **Settings** → **Inference**. The **Source** for each of **Chat** and **Search** is one of:

| Source | Meaning |
|---|---|
| **Provided by your administrator** | The deployment default you configure on this page. |
| **Your organization's own endpoint** | An endpoint the organization added. |
| **None** | No endpoint. |

<!-- screenshot: self-managed-inference-source -->

## Set the deployment default LLM endpoint

Set these keys in `.env` (Compose bundle) or in the server Secret (Helm chart):

| Key | Required | Description |
|---|---|---|
| `LLM_PROVIDER` | To use a deployment default | Set to `openai-compatible`. If you leave it unset, the deployment has no default LLM endpoint (see [If you leave inference unset](#if-you-leave-inference-unset)). |
| `LLM_BASE_URL` | Yes | The API's base URL, **including** `/v1`, for example `http://llm.example.internal:8000/v1`. AI Analyst appends `/chat/completions`. |
| `LLM_MODEL` | Yes | The model id that the API serves. |
| `LLM_CAPABILITY_PRESET` | Yes | `generic` or `reasoning`. See the table below. |
| `LLM_API_KEY` | No | Sent as a bearer token. Leave it unset if the API needs no key. |
| `LLM_FAST_MODEL` | No | A second model for quick calls, such as conversation titles. Defaults to `LLM_MODEL`. |

AI Analyst doesn't guess what a model supports from its name. You declare it with `LLM_CAPABILITY_PRESET`:

| Preset | Use it for | Context window AI Analyst assumes |
|---|---|---|
| `generic` | Models with tool calling and no reasoning options. | 128,000 tokens |
| `reasoning` | Models that accept a reasoning effort setting (`reasoning_effort`). | 128,000 tokens |

If `LLM_PROVIDER` is set and any required key is missing, or the preset is unknown, the server refuses to start and names the key.

=== "Compose bundle"

    In `.env`:

    ```sh
    LLM_PROVIDER=openai-compatible
    LLM_BASE_URL=http://llm.example.internal:8000/v1
    LLM_API_KEY=<API_KEY>
    LLM_MODEL=<MODEL_ID>
    LLM_CAPABILITY_PRESET=generic
    ```

    Then run `docker compose up -d --wait`.

=== "Helm chart"

    Add the keys to the server Secret, for example when you create it:

    ```sh
    kubectl -n ai-analyst create secret generic ai-analyst-secrets \
      --from-literal=LLM_PROVIDER=openai-compatible \
      --from-literal=LLM_BASE_URL=https://llm.example.internal/v1 \
      --from-literal=LLM_API_KEY=<API_KEY> \
      --from-literal=LLM_MODEL=<MODEL_ID> \
      --from-literal=LLM_CAPABILITY_PRESET=generic \
      ...
    ```

    If the Secret already exists, update it and restart the server pods. See [How you supply configuration](configuration-reference.md#how-you-supply-configuration).

!!! warning "Use only `openai-compatible`"
    `LLM_PROVIDER` also accepts `anthropic` and `openai`. These values select Actian Inference. They aren't supported on self-managed deployments. An Air-gapped deployment refuses to start with either value, and also refuses `ANTHROPIC_API_KEY` and `OPENAI_API_KEY`.

## Set the deployment default Embedding endpoint

Embeddings are optional. Without them, search in the semantic layer matches keywords only, and everything else keeps working.

AI Analyst knows two *Embedding presets*, models with calibrated search settings. You can also use any other model as a *custom* model.

| `EMBEDDING_MODEL` | Type | Dimensions | Notes |
|---|---|---|---|
| `text-embedding-3-small` | Preset | 1536 (fixed) | AI Analyst sends the model name `text-embedding-3-small`. |
| `qwen3-embedding-0.6b` | Preset | 1024 (fixed) | AI Analyst sends the model name `Qwen/Qwen3-Embedding-0.6B`. |
| Any other model id | Custom | Set `EMBEDDING_DIMENSIONS` | AI Analyst sends the id as you wrote it. Search uses a loose match threshold and scans all vectors, and the server logs a warning at startup that the model is uncalibrated. |

| Key | Required | Description |
|---|---|---|
| `EMBEDDING_MODEL` | To turn on embeddings | A preset id or your own model id. |
| `EMBEDDING_BASE_URL` | Yes on Air-gapped, and for `qwen3-embedding-0.6b` and custom models | The API's base URL, **without** `/v1`, for example `http://embeddings.example.internal:8080`. AI Analyst appends `/v1/embeddings`. |
| `EMBEDDING_API_KEY` | No | Sent as a bearer token. |
| `EMBEDDING_DIMENSIONS` | For a custom model | The length of the vectors the model returns, as a positive whole number. Don't set it for a preset. |
| `EMBEDDING_MAX_DISTANCE` | No, custom models only | The cosine distance above which a vector match is ignored, greater than 0 and at most 2. The default for custom models is `0.8`. |
| `EMBEDDING_QUERY_PREFIX` | No, custom models only | Text that AI Analyst puts in front of each search query before it embeds it, for models that expect an instruction. |

!!! warning "Always set `EMBEDDING_BASE_URL` to your own endpoint"
    On a Connected deployment, the `text-embedding-3-small` preset without `EMBEDDING_BASE_URL` calls a vendor-hosted API. If semantic-layer text must stay inside your network, always set the base URL to your own endpoint.

!!! note "Base URL paths differ"
    Include `/v1` in `LLM_BASE_URL`, but leave it out of `EMBEDDING_BASE_URL`. An `EMBEDDING_BASE_URL` that ends in `/v1` makes AI Analyst call `/v1/v1/embeddings`, which fails.

!!! warning "A wrong embedding setting stops the server"
    Leaving `EMBEDDING_MODEL` unset is fine. A setting that's present but wrong stops the server at startup. For example: a custom model without a valid `EMBEDDING_DIMENSIONS`, or a preset with `EMBEDDING_MAX_DISTANCE` or `EMBEDDING_QUERY_PREFIX`.

Example for a custom model in `.env` or the Secret:

```sh
EMBEDDING_MODEL=<MODEL_ID>
EMBEDDING_BASE_URL=http://embeddings.example.internal:8080
EMBEDDING_API_KEY=<API_KEY>
EMBEDDING_DIMENSIONS=1024
```

- **Vector length is checked.** Every vector the endpoint returns must have the configured number of dimensions. AI Analyst refuses any other length and logs an error, because such vectors can't be compared with the stored ones.
- **Text Embeddings Inference servers are checked too.** If your endpoint is a Hugging Face Text Embeddings Inference server, AI Analyst asks it which model it serves. The served model name must match the configured model, or AI Analyst refuses to embed. For a custom model, `EMBEDDING_MODEL` must equal the model id the server reports, for example `BAAI/bge-m3`. For a preset, the served revision must match too. The error in the server log names the model and revision it expects.
- **The Compose bundle passes only some embedding keys.** It passes `EMBEDDING_MODEL`, `EMBEDDING_BASE_URL`, `EMBEDDING_API_KEY` and `EMBEDDING_DIMENSIONS`. It doesn't pass `EMBEDDING_MAX_DISTANCE`, `EMBEDDING_QUERY_PREFIX` or the `EMBEDDING_NEXT_*` keys.
- **Leave the Helm chart's `embeddings` section off.** The chart contains an `embeddings` section that isn't for customer installs. Keep `embeddings.enabled: false` (the default) and set `EMBEDDING_*` in your Secret. If you turn the section on, it overrides your `EMBEDDING_*` settings.

When you set or change the embedding model, the server's background jobs embed your organizations' semantic layers. Until that finishes, search matches keywords for anything not yet embedded.

## Change the embedding model

You can switch models in one step or without a gap in semantic search.

**In one step.** Change the `EMBEDDING_*` keys and restart. On its next start, the server deletes the vectors of the old model and embeds everything again with the new one. Until that finishes, search matches keywords only for anything not yet re-embedded.

**Without a gap (Helm chart).** Use the `EMBEDDING_NEXT_*` keys. They take the same names as the `EMBEDDING_*` keys, with `NEXT_` added: `EMBEDDING_NEXT_MODEL`, `EMBEDDING_NEXT_BASE_URL`, `EMBEDDING_NEXT_API_KEY`, `EMBEDDING_NEXT_DIMENSIONS`, `EMBEDDING_NEXT_MAX_DISTANCE` and `EMBEDDING_NEXT_QUERY_PREFIX`.

1. Add the new model as `EMBEDDING_NEXT_*` in the Secret, and restart the server pods. AI Analyst now writes vectors for both models and fills in the new model's vectors in the background. Search keeps using the current model.
2. Leave the `EMBEDDING_NEXT_*` keys in place while the background jobs fill in the new vectors. AI Analyst doesn't report when this finishes, and search keeps using the current model meanwhile. If you move on before it finishes, search matches keywords only for anything not yet embedded.
3. Move the new values into the `EMBEDDING_*` keys, remove the `EMBEDDING_NEXT_*` keys, and restart the server pods. Search switches to the new model straight away. On the next start of the background jobs, AI Analyst deletes the old model's vectors.

On an Air-gapped deployment, `EMBEDDING_NEXT_BASE_URL` is required whenever `EMBEDDING_NEXT_MODEL` is set.

!!! note
    If you remove `EMBEDDING_MODEL`, AI Analyst keeps the stored vectors. Search matches keywords only until you set the model again.

## If you leave inference unset

**No LLM endpoint.** If you leave `LLM_PROVIDER` unset, the deployment has no default LLM endpoint. Every question is blocked before it reaches a model. Members see:

> **Chat needs an LLM endpoint**
>
> No language model is connected for your organization yet. Ask your administrator to connect an LLM endpoint, then try again.

Chat with the Steward is blocked too, and shows **Chat needs an LLM endpoint**. Chat stays blocked until you set the `LLM_*` keys, or until the organization adds its own LLM endpoint.

**No Embedding endpoint.** If you leave `EMBEDDING_MODEL` unset, search matches keywords only. Under **Settings** → **Inference**, **Search** shows the **Source** **None**.

## Let organizations use their own endpoints

!!! info "Availability"
    - **Who can do this:** members with the **Manage inference** capability for the organization.
    - **Plan:** On a self-managed deployment, included unless your Licence excludes customer-hosted inference.

An organization can add its own LLM endpoint and Embedding endpoint under **Settings** → **Inference**. An organization's endpoint replaces the deployment default for that organization only. AI Analyst tests an LLM endpoint before it saves it: the model must call a tool.

If the Licence excludes customer-hosted inference, administrators see *Your plan doesn't include customer-hosted inference, so your organization uses the deployment's default. Contact Actian to add it.*

!!! note "Organization endpoints and private addresses"
    AI Analyst checks the address of an organization's endpoint before every call. It refuses redirects, and it applies the same private-address rules as for warehouses. On the Helm chart, an endpoint on a private address works only when its range is listed in `jdbcBridge.networkPolicy.allowPrivateEgressCidrs`. Otherwise the test fails with *This deployment isn't allowed to call that address.* See [Warehouse connectivity](warehouse-connectivity.md#reach-warehouses-on-private-networks).

    The deployment default endpoints you set on this page aren't subject to these checks.

## Reach endpoints on your network

=== "Compose bundle"

    The server sits on a Docker network with no route off the host. To reach your LLM and Embedding APIs, do one of the following:

    - **Run the APIs as containers on the same host**, attached to the stack's network, `ai-analyst_internal`.
    - **Attach the stack to your network** with the override file:

        ```sh
        docker compose -f docker-compose.yml -f compose.customer-network.yml up -d --wait
        ```

        From then on, the host firewall keeps the deployment Air-gapped. Allow traffic out of that network only to your own address ranges.

=== "Helm chart"

    The chart doesn't ship a NetworkPolicy for the server pods. Your cluster's egress policy decides what the server can reach. Allow egress from the server pods to your LLM and Embedding APIs.

If an API uses a certificate from a private certificate authority, the server must trust it. See [Private certificate authorities](configuration-reference.md#private-certificate-authorities).

## Slow self-hosted models

`AGENT_STREAM_TIMEOUT_MS` limits how long one agent response may take. The default is `240000` (4 minutes). A self-hosted model that needs minutes to answer and call tools can hit this limit, and the question then fails. Raise the limit, for example:

```sh
AGENT_STREAM_TIMEOUT_MS=900000
```

## Air-gapped rules

On an Air-gapped deployment, including every Compose installation, the server refuses to start when:

- `LLM_PROVIDER` is `anthropic` or `openai`.
- `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` is set.
- `EMBEDDING_MODEL` is set without `EMBEDDING_BASE_URL`, or `EMBEDDING_NEXT_MODEL` without `EMBEDDING_NEXT_BASE_URL`.

See [Connected and Air-gapped deployments](network-posture.md).

## Check your configuration

1. Start or restart the server and confirm it starts. If it refuses, the log names the setting. See [How the server reports configuration problems](configuration-reference.md#how-the-server-reports-configuration-problems).
2. Check embeddings with `GET /api/health`. The `embeddings` field reads `ok` when the Embedding endpoint answers, `degraded` when it doesn't, and `unconfigured` when no model is set.
3. Sign in as an administrator and open **Settings** → **Inference**. **Chat** should show the **Source** **Provided by your administrator**, and so should **Search** if you set an Embedding endpoint.
4. Ask a question in Explorer.

## Troubleshooting

| Symptom | Likely cause | Fix |
|---|---|---|
| The server refuses to start and names `LLM_CAPABILITY_PRESET`, `LLM_BASE_URL` or `LLM_MODEL`. | `LLM_PROVIDER` is set but a required key is missing or unknown. | Set all the required keys. |
| Members see **Chat needs an LLM endpoint**. | No deployment default and no organization endpoint. | Set the `LLM_*` keys and restart. |
| Questions fail with an error after a few minutes. | The model takes longer than `AGENT_STREAM_TIMEOUT_MS`. | Raise `AGENT_STREAM_TIMEOUT_MS`. |
| Chat fails although the keys are set. | The server can't reach the API, or doesn't trust its certificate. | Check network reach and [private certificate authorities](configuration-reference.md#private-certificate-authorities). |
| Search matches keywords only, and `/api/health` shows `degraded` for `embeddings`. | The Embedding endpoint doesn't answer. | Check `EMBEDDING_BASE_URL` (without `/v1`), the key and network reach. Search works with keywords meanwhile. |
| The server log reports a vector of the wrong length. | `EMBEDDING_DIMENSIONS` doesn't match the model, or the endpoint serves a different model. | Correct `EMBEDDING_DIMENSIONS` or the model. |
| An organization's endpoint test fails with *This deployment isn't allowed to call that address.* | The endpoint is on a private address that the deployment doesn't allow. | Add its range to `jdbcBridge.networkPolicy.allowPrivateEgressCidrs` (Helm) or `WAREHOUSE_EGRESS_ALLOW_PRIVATE_CIDRS` (Compose). |