---
description: Size a self-managed Actian AI Analyst deployment, split the server into roles and scale them with the Helm chart, and monitor the deployment with health checks and logs.
---

# Scaling and operations

!!! info "Availability"
    - **Deployment:** Self-managed deployments only. Splitting the server into roles is available with the Helm chart; the Compose bundle always runs one all-in-one server.

## How the server is built

The backend is one server image that can run different *roles*. Each role is a part of the backend that you can run and scale on its own:

| Role | What it does | Scale it with |
|---|---|---|
| `api` | The HTTP API for Explorer and Studio, including the live stream of answers to the browser. | Concurrent users. |
| `orchestrator` | The agent loops that answer questions for AI Analyst and Steward. | Concurrent questions. |
| `semql` | Compiles semantic queries and runs them on your warehouses. | Query load. |
| `file-processing` | Converts uploaded CSV files to Parquet. Needs a lot of memory for large files. | Upload volume and file size. |
| `jobs` | Background and scheduled work. | Background load. |
| `chart-renderer` | Renders charts as images. | Chart rendering load. |
| `steward-edit` | Applies Steward's edits. | Steward edit load. |

The special value `all` runs every role in one process. Both packagings start that way.

Work passes between roles through a job queue in Postgres. Adding replicas needs no sticky sessions, and live events reach whichever `api` pod holds a browser's connection.

### Postgres carries the queue and live events

Postgres holds the product data, the job queue and the live event stream, so you run no Redis or message broker. Uploaded files, Parquet files and attachments live in object storage (the storage volume with Compose, your S3-compatible bucket with Helm), which every server replica must share.

Postgres is the main shared dependency to size. Size and tune it, including its connection limit, for your expected load. The Compose bundle starts Postgres with `max_connections=300`.

## Sizing

=== "Compose bundle"

    - **Host**: `amd64`, 4 CPU and 8 GB RAM recommended.
    - **Memory limits**: the server is capped at 3 GB (`SERVER_MEMORY_LIMIT`) and Keycloak at 1 GB (`KEYCLOAK_MEMORY_LIMIT`). Set both in `.env`.
    - **Don't lower the server's limit below 2 GB.** A 250 MiB CSV upload peaks near 1 GiB in the server while it converts to Parquet.

    The Compose bundle runs on one host with no high availability. To scale beyond one host, use the Helm chart.

=== "Helm chart"

    - **Chart defaults**: each server Deployment requests 250m CPU and 1Gi memory, with a 1Gi memory limit.
    - **Air-gapped example values**: the all-in-one server requests 500m CPU and 2Gi memory, with a 3Gi memory limit, because file processing runs in the same pod.
    - **When you split roles**: give `file-processing` more memory than the other roles, or keep it with the other worker roles and size that Deployment for it.

This page gives no sizing by number of users. Start from these values, watch resource use, and adjust.

## Split the server into roles (Helm)

In the Helm chart, `server.roles` is a list. Each entry becomes its own Deployment and ClusterIP Service, both named `<fullname>-server-<name>`, where `<name>` is the entry's `name`, not the role.

Each entry sets:

| Field | Meaning |
|---|---|
| `name` | The entry's name, used in the Deployment and Service names. |
| `roles` | The role or roles this Deployment runs. |
| `replicas` | How many pods this Deployment runs. |
| `port` | The container port. The Service always listens on port 8080. |
| `resources` | Optional. Without it, the entry uses the shared `server.resources` block. |

The default is a single all-in-one entry:

```yaml
server:
  roles:
    - name: server
      roles: "all"
      replicas: 1
      port: 8080
```

### Supported role combinations

Each Deployment must run a supported combination of roles, or its server refuses to start and lists the supported combinations:

| Combination | `roles` value |
|---|---|
| Any single role | For example `api` or `orchestrator`. |
| All worker roles | `semql,steward-edit,chart-renderer,jobs,file-processing` |
| Worker roles without file processing | `semql,steward-edit,chart-renderer,jobs` (pair it with a separate `file-processing` entry) |
| Everything | `all` |

For example, a split that scales the API and the agent loops separately and gives file processing its own memory:

```yaml
server:
  roles:
    - name: api
      roles: "api"
      replicas: 2
      port: 8080
    - name: orchestrator
      roles: "orchestrator"
      replicas: 2
      port: 8080
    - name: workers
      roles: "semql,steward-edit,chart-renderer,jobs"
      replicas: 2
      port: 8080
    - name: file-processing
      roles: "file-processing"
      replicas: 1
      port: 8080
      resources:
        requests:
          cpu: 500m
          memory: 2Gi
        limits:
          memory: 3Gi
```

Make sure every role runs in one of your Deployments. Run `helm upgrade` with your edited values file to apply the change.

### Route /api after a split

Your ingress sends `/api` to the Service of the Deployment that runs the `api` role. With the default all-in-one entry named `server`, that's `<fullname>-server-server`. If you give the `api` role its own entry named `api`, point `/api` at `<fullname>-server-api` instead.

The chart's own ingress (`ingress.enabled`) is off by default and built for Actian's hosted environment. If you use it anyway, set `ingress.serverServiceName: server-api` to match.

### If a split Deployment refuses to start

The chart passes one Secret to every server Deployment. By default, a Deployment that doesn't run `all` treats settings that only other roles use as errors, and refuses to start with a message such as `… (set but not declared by any loaded role)`. Set `server.strictEnv: false` in your values file when you split roles. With it off, the server logs the unused settings as a warning instead of refusing to start.

### Keycloak replicas

With the chart-managed Keycloak (`keycloak.enabled: true`), run one replica. That is the tested configuration. You can raise `keycloak.replicas`, but more replicas are untested: they share the Keycloak database and cluster their caches with each other, so the pods must reach each other on TCP port 7800.

## Health checks

Every server pod, whatever its roles, serves an unauthenticated health endpoint at `/<API_PATH_PREFIX>/health`. In the Compose bundle and the Helm Air-gapped example, that's `/api/health`.

The endpoint returns `status: "ok"`, a timestamp and an `embeddings` status. The embeddings status is for information only and never fails the check, because search falls back to keyword matching without an embedding API.

!!! note "What the health check doesn't cover"
    The health endpoint reports that the server process is up. It doesn't check Postgres, Keycloak or your inference APIs. Monitor those separately.

=== "Compose bundle"

    The long-running Compose services have health checks, and `docker compose up -d --wait` waits for them. Run `docker compose ps` to see each service's health.

=== "Helm chart"

    The chart's startup, readiness and liveness probes use `server.healthProbePath`. The chart default is `/health`. When the API is served under `/api` (`apiPathPrefix: api`), set `server.healthProbePath: /api/health`, as the Air-gapped example does. The web pods' probes use `/nginx-health`.

## Logs

The server writes structured JSON logs to standard output. Each line carries a `service` field that names the roles the process runs, so you can tell split Deployments apart in your log pipeline. It's `server-all` for the all-in-one server and `server-server-api` for a Deployment that runs only `api`. The default level is `info`.

The JSON output and the `info` default need `ENVIRONMENT=production`, which the Compose bundle and the Helm Air-gapped example Secret both set. With any other value, the server prints human-readable logs at `debug` level.

=== "Compose bundle"

    ```sh
    docker compose logs server
    docker compose logs -f --since 10m server
    ```

    When the server refuses to start, its log names the setting it objects to.

    The bundle doesn't pass `LOG_LEVEL` from `.env` to the server, so you can't change the level from `.env`.

=== "Helm chart"

    Use `kubectl logs` or your cluster's log pipeline. To change the level, set `LOG_LEVEL` in the server Secret.

## Current limitations

| Not available | What to do instead |
|---|---|
| Prometheus or other metrics endpoint | Use the health endpoint, logs and your platform's container metrics. |
| General rate limiting for the web app and API | Rate-limit at your proxy, ingress or WAF. AI Analyst does limit the Public API (60 requests per minute per API key and 300 per organization) and sign-in lookups per IP address. |
| Rotating the warehouse credential encryption key | Keep `WAREHOUSE_CREDENTIAL_ENCRYPTION_KEY` safe and backed up. See [Upgrade and backup](upgrade-and-backup.md). |
| PodDisruptionBudgets and autoscaling in the Helm chart | Set replica counts per role yourself. |
| NetworkPolicy for the server pods in the Helm chart | Define egress policy for your cluster. See [Connected and Air-gapped deployments](network-posture.md). |

## Troubleshooting

**A server pod refuses to start with "is not a known deploy composition".**
The entry's `roles` value isn't a supported combination. See [Supported role combinations](#supported-role-combinations).

**Helm pods keep failing their probes.**
Check that `server.healthProbePath` matches your API path prefix. With `apiPathPrefix: api`, use `/api/health`.

**Large CSV uploads fail or the server restarts during uploads.**
The server or the `file-processing` Deployment is running out of memory. Raise its memory limit. With Compose, don't set `SERVER_MEMORY_LIMIT` below 2 GB.

**After a split, the web app can't reach the API.**
Your ingress still points `/api` at the old Service. See [Route /api after a split](#route-api-after-a-split).