Scaling and operations¶

Availability

Deployment: Self-managed deployments only. Splitting the server into roles is available with the Helm chart; the Compose bundle always runs one all-in-one server.

How the server is built¶

The backend is one server image that can run different roles. Each role is a part of the backend that you can run and scale on its own:

Role	What it does	Scale it with
`api`	The HTTP API for Explorer and Studio, including the live stream of answers to the browser.	Concurrent users.
`orchestrator`	The agent loops that answer questions for AI Analyst and Steward.	Concurrent questions.
`semql`	Compiles semantic queries and runs them on your warehouses.	Query load.
`file-processing`	Converts uploaded CSV files to Parquet. Needs a lot of memory for large files.	Upload volume and file size.
`jobs`	Background and scheduled work.	Background load.
`chart-renderer`	Renders charts as images.	Chart rendering load.
`steward-edit`	Applies Steward's edits.	Steward edit load.

The special value all runs every role in one process. Both packagings start that way.

Work passes between roles through a job queue in Postgres. Adding replicas needs no sticky sessions, and live events reach whichever api pod holds a browser's connection.

Postgres carries the queue and live events¶

Postgres holds the product data, the job queue and the live event stream, so you run no Redis or message broker. Uploaded files, Parquet files and attachments live in object storage (the storage volume with Compose, your S3-compatible bucket with Helm), which every server replica must share.

Postgres is the main shared dependency to size. Size and tune it, including its connection limit, for your expected load. The Compose bundle starts Postgres with max_connections=300.

Sizing¶

Compose bundleHelm chart

Host: amd64, 4 CPU and 8 GB RAM recommended.
Memory limits: the server is capped at 3 GB (SERVER_MEMORY_LIMIT) and Keycloak at 1 GB (KEYCLOAK_MEMORY_LIMIT). Set both in .env.
Don't lower the server's limit below 2 GB. A 250 MiB CSV upload peaks near 1 GiB in the server while it converts to Parquet.

The Compose bundle runs on one host with no high availability. To scale beyond one host, use the Helm chart.

Chart defaults: each server Deployment requests 250m CPU and 1Gi memory, with a 1Gi memory limit.
Air-gapped example values: the all-in-one server requests 500m CPU and 2Gi memory, with a 3Gi memory limit, because file processing runs in the same pod.
When you split roles: give file-processing more memory than the other roles, or keep it with the other worker roles and size that Deployment for it.

This page gives no sizing by number of users. Start from these values, watch resource use, and adjust.

Split the server into roles (Helm)¶

In the Helm chart, server.roles is a list. Each entry becomes its own Deployment and ClusterIP Service, both named <fullname>-server-<name>, where <name> is the entry's name, not the role.

Each entry sets:

Field	Meaning
`name`	The entry's name, used in the Deployment and Service names.
`roles`	The role or roles this Deployment runs.
`replicas`	How many pods this Deployment runs.
`port`	The container port. The Service always listens on port 8080.
`resources`	Optional. Without it, the entry uses the shared `server.resources` block.

The default is a single all-in-one entry:

server:
  roles:
    - name: server
      roles: "all"
      replicas: 1
      port: 8080

Supported role combinations¶

Each Deployment must run a supported combination of roles, or its server refuses to start and lists the supported combinations:

Combination	`roles` value
Any single role	For example `api` or `orchestrator`.
All worker roles	`semql,steward-edit,chart-renderer,jobs,file-processing`
Worker roles without file processing	`semql,steward-edit,chart-renderer,jobs` (pair it with a separate `file-processing` entry)
Everything	`all`

For example, a split that scales the API and the agent loops separately and gives file processing its own memory:

server:
  roles:
    - name: api
      roles: "api"
      replicas: 2
      port: 8080
    - name: orchestrator
      roles: "orchestrator"
      replicas: 2
      port: 8080
    - name: workers
      roles: "semql,steward-edit,chart-renderer,jobs"
      replicas: 2
      port: 8080
    - name: file-processing
      roles: "file-processing"
      replicas: 1
      port: 8080
      resources:
        requests:
          cpu: 500m
          memory: 2Gi
        limits:
          memory: 3Gi

Make sure every role runs in one of your Deployments. Run helm upgrade with your edited values file to apply the change.

Route /api after a split¶

Your ingress sends /api to the Service of the Deployment that runs the api role. With the default all-in-one entry named server, that's <fullname>-server-server. If you give the api role its own entry named api, point /api at <fullname>-server-api instead.

The chart's own ingress (ingress.enabled) is off by default and built for Actian's hosted environment. If you use it anyway, set ingress.serverServiceName: server-api to match.

If a split Deployment refuses to start¶

The chart passes one Secret to every server Deployment. By default, a Deployment that doesn't run all treats settings that only other roles use as errors, and refuses to start with a message such as … (set but not declared by any loaded role). Set server.strictEnv: false in your values file when you split roles. With it off, the server logs the unused settings as a warning instead of refusing to start.

Keycloak replicas¶

With the chart-managed Keycloak (keycloak.enabled: true), run one replica. That is the tested configuration. You can raise keycloak.replicas, but more replicas are untested: they share the Keycloak database and cluster their caches with each other, so the pods must reach each other on TCP port 7800.

Health checks¶

Every server pod, whatever its roles, serves an unauthenticated health endpoint at /<API_PATH_PREFIX>/health. In the Compose bundle and the Helm Air-gapped example, that's /api/health.

The endpoint returns status: "ok", a timestamp and an embeddings status. The embeddings status is for information only and never fails the check, because search falls back to keyword matching without an embedding API.

What the health check doesn't cover

The health endpoint reports that the server process is up. It doesn't check Postgres, Keycloak or your inference APIs. Monitor those separately.

Compose bundleHelm chart

The long-running Compose services have health checks, and docker compose up -d --wait waits for them. Run docker compose ps to see each service's health.

The chart's startup, readiness and liveness probes use server.healthProbePath. The chart default is /health. When the API is served under /api (apiPathPrefix: api), set server.healthProbePath: /api/health, as the Air-gapped example does. The web pods' probes use /nginx-health.

Logs¶

The server writes structured JSON logs to standard output. Each line carries a service field that names the roles the process runs, so you can tell split Deployments apart in your log pipeline. It's server-all for the all-in-one server and server-server-api for a Deployment that runs only api. The default level is info.

The JSON output and the info default need ENVIRONMENT=production, which the Compose bundle and the Helm Air-gapped example Secret both set. With any other value, the server prints human-readable logs at debug level.

Compose bundleHelm chart

docker compose logs server
docker compose logs -f --since 10m server

When the server refuses to start, its log names the setting it objects to.

The bundle doesn't pass LOG_LEVEL from .env to the server, so you can't change the level from .env.

Use kubectl logs or your cluster's log pipeline. To change the level, set LOG_LEVEL in the server Secret.

Current limitations¶

Not available	What to do instead
Prometheus or other metrics endpoint	Use the health endpoint, logs and your platform's container metrics.
General rate limiting for the web app and API	Rate-limit at your proxy, ingress or WAF. AI Analyst does limit the Public API (60 requests per minute per API key and 300 per organization) and sign-in lookups per IP address.
Rotating the warehouse credential encryption key	Keep `WAREHOUSE_CREDENTIAL_ENCRYPTION_KEY` safe and backed up. See Upgrade and backup.
PodDisruptionBudgets and autoscaling in the Helm chart	Set replica counts per role yourself.
NetworkPolicy for the server pods in the Helm chart	Define egress policy for your cluster. See Connected and Air-gapped deployments.

Troubleshooting¶

A server pod refuses to start with "is not a known deploy composition". The entry's roles value isn't a supported combination. See Supported role combinations.

Helm pods keep failing their probes. Check that server.healthProbePath matches your API path prefix. With apiPathPrefix: api, use /api/health.

Large CSV uploads fail or the server restarts during uploads. The server or the file-processing Deployment is running out of memory. Raise its memory limit. With Compose, don't set SERVER_MEMORY_LIMIT below 2 GB.

After a split, the web app can't reach the API. Your ingress still points /api at the old Service. See Route /api after a split.