Skip to main content
Direct installation with helm is available using:
You must have access to the TextQL Docker registry. TextQL provides a PAT token that will grant you access. Before the first install, choose a size for the shared sandbox volume and plan to set its reclaim policy to Retain once it exists. See Sandbox Storage. When enabling global.egressProxy.enabled, your corporate proxy must refuse internal addresses, including host names that resolve to private addresses at that proxy.

Automatic updates

You can point Flux, ArgoCD or other GitOps tools to the TextQL registry for automatic releases. You can also set your own CI/CD pipeline that runs helm install. If you want these tools to auto install versions, we recommend doing so for patches only (e.g. ^1.1.X). We use Semantic Versioning (https://semver.org/) for our releases, so patches involve non-breaking changes and are safe to automatically install. For major and minor versions, check the Changelog page, which will provide the steps to upgrade, if needed. Contact TextQL for help with your specific setup.

One-time jobs during an upgrade

Some releases carry one-time jobs that change stored data. Most run in the compute-one-time-jobs-* Job while the new release rolls out. A job that must finish before the new release serves requests runs earlier, in the compute-one-time-jobs-before-serving Job, which the chart installs as a pre-upgrade hook on every upgrade. It first applies the new release’s database migrations, which are written to work with the previous release, then runs those jobs. Helm waits for it before it changes anything else, so the previous release keeps serving until it succeeds.
  • Allow enough time. Helm waits for the hook only as long as its upgrade timeout (helm upgrade --timeout, or spec.timeout on a Flux HelmRelease). A release note may ask for a longer timeout. When the timeout passes, the upgrade fails, but neither that nor a rollback stops the Job: it keeps running and changing data until it finishes, reaches its own 6-hour limit, or the next upgrade replaces it.
  • If it fails, Kubernetes retries the pod up to 3 times. After that the upgrade fails and the previous release keeps serving. Read the reason with kubectl logs job/compute-one-time-jobs-before-serving (the Job is kept for a day).
  • Secrets and ConfigMaps are read as the previous release left them. The hook runs before Helm applies the new release’s objects, so a Secret or ConfigMap key that this upgrade adds is unset in the hook, while settings given directly as chart values are already the new ones. A change that needs both, such as adding a connector encryption key and making it the primary key, takes two upgrades, as Rotate the connector encryption key describes. Done in one, the hook fails and the previous release keeps serving.
  • To retry, fix the cause and run the upgrade again (with Flux, flux reconcile helmrelease <name> --force, or let spec.upgrade.remediation.retries do it). Each upgrade replaces the hook Job. A job that already finished is recorded and does not run again, and the others start over. If Helm timed out while the hook was still running, the next upgrade stops it and starts it again.
A fresh install has no data to change, so it runs every one-time job in the regular compute-one-time-jobs-* Job, which starts with the jobs that must finish before serving. Compute waits for those before it serves. ArgoCD cannot tell an install from an upgrade, so it skips the compute-one-time-jobs-before-serving Job and the regular Job runs these jobs instead. New compute pods wait for them, so on an upgrade the previous release keeps serving through the rollout. A new pod that has waited 9 minutes exits and restarts, so pods in CrashLoopBackOff during a long job are expected. If a job fails, read the reason in the regular Job’s logs, fix the cause and sync again: each sync creates a new Job.

Rotate the connector encryption key

Stored credentials (database connector passwords, API connector keys, OAuth tokens, identity provider secrets and similar) are encrypted with the key named by connectorEncryptionPrimaryKey. Rotation takes two upgrades, so that no pod ever meets a credential under a key it has not loaded.
  1. Generate a new key:
  2. First upgrade: load the new key, keep the current one primary. Add the new key under a new ID and leave connectorEncryptionPrimaryKey unchanged:
    Where the key ring lives depends on global.secretsMode:
    • plaintext: set it in your values as above.
    • sealedSecrets: seal the whole ring as one value, then set connectorEncryptionKeys to the sealed output. connectorEncryptionPrimaryKey is a key ID, not a secret, and stays unsealed. Paste the ring (k1=<current base64 key>,k2=<new base64 key>) at the hidden prompt of read rather than into the command itself, so the keys stay out of your shell history. After the upgrade, wait for the controller to update the auth-secrets Secret before restarting:
    • externalSecrets or nativeExternalSecrets: update the <remoteKeyPrefix>-connector-encryption-keys secret in your external store, and wait for it to sync.
    • none: you manage the auth-secrets Kubernetes Secret yourself. Set its connector.encryption.keys entry to the new ring.
    Upgrade the chart, then restart the compute deployment. A change to the key ring alone does not restart its pods, and a pod only reads the ring when it starts:
    Every compute pod now holds both keys.
  3. Second upgrade: switch the primary. Set connectorEncryptionPrimaryKey: "k2" and upgrade. New credentials are encrypted with k2 at once, and in the background the compute service re-encrypts every stored credential still under k1. It logs one line per table, plus one for cached API connector drafts (table=api_revision_cache) and one for captured sandbox egress payloads (table=sandbox_egress):
  4. Run a final pass. Pods from before the second upgrade keep writing with k1 until the rollout finishes. Once it has finished, restart the compute deployment and check the lines from that start: every line, including api_revision_cache and sandbox_egress, must report rows_remaining=0.
    values_unmovable counts text that only looks like an encrypted value and values no loaded key can decrypt. They do not depend on k1 and do not block the next step. The sandbox_egress line appears only when its pass has finished. That table holds one row per outbound sandbox request, so on a busy deployment the pass takes hours, and a pod stops it after 5 hours or when the pod restarts. Each later start resumes the pass. While a pass is running, wait for the line. A pass that hit the 5 hour limit logs Retiring-key egress capture encryption backfill incomplete and does not start again by itself: restart the compute deployment when you see that message, or when the pod that was running the pass is gone.
  5. Remove the old key. Drop k1 from the ring and upgrade:
    Update the ring wherever your secrets mode keeps it, as in step 2, upgrade, and restart the compute deployment, waiting for the rollout to finish so every pod holds the reduced ring:
Do not remove the old key while any table reports rows_remaining above 0, or before the sandbox_egress line appears. Those values can only be decrypted with the old key and stop working the moment it leaves the ring. If a count stays above 0 after a restart, the log line rows still hold values only the retiring key can decrypt names the table; contact TextQL before removing the key. Rolling back after step 3 must keep k2 in the ring: credentials already moved to k2 cannot be decrypted without it.

Spanmetrics dashboards and alerting

The existing OpenTelemetry collector can generate call counts and duration histograms and send them to a dedicated metrics destination. Enable otel.spanMetrics.enabled and configure otel.spanMetrics.exporter. The chart wires the pipelines and authentication, so no collector configuration file or additional collector deployment is required. Spanmetrics export is independent of the existing OTLP export and Jaeger. Spans and native application metrics continue to use otel.exporter. Locally generated spanmetrics use only otel.spanMetrics.exporter, avoiding duplicate aggregation at the console gateway. Jaeger can also be enabled with otel.jaeger.enabled: true.

CloudWatch metrics and signed console export

Merge this example into your deployment’s existing values. Replace the Region, deployment ID and IAM role ARN with your own values:
The named Kubernetes Secret must contain deployment.private.key for the identity registered with the console. Signed console export does not support global.secretsMode: nativeExternalSecrets. If you only need the local metrics destination, omit otel.exporter and compute.deploymentId from this example. Associate the collector’s service account with a dedicated IAM role through IRSA using the annotation above, or EKS Pod Identity without that annotation. Grant the role cloudwatch:PutMetricData on *. AWS mode signs requests using SigV4 and the collector’s AWS credentials. The Region defaults to global.aws.region when otel.spanMetrics.exporter.aws.region is unset. See AWS ingestion permissions. Confirm native OTLP metrics ingestion and PromQL queries and alarms are available in your AWS Region. Allow DNS and HTTPS access to monitoring.<REGION>.amazonaws.com, plus the endpoints needed by your credential mechanism. Private clusters can use a CloudWatch interface VPC endpoint with private DNS and security groups allowing HTTPS from the collector. See AWS private connectivity. Signed console export also requires HTTPS access to console.textql.com.

Other OTLP metrics destinations

Use mode: otlp with any OTLP/HTTP base endpoint. The exporter appends /v1/metrics. For example:
Exporter headers are stored in a ConfigMap. Use Secret-backed environment references for credentials, as shown above.

Sampling, aggregation and rollouts

Use otel.traces.sampler: always_on for complete counts of emitted spans. Sampling before aggregation reduces observed counts and can affect error and latency measurements. Enabling all spans increases telemetry volume. Metrics count spans, so filter to a specific operation or span kind when measuring requests. Spanmetrics requires exactly one collector replica. The chart uses Recreate rollouts while it is enabled. Upgrades briefly interrupt collection, and in-memory export queues do not survive restarts. Aggregation defaults to a 30-second flush interval, a cardinality limit of 10,000, and a resource metrics cache of 1,000. Adjust these through otel.spanMetrics.flushInterval, otel.spanMetrics.aggregationCardinalityLimit, and otel.spanMetrics.resourceMetricsCacheSize. The metrics exporter also supports timeout, queue.numConsumers, queue.size and retry.maxElapsedTime, matching the gateway exporter settings.

Verify dashboards and alerts

After applying the values through Helm or GitOps, generate representative application traffic and check collector logs for export errors. In CloudWatch Query Studio, select PromQL and query:
The duration histogram is traces.span.metrics.duration, measured in milliseconds. Inspect emitted labels before choosing filters and include namespace or environment filters when sharing an AWS account across deployments. For example, this query shows P95 duration in milliseconds for inbound server spans in the compute service:
Save the queries as dashboard widgets. To create an alarm, add a comparison against your chosen threshold, set an evaluation interval and pending period, and connect your SNS notification topic. See CloudWatch PromQL queries and PromQL alarms. Validate a controlled error and notification delivery before relying on the alarm. Monitor collector availability separately so an export outage is visible when application metrics stop arriving.