helm is available using:
Retain once it exists. See Sandbox Storage.
When enabling global.egressProxy.enabled, your corporate proxy must refuse internal addresses, including host names that resolve to private addresses at that proxy.
Automatic updates
You can point Flux, ArgoCD or other GitOps tools to the TextQL registry for automatic releases. You can also set your own CI/CD pipeline that runshelm install.
If you want these tools to auto install versions, we recommend doing so for patches only (e.g. ^1.1.X). We use Semantic Versioning (https://semver.org/) for our releases, so patches involve non-breaking changes and are safe to automatically install.
For major and minor versions, check the Changelog page, which will provide the steps to upgrade, if needed.
Contact TextQL for help with your specific setup.
One-time jobs during an upgrade
Some releases carry one-time jobs that change stored data. Most run in thecompute-one-time-jobs-* Job while the new release rolls out. A job that must finish before the new release serves requests runs earlier, in the compute-one-time-jobs-before-serving Job, which the chart installs as a pre-upgrade hook on every upgrade. It first applies the new release’s database migrations, which are written to work with the previous release, then runs those jobs. Helm waits for it before it changes anything else, so the previous release keeps serving until it succeeds.
- Allow enough time. Helm waits for the hook only as long as its upgrade timeout (
helm upgrade --timeout, orspec.timeouton a FluxHelmRelease). A release note may ask for a longer timeout. When the timeout passes, the upgrade fails, but neither that nor a rollback stops the Job: it keeps running and changing data until it finishes, reaches its own 6-hour limit, or the next upgrade replaces it. - If it fails, Kubernetes retries the pod up to 3 times. After that the upgrade fails and the previous release keeps serving. Read the reason with
kubectl logs job/compute-one-time-jobs-before-serving(the Job is kept for a day). - Secrets and ConfigMaps are read as the previous release left them. The hook runs before Helm applies the new release’s objects, so a Secret or ConfigMap key that this upgrade adds is unset in the hook, while settings given directly as chart values are already the new ones. A change that needs both, such as adding a connector encryption key and making it the primary key, takes two upgrades, as Rotate the connector encryption key describes. Done in one, the hook fails and the previous release keeps serving.
- To retry, fix the cause and run the upgrade again (with Flux,
flux reconcile helmrelease <name> --force, or letspec.upgrade.remediation.retriesdo it). Each upgrade replaces the hook Job. A job that already finished is recorded and does not run again, and the others start over. If Helm timed out while the hook was still running, the next upgrade stops it and starts it again.
compute-one-time-jobs-* Job, which starts with the jobs that must finish before serving. Compute waits for those before it serves.
ArgoCD cannot tell an install from an upgrade, so it skips the compute-one-time-jobs-before-serving Job and the regular Job runs these jobs instead. New compute pods wait for them, so on an upgrade the previous release keeps serving through the rollout. A new pod that has waited 9 minutes exits and restarts, so pods in CrashLoopBackOff during a long job are expected. If a job fails, read the reason in the regular Job’s logs, fix the cause and sync again: each sync creates a new Job.
Rotate the connector encryption key
Stored credentials (database connector passwords, API connector keys, OAuth tokens, identity provider secrets and similar) are encrypted with the key named byconnectorEncryptionPrimaryKey. Rotation takes two upgrades, so that no pod ever meets a credential under a key it has not loaded.
-
Generate a new key:
-
First upgrade: load the new key, keep the current one primary. Add the new key under a new ID and leave
connectorEncryptionPrimaryKeyunchanged:Where the key ring lives depends onglobal.secretsMode:-
plaintext: set it in your values as above. -
sealedSecrets: seal the whole ring as one value, then setconnectorEncryptionKeysto the sealed output.connectorEncryptionPrimaryKeyis a key ID, not a secret, and stays unsealed. Paste the ring (k1=<current base64 key>,k2=<new base64 key>) at the hidden prompt ofreadrather than into the command itself, so the keys stay out of your shell history. After the upgrade, wait for the controller to update theauth-secretsSecret before restarting: -
externalSecretsornativeExternalSecrets: update the<remoteKeyPrefix>-connector-encryption-keyssecret in your external store, and wait for it to sync. -
none: you manage theauth-secretsKubernetes Secret yourself. Set itsconnector.encryption.keysentry to the new ring.
Every compute pod now holds both keys. -
-
Second upgrade: switch the primary. Set
connectorEncryptionPrimaryKey: "k2"and upgrade. New credentials are encrypted withk2at once, and in the background the compute service re-encrypts every stored credential still underk1. It logs one line per table, plus one for cached API connector drafts (table=api_revision_cache) and one for captured sandbox egress payloads (table=sandbox_egress): -
Run a final pass. Pods from before the second upgrade keep writing with
k1until the rollout finishes. Once it has finished, restart the compute deployment and check the lines from that start: every line, includingapi_revision_cacheandsandbox_egress, must reportrows_remaining=0.values_unmovablecounts text that only looks like an encrypted value and values no loaded key can decrypt. They do not depend onk1and do not block the next step. Thesandbox_egressline appears only when its pass has finished. That table holds one row per outbound sandbox request, so on a busy deployment the pass takes hours, and a pod stops it after 5 hours or when the pod restarts. Each later start resumes the pass. While a pass is running, wait for the line. A pass that hit the 5 hour limit logsRetiring-key egress capture encryption backfill incompleteand does not start again by itself: restart the compute deployment when you see that message, or when the pod that was running the pass is gone. -
Remove the old key. Drop
k1from the ring and upgrade:Update the ring wherever your secrets mode keeps it, as in step 2, upgrade, and restart the compute deployment, waiting for the rollout to finish so every pod holds the reduced ring:
rows_remaining above 0, or before the sandbox_egress line appears. Those values can only be decrypted with the old key and stop working the moment it leaves the ring. If a count stays above 0 after a restart, the log line rows still hold values only the retiring key can decrypt names the table; contact TextQL before removing the key.
Rolling back after step 3 must keep k2 in the ring: credentials already moved to k2 cannot be decrypted without it.
Spanmetrics dashboards and alerting
The existing OpenTelemetry collector can generate call counts and duration histograms and send them to a dedicated metrics destination. Enableotel.spanMetrics.enabled and configure otel.spanMetrics.exporter. The chart
wires the pipelines and authentication, so no collector configuration file or
additional collector deployment is required.
Spanmetrics export is independent of the existing OTLP export and Jaeger.
Spans and native application metrics continue to use otel.exporter. Locally
generated spanmetrics use only otel.spanMetrics.exporter, avoiding duplicate
aggregation at the console gateway. Jaeger can also be enabled with
otel.jaeger.enabled: true.
CloudWatch metrics and signed console export
Merge this example into your deployment’s existing values. Replace the Region, deployment ID and IAM role ARN with your own values:deployment.private.key for the
identity registered with the console. Signed console export does not support
global.secretsMode: nativeExternalSecrets. If you only need the local metrics
destination, omit otel.exporter and compute.deploymentId from this example.
Associate the collector’s service account with a dedicated IAM role through
IRSA using the annotation above, or EKS Pod Identity without that annotation.
Grant the role cloudwatch:PutMetricData on *. AWS mode signs requests using
SigV4 and the collector’s AWS credentials. The Region defaults to
global.aws.region when otel.spanMetrics.exporter.aws.region is unset.
See AWS ingestion permissions.
Confirm native OTLP metrics ingestion and PromQL queries and alarms are available
in your AWS Region. Allow DNS and HTTPS access to
monitoring.<REGION>.amazonaws.com, plus the endpoints needed by your credential
mechanism. Private clusters can use a CloudWatch interface VPC endpoint with
private DNS and security groups allowing HTTPS from the collector. See AWS
private connectivity.
Signed console export also requires HTTPS access to console.textql.com.
Other OTLP metrics destinations
Usemode: otlp with any OTLP/HTTP base endpoint. The exporter appends
/v1/metrics. For example:
Sampling, aggregation and rollouts
Useotel.traces.sampler: always_on for complete counts of emitted spans.
Sampling before aggregation reduces observed counts and can affect error and
latency measurements. Enabling all spans increases telemetry volume. Metrics
count spans, so filter to a specific operation or span kind when measuring
requests.
Spanmetrics requires exactly one collector replica. The chart uses Recreate
rollouts while it is enabled. Upgrades briefly interrupt collection, and
in-memory export queues do not survive restarts.
Aggregation defaults to a 30-second flush interval, a cardinality limit of
10,000, and a resource metrics cache of 1,000. Adjust these through
otel.spanMetrics.flushInterval, otel.spanMetrics.aggregationCardinalityLimit,
and otel.spanMetrics.resourceMetricsCacheSize. The metrics exporter also
supports timeout, queue.numConsumers, queue.size and
retry.maxElapsedTime, matching the gateway exporter settings.
Verify dashboards and alerts
After applying the values through Helm or GitOps, generate representative application traffic and check collector logs for export errors. In CloudWatch Query Studio, select PromQL and query:traces.span.metrics.duration, measured in
milliseconds. Inspect emitted labels before choosing filters and include
namespace or environment filters when sharing an AWS account across deployments.
For example, this query shows P95 duration in milliseconds for inbound server
spans in the compute service: