#Queues for k0smos

This is the canonical operator guide for the k0smos queue subsystem.

Shipped answer: one tenant normally needs two worker processes: one for the default logical queue and one for notifications. k0smos has one queue subsystem and two alternative transports. Changing from database to Redis does not create another logical queue or require another worker.

Queue workers capture the tenant theme, default locale and active modules at startup. After a tenant theme or default-locale change, gracefully restart its workers so subsequent email and jobs use the new effective snapshot. In-flight jobs finish with the snapshot they started with.

Run the read-only diagnostic before changing a deployment:

php bin/console queue:topology --host=k0smos.example.com
php bin/console queue:topology --host=k0smos.example.com --format=json

--host selects the tenant to inspect. --format=json only changes standard output from the human-readable table to a machine-readable document; it does not select or contact a server. Use it from deployment scripts, CI, monitoring, or configuration management to read the expected process count, systemd unit names, heartbeat state, and diagnostics without parsing terminal tables.

The command reads the selected tenant configuration, active modules, DB-backed notification settings, producer metadata, and worker heartbeats. It does not expose passwords, tokens, payloads, or other secrets.

#Worker Count Rule

Run one process for every distinct effective logical queue that has an asynchronous producer:

worker processes per tenant = distinct logical queue names with async producers

Several job or message types sharing one queue still need only one process. Two slots consuming the same busy queue are two processes for capacity, not a third logical queue.

#Shipped topology

The shipped total is therefore two processes per tenant. If notification email is disabled in DB settings, its producer is inactive and the topology command does not require that lane.

#When another queue is worth its cost

Prefer default plus notifications until monitoring shows a concrete isolation need. A custom name without a producer is a configuration error, not useful capacity.

#Do Not Confuse These Four Layers

Never pass a job type such as queue.async_message, notification.email.send, or deadline.check_reminders to queue:work --queue. The option expects a logical queue name.

#Transport Decision

Redis failover keeps new work operable during an outage; it does not migrate jobs that already exist only in Redis into SQL. Both modes require queue migrations because Redis mode uses the SQL fallback.

Redis provisioning — tenant configuration keys, systemd unit ordering, key namespace, instance requirements, and the transport-mismatch failure mode — is in the systemd worker guide.

#Authoritative Producer Inventory

QueueTopologyRegistry is the runtime source for producer metadata used by queue:topology. The registry is filtered by active modules and mutable settings, so documentation is not the deployment source of truth.

Typed messages are serialized into the transport job type queue.async_message, then QueuedMessageJobHandler resolves the typed handler. Legacy/raw-compatible job types remain consumable:

• front.contact_form.send_mail
• notification.email.send
• deadline.generate_recurring_ticket
• deadline.check_reminders

Those compatibility types do not add logical queues or processes. Low-level callers that bypass the typed producer guards can still leave raw jobs pending when no worker consumes the selected logical queue; use queue:topology and heartbeat monitoring to prevent that configuration.

#Effective Queue-name Sources

Logical names remain configurable and must match [A-Za-z0-9._-]+. QueueName provides central validation, while QueueTopologyRegistry detects custom names with no consumers and divergences from the shipped AI/notification lanes.

#Topology Diagnostics

Text output answers the operator question directly. JSON is stable for deployment automation and contains:

• selected/primary/fallback transport;
• effective logical queues and active producers;
• expected process count;
• expected worker id and systemd instance for each queue;
• current active heartbeat identifiers without payloads/secrets;
• missing workers;
• duplicate worker ids used on different queues;
• global worker-id overrides that would make a multi-queue deployment ambiguous;
• custom worker-default names without discovered producers;
• custom AI/notification queues and invalid configuration values.

Example worker tuple:

tenant:          k0smos.example.com
logical queue:   default
slot:            1
worker id:       k0smos.example.com.default.1
systemd unit:    k0smos-queue-worker@k0smos.example.com-default-1.service

The old tenant.slot generated id is retired because two queues using slot 1 produced ambiguous logs and heartbeat identifiers.

#Start And Supervise Workers

Direct CLI use:

TENANT_ENV=k0smos.example.com php bin/console queue:work --queue=default --worker-slot=1
TENANT_ENV=k0smos.example.com php bin/console queue:work --queue=notifications --worker-slot=1

Copy-ready systemd units, environment files, a two-process setup, scaling with slot 2, deployment order, monitoring, and a per-tenant target are documented in the systemd worker guide.

SIGTERM/SIGINT stop new reservations and let the current handler return. Keep handlers idempotent: retries, stale reservation reclaim, and process termination provide at-least-once rather than exactly-once delivery.

#Failure And Dead-letter Behavior

SQL stale reclaim selects only the worker's logical queue before applying its batch limit. Each update also matches the observed reservation timestamp and attempt number: a job reclaimed and reserved by another worker after the scan is skipped, without changing its new lease or increasing the recovered count. A worker's timeout therefore cannot shorten another queue's reservation policy. Reclaim preserves payloads, attempt budgets and errors; normal handler failure still decides retries/dead letters. It does not cancel an old running handler or provide exactly-once side effects.

• WorkerRestartRequired stops immediately without ack/fail, notifications, heartbeat or another reservation. Payment raises this after an uncertain SQL boundary retires the shared handle. The current job remains reserved; restart in a fresh process and allow normal stale-reservation reclaim. A lost commit reply may already have persisted settlement, so recovery must be idempotent.
• Worker exceptions retry while attempts remain.
• Exhausted DBAL jobs stay as queue_jobs.status = 'failed'.
• Exhausted Redis jobs stay in the Redis failed structures.
• Definitive worker failures create in-app admin notifications.
• Producer-side synchronous fallbacks create throttled in-app admin notifications.
• Processed jobs remain log-only by default.
• Retry/delete dead-letter administration is a separate operational feature; queue:topology is intentionally read-only.

Owned native diagnostics now exercise this Payment retirement contract through real SQL queues/heartbeats, QueueWorker and the lazy webhook handler: 12 cases per MySQL/PostgreSQL engine, the same private event and reserved job, a genuinely aged lease, fresh physical connection and successful second attempt without duplicate settlement/stock. Private diagnostic queues avoid default content dispatch; an additional older reservation in a different queue genuinely ages alongside it and must remain entirely unchanged during reclaim. Shutdown/notifier observers are local controls. This does not qualify production default routing/fallback, CLI supervision/signals, actual notifier delivery, Redis or provider/HTTP behavior. Missing post-commit events remain best effort. See Ecommerce for the bounded workflow.

Private Customer reset and Notification email producers enter content-exclusion scopes before enqueueing/fallback; their workers resolve mail/render dependencies lazily inside those scopes. Delivery errors reach queues as fixed codes without original SMTP/template exception chains. Sensitive dispatcher failures are replaced before persisted fallback alerts; ordinary producers keep their diagnostics. An inline handler failure escapes after one attempt in that call, while normal transport retry policy remains unchanged. This does not guarantee exactly-once email after uncertain transport or later job retries.

Reset jobs contain customer/locale/scheme only and issue their raw link in the worker. Notification jobs freeze recipient, subject, template context, locale and optional text; rendering happens later. Acknowledgement removes successful jobs, while failed DBAL/Redis storage retains private payloads with no bounded dead-letter purge delivered here. Review queue access and retention separately. These source protections do not rewrite historical logs or qualify external mail/provider/proxy diagnostics.

#Glossary

Transport — storage/availability adapter behind the queue contracts.

Logical queue — named worker lane selected by queue:work --queue.

Job type — low-level payload category resolved by JobHandlerRegistry.

Typed message — serializable application message implementing QueueMessageInterface.

Handler — application service that performs the job/message side effect.

Heartbeat — last-seen record for a worker id on one logical queue.

Synchronous fallback — producer executes the same handler in its current process because no worker is active or async dispatch failed.

Retry — a failed reservation returned to pending work while attempts remain.

Dead letter — permanently failed job retained for diagnosis after retry exhaustion.

#Environment-dependent Smoke Checks

These require real Redis, real workers, or an isolated container/tenant, so they are not part of the automated suite. Run them in an isolated environment only — never against a runtime tenant database. If a run diverges from the behavior documented here, update README.md, doc/public/en/complete.md, this guide, and the affected changelog before closing the check.

• Redis queue round-trip integration (skippable when Redis is unavailable): dispatch, reserve, retry, delayed job, and dead-letter.
• Deadline command on an isolated container/tenant: with no active worker both handlers run inline and log queue.sync_fallback; with a worker active the work is dispatched asynchronously.
• queue.driver=database with an active worker: advanced AI chat and attachment upload work end to end.
• queue.driver=database with no worker: advanced AI chat and attachment upload answer with a controlled 503, never a raw 500.
• queue.driver=redis with Redis stopped mid-session: AI registries and the queue transport fail over to SQL with a logged warning and the chat continues.
• Deadline CLI on an isolated tenant with no worker: TENANT_ENV=k0smos.example.com php bin/console k0smos:deadline:schedule processes reminders/recurrences inline and exits 0.

queue.async_message is not a queue: it is the job type shared by every typed message, whatever queue carries it. Imports, payment webhooks, contact mail, content changes and Data Exchange run on default; notification email and the shop password reset email run on the email queue (notifications by default). A permanent failure notifies administrators in the inbox with the name of the work, the queue, the attempts and the reason, and links to the work's own page when it has one. queue:topology lists each producer with its queue and page.

#Rebuild The Source Inventory

The former static occurrence appendix was retired so it cannot drift into a second architecture document. Generate audit evidence on demand under reports/ instead of committing another operator guide:

mkdir -p reports
rg -n "MessageBusInterface|QueueMessageInterface|MessageDispatchOptions|QueueManagerInterface|QueueWorkerHeartbeatInterface|dispatchOrRun|queue:work|queue\\.driver|queue_jobs|queue_worker_heartbeats" \
  src modules config tests -g '*.php' -g '*.md' -g '*.json' -g '*.service' \
  > reports/queue-occurrences.txt

Stable architectural entry points:

• src/Domain/Queue/
• src/Application/Queue/QueueTopologyRegistry.php
• src/Application/Queue/QueueWorker.php
• src/Infrastructure/Queue/
• src/Command/QueueWorkCommand.php
• src/Command/QueueTopologyCommand.php
• src/Container/Definition/QueueDefinitions.php

#Committed content updates

Canonical Ecommerce/Blog/Page changes first write SQL intent in their source transaction. The content.changes topology entry is drained by the default worker, at most 100 deliveries per five-second poll, with independent consumer checkpoints. Worker absence leaves updates queued; it does not run search providers inside the source HTTP request. Apply maintenance and active-module migrations before enabling these producer changes.

TENANT_ENV=k0smos.example.com php bin/console content:changes
TENANT_ENV=k0smos.example.com php bin/console content:changes --dispatch --limit=100
TENANT_ENV=k0smos.example.com php bin/console content:changes --retry --subscriber=discovery.freshness --limit=100
TENANT_ENV=k0smos.example.com php bin/console content:changes --reconcile=blog:article --limit=100

Recovery advances one durable cursor per source (ecommerce:product, blog:article, page:page when active). Repeat the recovery command until its JSON reports complete: true, then drain delivery. Use it after missed events, source imports or consumer activation. Each pass replays current IDs, including unpublished records, then tombstones missing tracked IDs. It never changes source content. Five failures quarantine a delivery; explicit bounded retry preserves active worker leases. Discovery offers the same operations for its subscriber in the backoffice; canonical replay additionally checks the source owner's edit permission. The core contract and external indexing limitations are in src/Application/ContentChange/AI.ContentChange.md.

#Discovery maintenance

Handler registration follows active owners: Ai reply/attachment consumers require Ai, native payment webhook handling requires Ecommerce, and Discovery consumers require Discovery. A tenant without Ecommerce can resolve the shared default worker without catalog/payment checkout dependencies. The isolated 20-combination Ai/Discovery/source matrix checks handler, job registry and worker resolution.

Discovery's delivery tests also execute registered bounded CLI actions and a message through the default SQL queue's serialization/reserve/handle/ack cycle. That queue case verifies an unavailable Media source fails safely; successful binary extraction is covered separately. Psapi/Wpapi import fixtures verify committed outbox delivery creates pending Discovery work while queue dispatch is unavailable and preserves local editorial ownership. Those checks do not measure production queue delay or a complete import-to-vector round trip.

When Discovery is active, discovery.maintenance queues identity-only coalesced wakeups on default. A handler advances one bounded SQL-owned phase and may schedule a continuation, up to 256 steps. Phase ownership and child work use conditional leases; duplicate wakeups cannot execute the same claim. External embedding/Qdrant calls and Office/PDF parsing never run inside an owner SQL transaction. Document jobs remain on the existing discovery.documents producer; maintenance redispatches one due current job after lost delivery, without parsing inside the maintenance step.

Schedule discovery:work --advance --steps=100 --seconds=45 periodically in addition to the default worker. It recovers lost wakeups, future retries, expiry/retention and recurring immutable-point deletion. The time bound prevents starting another step; a step already in progress retains its own provider timeout. Pausing stops new projection work but retains cleanup, and retired generations remain in the recovery keyset. Exhausted projections require reconciliation; expired final document attempts remain available for explicit retry. These operations never automatically switch a serving generation or approve AI relations.

#Brainstorming worker requirements

Brainstorming uses a maximum 120-second provider budget and a renewable 30-second SQL run lease. Set queue:work --reclaim-after=180 or a longer value for the default lane; transport reservation expiry does not renew with the module lease. Identity-only duplicate deliveries cannot claim an already running or terminal attempt. No provider invocation is replayed automatically. A shared default worker runs addressed agents sequentially; adding a second default worker slot permits independent CLI types to run concurrently. Host locks serialize discussions for the same OS identity and CLI type across tenants; a conflicting attempt fails busy and can be retried explicitly.

Use the worker's OS identity, HOME and actual executable paths when checking native CLI authentication. Discussion execution requires Linux process groups and whole-service descendant cleanup after an uncatchable kill (systemd KillMode=control-group or an equivalent supervisor). Keep the host connector key and any selected official token in the worker environment as well as HTTP. After deployment, restart workers to load the new typed handler.

Polling or brainstorming:recover ends expired leases and queued attempts left unclaimed for 180 seconds in bounded batches. Planned workflow turns allow 1,140 seconds, reserving eight maximum 120-second provider budgets beyond that grace period for a single worker. Recovery never invokes a provider directly. The always-declared minimal handler can reject already queued work after the module is disabled without resolving the active module's application services. Private transcript reads and CLI execution run inside a scoped telemetry exclusion; queue failures contain fixed codes, never prompts or raw stderr.

Finite Brainstorming workflows use the same default lane and ID-only run jobs. A phase transition commits its turn and immutable context before dispatch; completion drives the next automatic step even with no connected browser/MCP client. Moderated phases wait for an explicit operator action. A failed or partial step blocks dependent work. brainstorming:recover also reconciles up to 100 workflow records and redispatches queued, never-claimed attempts; it never repeats an already claimed model call. Run recovery before the applicable stale threshold to recover lost dispatch; stale terminal attempts still need explicit retry.

Apply the new workflow/artifact migrations before deploying these services and restart workers. Result-plan retrieval and document-only writes are independent of worker availability. Plans are not queue payloads or expiring job results; they remain canonical tenant SQL documents after execution and transcript deletion.

On 2026-10-03 the selected Brainstorming development tenant's workflow/artifact migrations were confirmed read-only and its empty default lane was gracefully restarted under the existing Restart=always supervisor. A fresh process and active heartbeat were verified with reclaim-after 600 seconds and KillMode=control-group. This is tenant-specific readiness evidence, not a rollout to every tenant. Authenticated native workflow/MCP acceptance used disposable SQL and synthetic text; owner runtime UI/Hermes acceptance remains separate.