Envelope Format & Redis Streams

This doc captures the shared envelope structure used across allocator and OMS flows, and the Redis stream/key naming conventions.

Envelope Model

All cross‑box messages share the same envelope abstraction (Python Envelope model in runtime_common.envelope):

Envelope(
    kind: str,          # e.g. "alloc.submit", "oms.submit", "oms.flatten"
    tenant: str,        # usually "nts"
    ts: int,            # ms since epoch (server-side timestamp)
    nonce: str,         # URL-safe base64 random bytes (16 bytes)
    payload: dict,      # message-specific body
    idem_key: str | None,
    sig: str | None,    # HMAC-SHA256 over canonical payload
)

Signing

Messages are signed with sign(...) in runtime_common.envelope:

env = sign(
    kind="oms.submit",
    tenant="nts",
    payload=payload_dict,
    secret=ENVELOPE_SECRET,
    idem_key=idem_key,   # optional
    ts_ms=None,          # optional, auto now()
)

The signature is HMAC‑SHA256 over a canonical JSON representation of the envelope fields. Each worker verifies incoming messages with verify(env_dict, secret=ENVELOPE_SECRET).

On‑wire Container

On Redis streams, you always send a single field named "json" whose value is a JSON string:

{
  "json": "{ \"envelope\": { ... } }"
}

The outer object (container) may contain an envelope field; if not, the worker treats the container itself as the envelope.

Workers do roughly:

raw = fields.get("json")
container = json.loads(raw)
env_dict = container.get("envelope", container)
env = verify(env_dict, secret=ENVELOPE_SECRET)

Redis Streams & Keys

Allocator

• Stream: allocator:jobs
• Producer: allocator_api (POST /allocator/jobs)
• Consumer group: allocator

• Fields:

  • kind: "alloc.submit"
  • version: API version string
  • source: "allocator_api"
  • tenant: "nts"
  • idem_key: idem key
  • json: JSON string container with the signed envelope

• Status hash: allocator:job:{job_id}

• Written by allocator_worker.

• Typical fields:

  • status: "PENDING" | "RUNNING" | "DONE" | "ERROR"
  • tenant: "nts"
  • created_at, updated_at: ISO timestamps
  • artifact_s3_key: S3 key to report
  • error: optional error message

• Idempotency key: idem:ops:{idem_key}

• Set by allocator_api with EX=600 and NX for 10‑minute idempotency window.

OMS / IB

• Stream: oms:commands
• Producers:
  • Tools scripts (send_ib_submit_bracket.py)
  • Future: Blotter / trading desk UI.
• Consumer group: oms

• Envelope kind values:

  • "oms.ping"
  • "oms.submit"
  • "oms.flatten"
  • "oms.gateway_status"

• Stream: ib:events

• Producer: ibgatekeeper worker.
• No consumer groups by default; read with XRANGE/XREVRANGE or via ib_events_api.
• Field "json" is a JSON object representing a single event.

These conventions let you safely add new event types without changing the HTTP APIs or the stream handling boilerplate.