System Overview

This doc describes the current Palisades / Optimizer / IBKR integration as of Dec 2025.

At a high level you have four main planes:

1. Control plane (Palisades)
2. Hosts FastAPI services:
   • allocator_api (allocator jobs)
   • ib_events_api (read‑only access to ib:events)
3. Hosts Streamlit apps:
   • Household Allocator UI
   • Trading Desk (Workbench + Blotter)

4. Initiates commands into Redis streams (e.g. allocator:jobs, oms:commands).

5. Compute plane (Optimizer)

6. EC2 instance optimizer on a private subnet.

7. Runs allocator_worker as a systemd service:

   • Consumes jobs from allocator:jobs (Redis stream).
   • Runs simulations / household optimizer.
   • Writes artifacts to S3 (nts-household-artifacts).
   • Writes status into allocator:job:{job_id} hashes.

8. Execution plane (IBKR)

9. EC2 instance ibkr on a private subnet.

10. Runs:

    • IB Gateway / TWS in paper mode.
    • ibgatekeeper worker as a systemd service:
    • Consumes OMS commands from oms:commands (Redis stream).
    • Connects to TWS via the Python API client.
    • Emits synthetic and real order events into ib:events.

11. Data plane (Redis + S3)

12. Redis:
    • Central message bus and state store.
    • Protected with ACL users:
    • allocator_worker user (limited to allocator:* / idem:* etc.).
    • ibgatekeeper user (limited to oms:* and ib:*).
13. S3:
    • nts-household-artifacts bucket where allocator worker writes PPTX / reports.

Core Flows

1. Allocator Flow

1. Client (Streamlit / Notebook / external user) calls POST /allocator/jobs on Palisades.
2. allocator_api:
3. Validates payload (SubmitAllocReq).
4. Derives an idempotency key.
5. Enqueues a message into allocator:jobs with a JSON envelope.
6. allocator_worker (Optimizer box):
7. Reads from allocator:jobs using a consumer group.
8. Verifies the envelope signature.
9. Extracts job payload and parameters (n_sims, n_jobs, grid_step, payout_rate, etc.).
10. Runs the simulation and household optimizer.
11. Writes status to allocator:job:{job_id} hash.
12. Uploads a PPTX report to S3 and stores the S3 key in the hash.
13. Client polls:
14. GET /allocator/jobs (list)
15. GET /allocator/jobs/{job_id} (detail)
16. GET /allocator/jobs/{job_id}/report → presigned S3 URL for download.

2. OMS / IBKR Flow

1. Workbench / blotter or tools script signs an OMS envelope (kind="oms.submit") and pushes to oms:commands via xadd.
2. ibgatekeeper:
3. Consumes oms:commands with consumer group oms.
4. Verifies envelope signature.
5. For oms.submit:
   • Logs the number of orders.
   • If TWS is connected, logs a synthetic ib.order_received event.
   • Emits oms.submit and ib.order_received into ib:events.
6. (Under development): turns a subset of those orders into real IB bracket orders via the TWS API.
7. ib_events_api (Palisades):
8. Exposes /ib/events/tail endpoint to tail ib:events.
9. Supports filters like limit, msg_id, order_index.
10. Trading Desk Streamlit Blotter:
11. Calls ib_events_api to fetch a tail of events.
12. Builds tables:
    • IB events overview.
    • Order lifecycle per (msg_id, order_index).

This document is intentionally high‑level; see the per‑component docs for exact payload shapes and endpoints.