- Date Created: 2025-12-19
- Last Modified: 2025-12-19
Progress Log: V2 Queue Symmetry Design & Analysis¶
Task Description¶
Analyzed the EventQueue and CommandQueue implementations to verify architectural symmetry in the unified device response protocol (v1.11.3+). The goal was to confirm that both input (commands) and output (events) paths follow mirrored designs for consistency and maintainability.
Scope:
- Compare EventQueue and CommandQueue structural design
- Verify 4-layer architecture (Layer 0 domain → Layer 1 validation → Layer 2 conversion → Layer 3 serialization)
- Identify symmetric vs. asymmetric patterns
- Document flow diagrams and design findings
Outcome¶
✅ Verification Complete: High Symmetry Achieved¶
Both queues implement a clean, mirrored architecture with excellent separation of concerns. The design successfully maintains symmetry at the type and abstraction level while allowing operational asymmetry where it makes sense.
Key Findings¶
1. Layer 0 (Domain Data) - FULLY SYMMETRIC¶
// Command Path
typedef struct {
char name[32]; // Command name
uint8_t arg_count;
char args[MAX_COMMAND_ARGS][MAX_ARG_LENGTH];
} command_t;
// Event Path
typedef struct {
uint16_t hit1, hit2, hit3; // Detection counts
int16_t adc; // ADC reading
// ... optional fields controlled by ENABLE_* flags
} event_t;
Symmetry:
- Both are lightweight, structured representations of domain data
- Both can carry optional fields via conditional compilation
- Both are queued "as-is" (Layer 0 in queue)
- FreeRTOS queue holds the raw type (command_t or event_t)
Design Benefit: Minimal copying, zero conversion overhead at queue level.
2. Layer 1 (Validation/Response) - FULLY SYMMETRIC¶
// Command Response (Layer 1)
typedef struct {
device_response_type_t type;
device_response_status_t status;
device_response_code_t error_code;
char error_message[256];
uint64_t sent_us;
const JsonDocument* payload; // Payload Pointer
} command_response_t;
// Event Response (Layer 1)
typedef struct {
device_response_type_t type;
device_response_status_t status;
device_response_code_t error_code;
char error_message[256];
uint64_t sent_us;
// ... flattened event fields (hit1, hit2, hit3, adc, etc.)
} event_response_t;
Symmetry:
- Both have identical envelope (
type,status,error_code,error_message,sent_us) - Both carry payload data (command responses via pointer, events via flattened fields)
- Both have mirror helper functions:
command_response_ok()↔event_response_ok()command_response_error()↔event_response_error()- Both encode errors with semantic codes (
DEVICE_CODE_INVALID_ARG,DEVICE_CODE_OUT_OF_RANGE, etc.)
Design Benefit: Consistent error handling and validation semantics across both paths.
3. FreeRTOS Queue - FULLY SYMMETRIC¶
| Property | CommandQueue | EventQueue | Symmetry |
|---|---|---|---|
| Static Allocation | xQueueCreateStatic() |
xQueueCreateStatic() |
✅ Identical |
| Queue Size | 10 items | 200 items | ✅ Sized for use case (commands sparse, events frequent) |
| Item Type | command_t |
event_t |
✅ Both Layer 0 types |
| Non-Blocking | xQueueSend(..., 0) |
xQueueSend(..., 0) |
✅ Identical |
| Overflow Handling | Drops oldest (DOS safety) | Increments counter | ✅ Both safe |
| Stream Control | Command::getInstance().get_stream() check |
Same | ✅ Identical |
Design Benefit: Deterministic behavior, no dynamic allocation, safe overflow handling.
4. Processing Flow - ASYMMETRIC (BY DESIGN)¶
CommandQueue (Input Path):
receive() → parse() → xQueueSend(command_t) → execute() → dispatch() → handler → send()
Ownership: CommandQueue manages lifecycle (receive, queue, execute)
Responsibility: Parse, validate args, route to handler
EventQueue (Output Path):
enqueue(event_t from caller) → xQueueSend(event_t) → flush() → serialize → send()
Ownership: Caller manages detection & sensors, EventQueue manages queueing & serialization
Responsibility: Buffer high-frequency events, serialize to JSON
Analysis:
This asymmetry is intentional and correct:
-
CommandQueue::receive() must be internal because:
- Serial reception happens in main loop
- Parsing is domain-specific (requires command table)
- Error recovery (buffer overflow, malformed input) must be atomic
-
EventQueue::enqueue() is external because:
- Detection happens in detection loop (different context)
- Caller has raw sensor data (temperature, ADC, hits)
- Separation: detection logic ≠ queueing logic
Why This is Good:
- CommandQueue encapsulates serial I/O (implementation detail)
- EventQueue is a pure queue service (open to any caller)
- Both avoid mixing domain logic with infrastructure
5. Layer 2 (Type Conversion) - FULLY SYMMETRIC¶
// Command path
device_response_t DeviceResponse::from_command(const command_response_t& cmd);
// Event path
device_response_t DeviceResponse::from_event(const event_response_t& event);
Symmetry:
- Both convert Layer 1 → Layer 2 (typed struct → transport envelope)
- Both create
device_response_twith identical envelope fields - Both copy payload (command via pointer, event via JsonDocument pointer set in from_event)
- Both deterministic, side-effect-free
Design Benefit: Clean, predictable transformation layer.
6. Layer 3 (Serialization) - FULLY SYMMETRIC¶
// Single unified serialization for both paths
DeviceResponse::send(const device_response_t& response); // Outputs JSONL
Symmetry:
- Both command and event responses use identical send() function
- No type-specific branching (type is a field in device_response_t, not behavior discriminator)
- Handles all optional fields via conditional compilation
Design Benefit: Unified output format, single serialization path.
Flow Diagram: Complete Symmetry¶
┌─── COMMAND INPUT PATH ────────────────────────┐ ┌─── EVENT OUTPUT PATH ──────────────────────┐
│ │ │ │
│ Layer 0 (Domain) │ │ Layer 0 (Domain) │
│ ┌────────────────────────┐ │ │ ┌──────────────────────────┐ │
│ │ Serial Input Stream │ │ │ │ Detection Loop │ │
│ │ │ │ │ │ (cosmic_read()) │ │
│ │ "SET_POLL_COUNT 200\n"│ │ │ │ → hit1=95, hit2=87,... │ │
│ └───────────┬────────────┘ │ │ └────────────┬─────────────┘ │
│ │ │ │ │ │
│ ┌───────────v────────────┐ │ │ ┌────────────v──────────────┐ │
│ │ CommandQueue::receive()│ │ │ │ EventQueue::enqueue() │ │
│ │ - Parse line │ │ │ │ - Check stream enabled │ │
│ │ - Split by whitespace │ │ │ │ - Queue event_t │ │
│ │ - Resolve aliases │ │ │ │ - Track overflows │ │
│ └───────────┬────────────┘ │ │ └────────────┬──────────────┘ │
│ │ │ │ │ │
│ Layer 0→Layer 1 │ │ Layer 0→Layer 1 │
│ ┌───────────v────────────┐ │ │ ┌────────────v──────────────┐ │
│ │ command_t │ │ │ │ event_t (Layer 0) │ │
│ │ ├─ name="SET_PC" │ │ │ │ ├─ hit1=95 │ │
│ │ ├─ arg_count=1 │ │ │ │ ├─ hit2=87 │ │
│ │ └─ args[0]="200" │ │ │ │ ├─ hit3=92 │ │
│ │ │ │ │ │ ├─ adc=2048 │ │
│ │ [QUEUED in FreeRTOS] │ │ │ │ └─ ... (sensor data) │ │
│ └───────────┬────────────┘ │ │ │ │ │
│ │ │ │ │ [QUEUED in FreeRTOS] │ │
│ ┌───────────v────────────┐ │ │ └────────────┬──────────────┘ │
│ │ CommandQueue::execute()│ │ │ │ │
│ │ - Dequeue command_t │ │ │ ┌────────────v──────────────┐ │
│ │ - Call dispatch() │ │ │ │ EventQueue::flush() │ │
│ │ - Find handler │ │ │ │ - Check stream enabled │ │
│ │ - Call handler(cmd) │ │ │ │ - Dequeue all events │ │
│ └───────────┬────────────┘ │ │ │ - For each event: │ │
│ │ │ │ │ - Create response │ │
│ Layer 1: Handler generates response │ │ │ - Convert to DR │ │
│ ┌───────────v────────────┐ │ │ └────────────┬──────────────┘ │
│ │command_response_t │ │ │ │ │
│ │ ├─ type=RESPONSE │ │ │ Layer 1: Validation │
│ │ ├─ status=OK │ │ │ ┌────────────v──────────────┐ │
│ │ ├─ payload→{...} │ │ │ │event_response_t │ │
│ │ └─ sent_us=123456 │ │ │ │ ├─ type=EVENT │ │
│ │ │ │ │ │ ├─ status=OK │ │
│ │ [from handler] │ │ │ │ ├─ hit1, hit2, hit3, adc │ │
│ └───────────┬────────────┘ │ │ │ └─ sent_us=123456 │ │
│ │ │ │ └────────────┬──────────────┘ │
│ Layer 1→Layer 2 │ │ │ │
│ ┌───────────v────────────────────────────┐ │ │ Layer 1→Layer 2 │
│ │ DeviceResponse::from_command() │ │ │ ┌────────────v──────────────┐ │
│ │ - Copy type, status, error fields │ │ │ │ DeviceResponse::from_event() │
│ │ - Copy payload pointer │ │ │ │ - Copy envelope fields │ │
│ │ - Return device_response_t │ │ │ │ - Build JsonDocument │ │
│ └───────────┬────────────────────────────┘ │ │ │ - Flatten event fields │ │
│ │ │ │ │ - Set payload pointer │ │
│ ┌───────────v────────────┐ │ │ └────────────┬──────────────┘ │
│ │ device_response_t │ │ │ │ │
│ │ ├─ type=RESPONSE │ │ │ ┌────────────v──────────────┐ │
│ │ ├─ status=OK │ │ │ │ device_response_t │ │
│ │ ├─ sent_us=123456 │ │ │ │ ├─ type=EVENT │ │
│ │ └─ payload→{...} │ │ │ │ ├─ status=OK │ │
│ │ │ │ │ │ ├─ payload→{ hit1,hit2..}│ │
│ │ [Layer 2: Transport] │ │ │ │ └─ sent_us=123456 │ │
│ └───────────┬────────────┘ │ │ │ │ │
│ │ │ │ │ [Layer 2: Transport] │ │
│ │ │ │ └────────────┬──────────────┘ │
│ │ │ │ │ │
│ │ Layer 3: Unified Serialization│ │ │ │
│ │ ┌─────────────────────────────┴──┴───────────────┴────────┐ │
│ │ │ DeviceResponse::send(device_response_t) │ │
│ │ │ - Serialize to JSON Lines format │ │
│ │ │ - Merge payload fields with envelope │ │
│ │ │ - Output to Serial (115200 baud) │ │
│ │ └────────────────────┬────────────────────────────────────┘ │
│ │ │ │
│ └───────────────────────┼────────────────────────────────────┘
│ │
│ ┌──────────────────────────────────────────────────────────────┐
│ │ Serial Output (JSONL, one per line) │
│ │ │
│ │ {"type":"response","status":"ok","sent_us":123456, ...} │
│ │ {"type":"event","status":"ok","sent_us":123457, ...} │
│ │ {"type":"event","status":"ok","sent_us":123458, ...} │
│ │ │
│ └──────────────────────────────────────────────────────────────┘
Learnings¶
1. Architectural Consistency Achieved¶
The v2 protocol successfully implements mirrored architecture for input and output:
- Layer 0 (Domain): Lightweight, structured representation of domain concepts
- Layer 1 (Validation): Typed response with error handling semantics
- Layer 2 (Conversion): Deterministic translation to transport format
- Layer 3 (Serialization): Unified output handler
This 4-layer pattern enables clean separation and testability.
2. Smart Operational Asymmetry¶
While structurally symmetric, the operational flow differs by context:
- CommandQueue: Input from serial → parsing → queueing → execution → output
- EventQueue: Input from sensors → queueing → serialization → output
This asymmetry reflects different operational constraints:
- Serial input requires atomic receive/parse for error recovery
- Sensor output decouples detection from transmission for performance
The key insight: Structural symmetry + Operational asymmetry = Flexibility + Clarity
3. Phase 6 Payload Pointer Pattern Benefits¶
Both command and event responses use the same payload approach:
- Command:
const JsonDocument* payloadincommand_response_t - Event: Fields are flattened but serialized identically
This unifies the serialization layer completely.
4. Stream Control Unification¶
Both paths respect the same flag:
bool stream_enabled = Command::getInstance().get_stream();
This enables runtime control of both command responses and event output via single SET_STREAM command.
5. Memory Efficiency¶
- CommandQueue: 10 items × ~65 bytes = ~650 bytes
- EventQueue: 200 items × ~160 bytes = ~32 KB
- Total: ~32.65 KB (9.9% of ESP32's 320KB RAM)
- All static allocation (no heap fragmentation)
Design Summary¶
✅ Strengths¶
- High Structural Symmetry: Type-level mirroring ensures consistency
- Clear Layering: 4-layer architecture with distinct responsibilities
- Flexible Payload Handling: Phase 6 pointer pattern supports any response structure
- Unified Serialization: Single
send()function for both types - Deterministic: No dynamic allocation, predictable behavior
- Smart Asymmetry: Operational differences justified by context
🔍 Considerations¶
- Event Flattening vs. Pointer:
- Commands use pointer (flexible, no type restrictions)
- Events flatten fields (optimized for space in struct)
- Both serialize identically → No user-facing difference
-
Trade-off: Event struct larger but creation simpler
-
enqueue() External API:
- Caller must populate all event fields before queueing
- Enables detection loop control but requires discipline
-
Could add event builder helper if complexity grows
-
Queue Size Asymmetry:
- Commands: 10 items (fast processing)
- Events: 200 items (high-frequency bursts)
- Sizes empirically justified by workload
Next Steps¶
- Documentation: Add flow diagrams to REFACTORING_ROADMAP.md or create design guide
- Code Review: Update code comments to reference this symmetry analysis
- Testing: Verify queue behavior under load (burst events, command backlog)
- Event Builder: Consider adding helper function to construct event_t with sensor reads
- Asymmetry Documentation: Add comments explaining why receive/execute are internal vs. enqueue external