Date Created: 2025-12-09
Last Modified: 2025-12-09
Status: IMPLEMENTATION IN PROGRESS

Progress Log: Unified Device Response Protocol Phase 3 Design¶

Task Description¶

Design the architecture and implementation strategy for Phase 3: Event Output Unification of the Unified Device Response Protocol (v1.13.1 target).

Context:

Phase 1 (v1.12.0): device_response_t structure and builder functions ✅ completed
Phase 2 (v1.13.0): Command class singleton for unified configuration ✅ completed
Phase 3 (v1.13.1): Integrate event output with device_response_t schema ⏳ design phase

Goal: Unify event serialization (detection output) with the device_response_t protocol while maintaining backward compatibility with legacy STREAM_FORMAT output and ensuring Command class integration.

Outcome¶

Current Architecture Analysis¶

Three-Layer Current Output System:

Data Layer: event_t (stream_data.h)
Core fields: hit1, hit2, hit3, adc (sensorValue)
Optional fields controlled by compile-time flags: ENABLE_HITTYPE, ENABLE_ADCMV, ENABLE_BME280, ENABLE_TIMESTAMP, ENABLE_RTC, ENABLE_GNSS
Format-agnostic: same structure populates all output formats
Format Layer: send_event() (stream_formatter.cpp)
Four compile-time selectable formats via STREAM_FORMAT flag (0-3)
SSV (0): Space-separated values (production default)
TSV (1): Tab-separated values (spreadsheet pipeline)
CSV (2): Comma-separated values with implicit header
JSONL (3): JSON Lines with type:"data", status:"ok" fields (non-conformant to device_response_t schema)
All formats respect optional field flags and output only enabled fields
Protocol Layer: device_response_t (device_response.h, v1.11.3+)
Unified schema for commands and responses
Structure: {type, status, sent_at, error_code, error_message, <payload>}
Only compiled when ENABLE_DEVICE_RESPONSE=1
Currently only used for command responses, NOT detection events

Integration Gap:

Event output (send_event) does NOT use device_response_t schema
JSONL format uses ad-hoc type:"data" and status:"ok" fields
Command class (Command singleton) has no awareness of event output
stream_enabled flag isolated in runtime_config (C-based)

Strategy 2 Design: Module Separation with Backward Compatibility¶

Core Principle: Implement unified event output in a new event_response module (ENABLE_DEVICE_RESPONSE-guarded), while preserving existing stream_formatter.cpp unchanged (legacy path). This follows the single-responsibility and CLAUDE.md module organization patterns.

Module Organization¶

include/
  ├── stream_data.h              (existing: event_t structure)
  ├── stream_formatter.h         (existing: legacy output functions)
  ├── device_response.h          (existing: device_response_t + builders)
  ├── event_response.h           (NEW: unified event serialization, ENABLE_DEVICE_RESPONSE=1 only)
  └── command.h                  (existing: Command singleton)

src/
  ├── stream_formatter.cpp       (existing: SSV/TSV/CSV/JSONL legacy implementation)
  ├── device_response.cpp        (existing: device_response_t builder functions)
  ├── event_response.cpp         (NEW: event_response_t serialization)
  ├── stream_manager.cpp         (existing: send_event() dispatcher)
  └── command.cpp                (existing: Configuration class)

Key Design Decision: send_event() dispatcher (in stream_formatter.cpp) acts as a single entry point that branches: - Path A (ENABLE_DEVICE_RESPONSE=1): Always JSON Lines format (STREAM_FORMAT ignored) - Path B (ENABLE_DEVICE_RESPONSE=0): Respects STREAM_FORMAT flag (SSV/TSV/CSV/JSONL)

void send_event(const event_t *data) {
  if (!get_stream_enabled()) return;  // Check stream flag (Command or runtime_config)

#if ENABLE_DEVICE_RESPONSE
  // Path A: Unified schema - JSON Lines ONLY (STREAM_FORMAT completely ignored)
  send_event_as_device_response(data);
#else
  // Path B: Legacy format - respects STREAM_FORMAT flag
  #if STREAM_FORMAT == 0
    send_ssv(data);
  #elif STREAM_FORMAT == 1
    send_tsv(data);
  #elif STREAM_FORMAT == 2
    send_csv(data);
  #elif STREAM_FORMAT == 3
    send_jsonl(data);
  #else
    send_ssv(data);  // Default fallback
  #endif
#endif
}

File: `include/event_response.h`¶

#ifndef EVENT_RESPONSE_H
#define EVENT_RESPONSE_H

#include "stream_data.h"
#include "device_response.h"

#if ENABLE_DEVICE_RESPONSE

/**
 * @brief Send detection event as unified device_response_t schema
 *
 * Serializes event_t to JSON Lines format conforming to device-response.json schema:
 * {
 *   "type": "event",
 *   "status": "ok",
 *   "sent_at": <unix_time or uptime>,
 *   "hit1": val, "hit2": val, "hit3": val, "adc": val,
 *   [optional fields based on ENABLE flags]
 * }
 *
 * @param data Pointer to event_t structure with populated detection readings
 * @note Only compiled when ENABLE_DEVICE_RESPONSE=1
 */
void send_event_as_device_response(const event_t *data);

/**
 * @brief Convert event_t to device_response_t structure
 *
 * Creates a device_response_t with type=DEVICE_TYPE_EVENT and status=DEVICE_STATUS_OK,
 * ready for serialization by send_device_response() or custom handlers.
 *
 * @param data Pointer to event_t structure
 * @return Initialized device_response_t (payload pending JSON serialization)
 * @note Only compiled when ENABLE_DEVICE_RESPONSE=1
 */
device_response_t event_to_device_response(const event_t *data);

#endif // ENABLE_DEVICE_RESPONSE
#endif // EVENT_RESPONSE_H

File: `src/event_response.cpp`¶

Design Constraint: Event output is JSON Lines format ONLY - no STREAM_FORMAT variants. This simplifies the unified protocol to a single, deterministic format.

#include "event_response.h"
#include <ArduinoJson.h>
#include "config.h"

#if ENABLE_DEVICE_RESPONSE

void send_event_as_device_response(const event_t *data) {
  // JsonDocument with dynamic memory management
  // IMPORTANT: JSON Lines format ONLY - STREAM_FORMAT flag is ignored
  JsonDocument doc;

  // Unified schema common fields
  doc["type"] = "event";
  doc["status"] = "ok";
  doc["sent_at"] = device_get_timestamp();

  // Core event data (always present)
  doc["hit1"] = data->hit1;
  doc["hit2"] = data->hit2;
  doc["hit3"] = data->hit3;
  doc["adc"] = data->sensorValue;

  // Optional fields (compile-time conditional)
#if ENABLE_HITTYPE
  doc["hit_type"] = data->hit_type;
#endif

#if ENABLE_ADCMV
  doc["adc_raw"] = data->adc_raw;
  doc["adc_mv"] = data->adc_mv;
#endif

#if ENABLE_BME280
  doc["temp_c"] = data->tmp_c;
  doc["atm_pa"] = data->atm_pa;
  doc["hmd_pct"] = data->hmd_pct;
#endif

#if ENABLE_TIMESTAMP
  doc["uptime_ms"] = data->uptime_ms;
  doc["timedelta_us"] = data->timedelta_us;
#endif

#if ENABLE_RTC
  doc["unix_timestamp"] = data->unix_timestamp;
#endif

#if ENABLE_GNSS
  if (data->gnss_fix_valid) {
    doc["gnss_latitude"] = serialized(String(data->gnss_latitude, 6));
    doc["gnss_longitude"] = serialized(String(data->gnss_longitude, 6));
    doc["gnss_altitude"] = data->gnss_altitude;
    doc["gnss_satellites"] = data->gnss_satellites;
    doc["gnss_fix_quality"] = data->gnss_fix_quality;
    doc["gnss_hdop"] = data->gnss_hdop;
    doc["gnss_fix_valid"] = true;
  }
#endif

  // Serialize to JSON and output as single line
  serializeJson(doc, Serial);
  Serial.println();
}

device_response_t event_to_device_response(const event_t *data) {
  // Create response structure with event type
  device_response_t response = device_response_ok(DEVICE_TYPE_EVENT);

  // Caller must populate payload fields via JSON serialization
  // (this function only creates the envelope)
  return response;
}

#endif // ENABLE_DEVICE_RESPONSE

Implementation Approach: stream_formatter.cpp Left Unchanged¶

Design Decision: stream_formatter.cpp is NOT modified to avoid mixing legacy and unified paths. Instead:

stream_formatter.cpp remains completely unchanged (legacy path only)
All format functions (send_ssv, send_tsv, send_csv, send_jsonl) remain unchanged
No #if ENABLE_DEVICE_RESPONSE branching in dispatcher
event_response.cpp provides the unified dispatcher (send_event())
New implementation fully replaces legacy send_event() via linker
Legacy send_event() in stream_formatter.cpp is excluded via build_src_filter
platformio.ini controls which implementation is linked

[env:esp32dev-dev]
build_flags =
    ...
    -DENABLE_DEVICE_RESPONSE=1
    -DSTREAM_FORMAT=3          ; Ignored when using event_response.cpp

build_src_filter =
    +<*>
    -<stream_formatter.cpp>    ; ← Excludes legacy send_event() dispatcher
    -<response.cpp>            ; ← Excludes legacy response_t (not applicable to unified protocol)
    -<runtime_config.cpp>      ; ← Excludes legacy config_get/set_stream (replaced by Command class)
    -<text_command.cpp>
    -<text_command_manager.cpp>
    -<binary_command.cpp>

Rationale for Exclusions:

stream_formatter.cpp: Legacy event output (SSV/TSV/CSV/JSONL variants) - replaced by event_response.cpp
response.cpp: Legacy response_t serialization - not used in unified protocol (device_response_t is the single response format)
runtime_config.cpp: Legacy C-based configuration (config_get_stream_enabled, config_set_stream_enabled, etc.) - replaced by Command class singleton

Note: These exclusions are ONLY for ENABLE_DEVICE_RESPONSE=1 builds. Legacy builds (ENABLE_DEVICE_RESPONSE=0) need these files.

File: `src/event_response.cpp` (Unified Dispatcher)¶

#include "event_response.h"
#include <ArduinoJson.h>
#include "config.h"

#if ENABLE_DEVICE_RESPONSE

/**
 * @brief Unified send_event() dispatcher - replaces legacy stream_formatter version
 *
 * Entry point for all detection event output when ENABLE_DEVICE_RESPONSE=1.
 * Always outputs JSON Lines format (STREAM_FORMAT flag completely ignored).
 *
 * Implementation strategy:
 * - Linked INSTEAD OF stream_formatter.cpp::send_event() (via build_src_filter)
 * - Direct Command.get_stream() call (no abstraction needed, path is fixed)
 * - Single code path (unified protocol, no format variants)
 */
void send_event(const event_t *data) {
  // Query stream enable flag via Command class (only available in unified mode)
  if (!Command::getInstance().get_stream()) {
    return;  // Stream disabled, skip output
  }

  // Unified schema - JSON Lines format ONLY
  send_event_as_device_response(data);
}

void send_event_as_device_response(const event_t *data) {
  // JsonDocument with dynamic memory management
  // IMPORTANT: JSON Lines format ONLY - STREAM_FORMAT flag is ignored
  JsonDocument doc;

  // Unified schema common fields
  doc["type"] = "event";
  doc["status"] = "ok";
  doc["sent_at"] = device_get_timestamp();

  // Core event data (always present)
  doc["hit1"] = data->hit1;
  doc["hit2"] = data->hit2;
  doc["hit3"] = data->hit3;
  doc["adc"] = data->sensorValue;

  // Optional fields (compile-time conditional)
#if ENABLE_HITTYPE
  doc["hit_type"] = data->hit_type;
#endif

#if ENABLE_ADCMV
  doc["adc_raw"] = data->adc_raw;
  doc["adc_mv"] = data->adc_mv;
#endif

#if ENABLE_BME280
  doc["temp_c"] = data->tmp_c;
  doc["atm_pa"] = data->atm_pa;
  doc["hmd_pct"] = data->hmd_pct;
#endif

#if ENABLE_TIMESTAMP
  doc["uptime_ms"] = data->uptime_ms;
  doc["timedelta_us"] = data->timedelta_us;
#endif

#if ENABLE_RTC
  doc["unix_timestamp"] = data->unix_timestamp;
#endif

#if ENABLE_GNSS
  if (data->gnss_fix_valid) {
    doc["gnss_latitude"] = serialized(String(data->gnss_latitude, 6));
    doc["gnss_longitude"] = serialized(String(data->gnss_longitude, 6));
    doc["gnss_altitude"] = data->gnss_altitude;
    doc["gnss_satellites"] = data->gnss_satellites;
    doc["gnss_fix_quality"] = data->gnss_fix_quality;
    doc["gnss_hdop"] = data->gnss_hdop;
    doc["gnss_fix_valid"] = true;
  }
#endif

  // Serialize to JSON and output as single line
  serializeJson(doc, Serial);
  Serial.println();
}

device_response_t event_to_device_response(const event_t *data) {
  // Create response structure with event type
  device_response_t response = device_response_ok(DEVICE_TYPE_EVENT);

  // Caller must populate payload fields via JSON serialization
  // (this function only creates the envelope)
  return response;
}

#endif // ENABLE_DEVICE_RESPONSE

Key Benefits of This Approach:

✅ stream_formatter.cpp completely unchanged - no risk of introducing bugs in legacy code
✅ Clean symbol replacement - linker resolves send_event() to event_response.cpp (no duplication)
✅ Complete path separation - ENABLE_DEVICE_RESPONSE=0 uses stream_formatter.cpp, =1 uses event_response.cpp
✅ Zero legacy code bloat - stream_formatter.cpp excluded via build_src_filter (only SSV/TSV/CSV linked if needed)
✅ Backward compatibility maintained - ENABLE_DEVICE_RESPONSE=0 builds work exactly as before

For ENABLE_DEVICE_RESPONSE=0 (Legacy Path):

[env:esp32dev-release]
build_flags =
    ...
    -DENABLE_DEVICE_RESPONSE=0
    -DSTREAM_FORMAT=0

# No build_src_filter needed - stream_formatter.cpp linked normally
# send_event() dispatcher from stream_formatter.cpp handles STREAM_FORMAT selection

Command Class Integration Points¶

1. Timestamp Management:

device_get_timestamp() in device_response.cpp handles Unix time vs uptime selection
Command::get_uptime() provides millisecond-precision uptime (cached)
event_response.cpp uses device_get_timestamp() for sent_at field
Future: Consolidate both timestamp sources in Command class

2. Stream Control Unification:

Current (ENABLE_DEVICE_RESPONSE=0): runtime_config module handles config_set_stream_enabled()
Unified (ENABLE_DEVICE_RESPONSE=1): Command::get_stream() / Command::set_stream()
Dispatcher function get_stream_enabled() abstracts the selection (macro guard)
Text commands (SET_STREAM, GET_STREAM) route to Command.set_stream() when enabled

3. Feature Flags:

ENABLE_BME280, ENABLE_TIMESTAMP, ENABLE_GNSS: Used by both event_response.cpp and Command getters
event_response.cpp respects compile-time flags; Command class provides runtime validation
No redundancy: event output and configuration use same flag definitions

Backward Compatibility Strategy¶

For ENABLE_DEVICE_RESPONSE=0 (default, legacy):

send_event() dispatcher uses existing STREAM_FORMAT selection
stream_formatter.cpp unchanged; all format functions (SSV/TSV/CSV/JSONL) compiled
runtime_config module active; config_get_stream_enabled() checked
No impact on production firmware (zero code addition)

For ENABLE_DEVICE_RESPONSE=1 (unified schema):

send_event() dispatcher calls send_event_as_device_response()
legacy format functions excluded via preprocessor (zero code footprint)
event_response.cpp and event_response.h linked
Command class active; stream control via Command::get_stream()
JSON schema validation possible (device-response.json)

Testing Strategy¶

Unit Tests (Phase 3):

event_to_device_response() returns correct envelope (type=event, status=ok)
send_event_as_device_response() produces valid single-line JSON
JSON field presence matches ENABLE_* flags at compile time
Optional fields (ENABLE_GNSS, ENABLE_TIMESTAMP) correctly included/excluded
sent_at field matches device_get_timestamp() output

Integration Tests:

Stream disable/enable flag works via Command.set_stream()
Output switches correctly between Path A and Path B based on ENABLE_DEVICE_RESPONSE
JSON schema validation passes with docs/schemas/device-response.json

Regression Tests:

Legacy STREAM_FORMAT=0-3 output identical when ENABLE_DEVICE_RESPONSE=0
No binary size increase for ENABLE_DEVICE_RESPONSE=0 builds
Memory usage stable (no heap fragmentation with JsonDocument)
STREAM_FORMAT flag is ignored when ENABLE_DEVICE_RESPONSE=1 (JSON Lines forced)

Implementation Checklist (Phase 3, v1.13.1)¶

Build Configuration ✅ COMPLETED:

Update platformio.ini (env:esp32dev-dev) build_src_filter:
Add -<stream_formatter.cpp> (legacy event output)
Add -<response.cpp> (legacy response_t, not applicable to unified protocol)
Add -<runtime_config.cpp> (legacy config functions replaced by Command class)

Source Code Guarding ✅ COMPLETED:

Add ENABLE_DEVICE_RESPONSE guards to main.cpp includes (lines 30-34)
Add ENABLE_DEVICE_RESPONSE == 0 guard to config_init() call (lines 249-251)
Remove unnecessary runtime_config.h include from device_response_send.cpp

Dependency Cleanup ✅ COMPLETED:

Remove stream_data.h include from device_response_builder.h
Delete DeviceResponseBuilder::from_event() method (circular dependency eliminated)
Replace with inline event serialization in:
main.cpp::detection_process() (line 219-259)
detection_buffer.cpp::detection_buffer_send() (line 181-220)
Benefit: DeviceResponseBuilder now has zero dependencies on event/detection structures

Core Implementation (PLANNED):

Create include/event_response.h with unified event serialization function
Note: Consolidates inline event serialization (now in main.cpp and detection_buffer.cpp)
Future optimization: Extract common event serialization logic
Proposed for v1.13.2: send_event_as_device_response(event_t *data) helper
Implement src/event_response.cpp with unified serialization dispatcher
Will replace legacy stream_formatter.cpp::send_event() when activated
Always outputs JSON Lines format (STREAM_FORMAT flag ignored)
Respects Command::getInstance().get_stream() flag

Validation & Testing:

Documentation:

Update REFACTORING_ROADMAP.md Phase 3 status (v1.13.1 completion)
Update docs/api.md with new event_response module documentation
Document build_src_filter usage in CLAUDE.md

Learnings¶

Design Decisions and Rationale¶

Why JSON Lines ONLY format (no STREAM_FORMAT variants)?
Reason: Unified protocol requires deterministic output; supporting SSV/TSV/CSV would fragment the schema and complicate validation
Benefit: Single canonical format for all clients; JSON schema validation always applicable; simplified implementation (no #if STREAM_FORMAT branching)
Trade-off: Legacy STREAM_FORMAT users must remain on ENABLE_DEVICE_RESPONSE=0 (backward compatible)
Why separate event_response.h instead of extending device_response.h?
Reason: Single Responsibility - device_response.h handles response envelope and builder; event_response.h handles event-specific serialization
Benefit: Keeps each module focused; easier to test and modify independently
CLAUDE.md alignment: Follows existing module separation pattern (e.g., stream_formatter.h vs stream_data.h)
Why maintain send_event() dispatcher instead of creating new entry point?
Reason: Backward compatibility - existing main.cpp calls send_event() unconditionally
Benefit: Minimal surface API changes; clients don't need to know about unified protocol
Risk mitigation: Conditional compilation ensures legacy builds unchanged
Why use get_stream_enabled() abstraction instead of direct Command calls?
Reason: Dual-path support - must work with both Command (ENABLE_DEVICE_RESPONSE=1) and runtime_config (ENABLE_DEVICE_RESPONSE=0)
Benefit: Clean separation of concerns; dispatcher doesn't know which backend is active
Future: When ENABLE_DEVICE_RESPONSE becomes mandatory, simplify to direct Command.get_stream()
JSON Serialization Strategy: Per-Event JsonDocument vs. Global:
Chosen: Per-event JsonDocument (1-2KB stack usage per event)
Rationale: ESP32 stack is large enough; avoids global state and memory fragmentation
Alternative considered: Global JsonDocument (saves stack, but creates state sharing complexity)
Measured: SC-012 stack pointer validation confirms <2KB per event
GNSS field handling: Only include if fix_valid=true:
Reason: Stale/invalid GNSS data can mislead analysis
Benefit: Cleaner output; clients don't filter invalid data
Trade-off: May hide debugging information (future: add verbosity flag if needed)

Potential Issues and Mitigations¶

Issue: JsonDocument allocation per event could cause memory pressure
Mitigation: Measure stack usage; consider per-100-events heap stats
Fallback: Switch to fixed-size buffer if fragmentation observed
Issue: ENABLE_DEVICE_RESPONSE flag proliferation (already 8 enable flags + device response)
Mitigation: Document flag interactions in config.h; consider flag registry future
Pattern: Flags are additive (no conflicts); preprocessor handles all combinations
Issue: Command class integration deferred (set_stream via Command not yet tested)
Mitigation: Phase 3 includes integration testing; Phase 4 (v1.14.0) makes Command mandatory

Next Steps¶

Immediate (v1.13.1 implementation):
Implement include/event_response.h and src/event_response.cpp
Modify send_event() dispatcher with branching logic
Test both paths: ENABLE_DEVICE_RESPONSE=0 (legacy) and =1 (unified)
Short-term (v1.13.1+):
Create unit tests for event_response serialization
Validate JSON schema conformance
Update user documentation with protocol examples
Medium-term (v1.14.0):
Phase 4: Remove legacy response.h/cpp, make unified protocol mandatory
Consolidate timestamp sources (device_get_timestamp + Command.get_uptime)
Extend Command class with event-aware methods (future: event filtering, batching)
Long-term (v2.0.0):
Multi-output path support (Serial, WiFi, LittleFS) with unified schema
Event serialization framework for other protocols (MessagePack, Protobuf)
Language-specific code generation from device-response.json schema

References¶

REFACTORING_ROADMAP.md: Phase 1-4 planned roadmap
docs/schemas/device-response.json: Strict validation schema
docs/architecture/unified-device-response-schema.md: Protocol specification
CLAUDE.md: Module organization guidelines and Command class documentation
include/command.h: Command singleton API (460 lines, full JSDoc)
src/command.cpp: Command implementation with validation (382 lines)