Date Created: 2025-12-08
Last Modified: 2025-12-08

Progress Log: Unified Device Response and Event Structure Design¶

Task Description¶

Analyze and design unified data structures for device responses (response_t) and detection events (event_t) to eliminate redundancy and provide consistent output handling across different output modes (JSONL, SSV, TSV, CSV, etc.).

Current State Analysis¶

Two separate data structures currently handle different output types:

response_t - Used by text command handlers (defined in include/response.h)
Contains: status (enum: RESPONSE_OK/RESPONSE_ERROR), pair_count (0-4), union variants with 0-4 key-value pairs
Builder functions: response_ok(), response_error(), response_string(), response_int(), response_uint(), response_pairs() (overloaded for 2/3/4 pairs)
Serialization: send_response() dispatcher converts response_t to JSONL format with variant routing
Output format: {"type":"response","status":"ok"|"error"[,"key":value,...]} (JSONL)
Responsibility: Command acknowledgments, structured data responses with fixed pair slots
Limitation: Fixed structure (0-4 key-value pairs), not extensible for arbitrary field sets
event_t - Used for detection event output (defined in include/stream_data.h)
Core fields (always present): hit1, hit2, hit3, sensorValue (ADC reading)
Optional fields (controlled by ENABLE flags):
- hit_type (ENABLE_HITTYPE=1) - Detection pattern bitmask
- adc_raw, adc_mv (ENABLE_ADCMV=1) - Raw ADC and millivolt conversion
- tmp_c, atm_pa, hmd_pct (ENABLE_BME280=1) - Environmental sensor data
- uptime_ms, timedelta_us (ENABLE_TIMESTAMP=1) - Timing data
- unix_timestamp (ENABLE_RTC=1) - Absolute time
- GNSS fields (ENABLE_GNSS=1) - Satellite positioning data
Responsibility: Carry detection event data through pipeline to formatter
Serialization: Multiple format support via stream_formatter:
- SSV (Space-Separated Values)
- TSV (Tab-Separated Values)
- CSV (Comma-Separated with header)
- JSONL (JSON Lines)
Limitation: Event-only data structure, no command response integration

Design Challenge¶

Command responses (response_t) and event data (event_t) serve different purposes with different output constraints
Response structure is fixed (pair slots), event structure is flexible (optional fields via preprocessor)
No unified interface for representing "device output" as a general concept
Current code paths: Commands → send_response() (JSONL), Events → stream_formatter (multi-format)
Potential for consolidating response generation into same formatting pipeline as events

Proposed: Minimal Unified DeviceResponse Structure¶

Core requirement: Single output type covering both responses and events.

typedef struct {
  const char* type;          // "response" | "event"
  const char* status;        // "ok" | "error"
  uint32_t sent_at;          // Unix timestamp (optional, RTC dependent)
  JsonDocument payload;      // JSON object (flat or nested) for response/event data
} device_response_t;

Rationale (SRP + YAGNI):

Flexible JsonDocument payload: Supports both flat and nested JSON structures, adapts to response or event needs
Minimal fields: Only type, status, timestamp, and payload
Single serializer: send_device_response(device_response_t) outputs JSONL
No breaking changes: Existing code paths unchanged initially
Gradual migration: Handlers can adopt device_response_t incrementally

Implementation Path (Minimal):

Define device_response_t in new header include/device_response.h
Implement send_device_response() to serialize to JSONL
Provide minimal builder functions for response/event types
Keep response_t and event_t in place (no refactoring yet)
Let handlers opt-in to device_response_t voluntarily

Examples of DeviceResponse¶

GET_VERSION

{
    "type": "response",
    "status": "ok",
    "sent_at": 123456.123456,
    "version": "version": "1.10.0",
}

GET_MAC_ADDRESS

{
    "type": "response",
    "status": "ok",
    "sent_at": 123456.123456,
    "mac_address": "AA:BB:CC:DD:EE:FF",
}

GET_UPTIME

{
    "type": "response",
    "status": "ok",
    "sent_at": 123456.123456,
    "uptime_ms": 123456,
}

GET_STREAM / SET_STREAM

{
    "type": "response",
    "status": "ok",
    "sent_at": 123456.123456,
    "stream": stream_enabled,
}

GET_RTC_TIME / SET_RTC_TIME

{
    "type": "response",
    "status": "ok",
    "sent_at": 123456.123456,
    "rtc_time": 17320466789,
}

GET_GNSS_TIME

{
    "type": "response",
    "status": "ok",
    "sent_at": 123456.123456,
    "gnss_time": 17320466789,
}

GET_GNSS_POSITION

{
    "type": "response",
    "status": "ok",
    "sent_at": 123456.123456,
    "gnss_position": {
        "latitude": 123.4567,
        "longitude": 123.4567,
        "altitute": 123.4567,
    }
}

GET_GNSS

{
    "type": "response",
    "status": "ok",
    "sent_at": 123456.123456,
    "gnss": {
        "unixtime": 1234567890,
        "latitude": 123.4567,
        "longitude": 123.4567,
        "altitude": 123.4567,
        "satellites": 4,
        "hdop": 1.04,
        "fix_valid": true,
    }
}

GET_BME280

{
    "type": "response",
    "status": "ok",
    "sent_at": 123456.123456,
    "bme280": {
        "tmp_c": 1234567890,
        "atm_pa": 123.4567,
        "hmd_pct": 123.4567,
    }
}

SET_THRESHOLD / GET_THRESHOLD

{
    "type": "response",
    "status": "ok",
    "sent_at": 123456.123456,
    "threshold": {
        "channel": 1,
        "threshold": 300,
    }
}

SET_DEADTIME / GET_DEADTIME

{
    "type": "response",
    "status": "ok",
    "sent_at": 123456.123456,
    "deadtime_ms": 0,
}

GET_EVENT

{
    "type": "response",
    "status": "data",
    "sent_at": 123456.123456,
    "payload": {
        "hit1": 100,
        "hit2": 100,
        "hit3": 100,
        "adc": 1004,
        "tmp_c": 28.5,
        "atm_pa": 101300,
        "hmd_pct": 44.0,
        "uptime_ms": 123456,
        "timedelta_us": 123456.123456,
        "adc_mv": 934,
        "adc_raw": 1023,
        "rtc": {
            "unixtime": 123456.123456,
        },
        "gnss": {
            "unixtime": 123456.123456,
        }
    }
}

Phased Implementation Strategy with ENABLE_DEVICE_RESPONSE Flag¶

To enable safe, incremental adoption of DeviceResponse without breaking existing clients:

Configuration Flag (in config.h)¶

// ============================================================================
// Device Response Format Configuration (v1.12.0+)
// ============================================================================
// Controls whether unified DeviceResponse format is compiled into firmware.
// - ENABLE_DEVICE_RESPONSE=1: Include device_response_t and send_device_response()
// - ENABLE_DEVICE_RESPONSE=0: Use legacy response_t format (default, backward compatible)
// Default: 0 (disabled for backward compatibility)
// Override via platformio.ini build_flags: -DENABLE_DEVICE_RESPONSE=1
//
// DeviceResponse format:
// - All responses include: type, status, sent_at (Unix timestamp), payload
// - Unifies command responses and detection events under single output format
// - sent_at enables client-side logging with absolute timestamps (timezone-independent)
// - payload supports both flat and nested JSON for flexible data representation
//
// Memory overhead when enabled:
//   - Flash: ~500-800 bytes (device_response.cpp + builder functions)
//   - RAM: ~200 bytes (per JsonDocument, allocated on-demand)
//
// Backward compatibility:
// - When ENABLE_DEVICE_RESPONSE=0: send_response(response_t) works as before
// - When ENABLE_DEVICE_RESPONSE=1: New handlers use send_device_response(device_response_t)
// - Clients can support both formats (check "sent_at" field presence)
//
// Migration path (v1.12.0 → v1.14.0):
// 1. v1.12.0: ENABLE_DEVICE_RESPONSE=0 (default), new infrastructure available for opt-in
// 2. v1.13.0: ENABLE_DEVICE_RESPONSE=0 (default), gradually migrate commands to device_response_t
// 3. v1.14.0: ENABLE_DEVICE_RESPONSE=1 (default), complete migration, legacy response_t deprecated

#ifndef ENABLE_DEVICE_RESPONSE
#define ENABLE_DEVICE_RESPONSE 0   // 1=unified format, 0=legacy format (default)
#endif

Phase 1: v1.12.0 - Infrastructure & Testing (ENABLE_DEVICE_RESPONSE optional)¶

Goal: Build, test, and validate new infrastructure with zero impact on existing code.

Implementation: - ✅ Define device_response_t structure in include/device_response.h - ✅ Implement send_device_response() serialization in src/device_response.cpp - ✅ Create builder functions: device_response_ok(), device_response_error(), device_response_event() - ✅ Keep response_t and event_t completely unchanged - ✅ Zero compile-time overhead when ENABLE_DEVICE_RESPONSE=0

Testing:

# Build with legacy format (default)
task build                                    # ENABLE_DEVICE_RESPONSE=0

# Build with new format (test)
PLATFORMIO_BUILD_FLAGS="-DENABLE_DEVICE_RESPONSE=1" pio run -e esp32dev-release

# Test both formats with serial monitor
task monitor

Code Example (both formats compile-time switchable):

// text_command_manager.cpp - GET_VERSION handler
#if ENABLE_DEVICE_RESPONSE
  // New format with sent_at and payload
  send_device_response(device_response_ok("version", config_get_version()));
  // Output: {"type":"response","status":"ok","sent_at":1732046789,"version":"1.10.0"}
#else
  // Legacy format without sent_at
  send_response(response_string("version", config_get_version()));
  // Output: {"type":"response","status":"ok","version":"1.10.0"}
#endif

Phase 2: v1.13.0 - Selective Handler Migration (ENABLE_DEVICE_RESPONSE default still 0)¶

Goal: Migrate a subset of commands while maintaining full backward compatibility.

Strategy: - ✅ Migrate "simple query" commands first: GET_VERSION, GET_UPTIME, GET_MAC_ADDRESS - ✅ Test with clients that support both formats - ✅ Leave "complex" commands unchanged: GET_STATUS, GET_GNSS_STATUS - ✅ Default flag remains ENABLE_DEVICE_RESPONSE=0 for production stability

Commands to migrate (v1.13.0): - GET_VERSION → use device_response_ok("version", ...) - GET_UPTIME → use device_response_ok("uptime_ms", ...) - GET_MAC_ADDRESS → use device_response_ok("mac_address", ...) - GET_RTC_TIME / SET_RTC_TIME → use device_response_ok("rtc_timestamp", ...)

Client-side compatibility (Python example):

def parse_device_response(raw_json: str):
    """Support both legacy and new formats"""
    response = json.loads(raw_json)

    # Detect format based on presence of 'sent_at'
    if 'sent_at' in response:
        # New DeviceResponse format
        sent_at = response['sent_at']
        payload = response.get('payload', response)
        device_time = datetime.utcfromtimestamp(sent_at)
    else:
        # Legacy response_t format
        sent_at = None
        payload = response
        device_time = None

    return {
        'type': response.get('type'),
        'status': response.get('status'),
        'sent_at': sent_at,
        'device_time': device_time,
        'payload': payload
    }

Phase 3: v1.14.0 - Full Unification (ENABLE_DEVICE_RESPONSE default becomes 1)¶

Goal: Make DeviceResponse the default output format; legacy format becomes opt-out.

Changes: - ✅ All command handlers use device_response_t (no more response_t) - ✅ Detection events use device_response_event() builder - ✅ Single output path: send_device_response() handles all cases - ✅ Legacy response_t marked deprecated (kept for 1 version for compatibility) - ✅ Default: ENABLE_DEVICE_RESPONSE=1

Output Consistency:

// Command response (GET_VERSION)
{"type":"response","status":"ok","sent_at":1732046789,"version":"1.10.0"}

// Detection event
{"type":"event","status":"data","sent_at":1732046789,"hit1":100,"hit2":85,...}

// Error response
{"type":"response","status":"error","sent_at":1732046789,"error_code":2,"error_message":"..."}

Summary: Phased Rollout Timeline¶

Version	ENABLE_DEVICE_RESPONSE	Status	Focus
v1.11.x	N/A	Legacy format only	Current production
v1.12.0	0 (optional)	Infrastructure ready	New system available for testing
v1.13.0	0 (default)	Selective migration	Some commands migrated, most unchanged
v1.14.0	1 (default)	Full adoption	All commands unified, legacy deprecated
v1.15.0	1 (only)	Legacy removed	Cleanup: remove response_t completely

Zero-Cost Abstraction Guarantee¶

✅ When ENABLE_DEVICE_RESPONSE=0: - No compilation of device_response.cpp (linker strips unused code) - No runtime overhead (no extra function calls) - Binary size identical to current firmware - Output format unchanged (100% backward compatible)

✅ When ENABLE_DEVICE_RESPONSE=1: - Additional ~600 bytes Flash (builder functions + serialization logic) - JsonDocument allocated on-demand per response (~200 bytes temporary) - sent_at computed once per response (minimal CPU cost) - New output format with rich metadata

Final Implementation Design: Minimal + Payload-Based¶

After design analysis, the following implementation approach is recommended:

Core Structure: Simple and Focused¶

device_response.h (~30-40 lines):

/**
 * @brief Unified device response structure for both commands and events
 *
 * Combines:
 * - Command responses (GET_VERSION, GET_THRESHOLD, etc.)
 * - Detection events (cosmic ray signals)
 * - Error responses
 *
 * All serialized to JSONL format with consistent metadata.
 */

typedef struct {
  const char* type;        // "response" | "event"
  const char* status;      // "ok" | "error" | "data"
  uint32_t sent_at;        // Unix timestamp for client-side logging
  JsonDocument payload;    // Flexible JSON container (flat or nested)
} device_response_t;

// Builder functions (minimal, optional)
inline device_response_t device_response_ok(
    const char* key, int32_t value) {
  JsonDocument payload;
  payload[key] = value;
  return {"response", "ok", device_get_timestamp(), payload};
}

inline device_response_t device_response_ok(
    const char* key, const char* value) {
  JsonDocument payload;
  payload[key] = value;
  return {"response", "ok", device_get_timestamp(), payload};
}

device_response_t device_response_ok_payload(
    const JsonDocument& payload);

device_response_t device_response_error(
    uint8_t error_code, const char* message);

device_response_t device_response_event(
    const JsonDocument& event_payload);

void send_device_response(const device_response_t& resp);

Key Design Decisions¶

1. JsonDocument Payload (No Wrapper Class)¶

Rationale: - ✅ JsonDocument is powerful and flexible enough - ✅ No redundant Payload wrapper class - ✅ Direct use keeps code simple (YAGNI principle) - ✅ Can add helper functions/classes later if patterns emerge

Usage Pattern:

// Simple case: use builder
send_device_response(device_response_ok("version", "1.10.0"));

// Complex case: direct JsonDocument
JsonDocument payload;
JsonObject gnss = payload["gnss"].to<JsonObject>();
gnss["latitude"] = 37.3874;
gnss["longitude"] = 121.9724;
send_device_response(device_response_ok_payload(payload));

2. Inline Builders for Type Safety¶

Rationale: - ✅ Compiler catches type mismatches early - ✅ Enables move semantics (std::move) - ✅ Zero runtime overhead (inlined by compiler) - ✅ Clear intent via function overloading

Implementation:

inline device_response_t device_response_ok(
    const char* key, int32_t value) {
  JsonDocument payload;
  payload[key] = value;
  return {"response", "ok", device_get_timestamp(), std::move(payload)};
}

inline device_response_t device_response_ok(
    const char* key, const char* value) {
  JsonDocument payload;
  payload[key] = value;
  return {"response", "ok", device_get_timestamp(), std::move(payload)};
}

3. Timestamp Helper Function¶

Rationale: - ✅ Centralized timestamp logic (RTC vs uptime fallback) - ✅ Single source of truth for sent_at calculation - ✅ Easy to modify behavior without changing all builders

Implementation:

// device_response.h
static inline uint32_t device_get_timestamp(void) {
  #if ENABLE_RTC
    return rtc_get_time();  // Unix timestamp from RTC
  #else
    // Fallback: device uptime in seconds
    // Note: Not absolute Unix time, but device seconds from boot
    return millis() / 1000;
  #endif
}

4. Conditional Compilation for Zero Overhead¶

Rationale: - ✅ When ENABLE_DEVICE_RESPONSE=0, device_response.cpp not compiled - ✅ Linker strips all unused code automatically - ✅ No binary size increase when disabled - ✅ Pure opt-in adoption model

Implementation:

// text_command_manager.cpp
#if ENABLE_DEVICE_RESPONSE
  send_device_response(device_response_ok("version", config_get_version()));
#else
  send_response(response_string("version", config_get_version()));
#endif

Output Format Examples¶

All formats use JSONL with consistent structure:

// Simple command response
{"type":"response","status":"ok","sent_at":1732046789,"version":"1.10.0"}

// Error response
{"type":"response","status":"error","sent_at":1732046789,"error_code":2,"error_message":"..."}

// Complex response with nested structure
{"type":"response","status":"ok","sent_at":1732046789,"gnss":{"latitude":37.3874,"longitude":121.9724,"altitude":45.9}}

// Detection event
{"type":"event","status":"data","sent_at":1732046789,"hit1":100,"hit2":85,"hit3":92,"adc":2048}

Implementation Checklist (Phase 1)¶

Create include/device_response.h with struct + inline builders
Implement src/device_response.cpp with serialization logic
Add ENABLE_DEVICE_RESPONSE flag to include/config.h
Wrap builder calls with #if ENABLE_DEVICE_RESPONSE in handlers (optional)
Test compilation with both ENABLE_DEVICE_RESPONSE=0 and 1
Verify binary size unchanged when disabled
Test serial output format (both formats via flag)

Benefits of This Approach¶

Benefit	Details
Minimal	Only 30-40 lines of header, simple implementation
Flexible	JsonDocument payload handles any JSON structure
Zero-cost	Fully conditional compilation when disabled
Type-safe	Inline builders with overloading catch errors early
Move semantics	Efficient JsonDocument transfer via `std::move`
Backward compatible	Legacy response_t unchanged during Phase 1
Extensible	Easy to add helper functions/classes later (Phase 2+)

Future Enhancements (Phase 2+)¶

If certain payload patterns emerge frequently, can add: - Helper functions: device_response_threshold(), device_response_gnss_status() - Payload builder class: Only when 3+ handlers share same structure - Template metaprogramming: For fully generic payload construction

Philosophy: Start minimal, add abstractions only when patterns justify them.

Outcome¶

Design document framework created (in-progress) covering:

Data Structure Analysis
Documented response_t structure: 3-layer architecture (types → builders → serialization)
Documented event_t structure: core + optional fields with preprocessor control
Identified output patterns:
- Responses: Fixed pair count (0-4), always JSONL, always includes type/status headers
- Events: Variable field count (4-19 depending on ENABLE flags), multi-format, no type header
Unified Structure Proposal
Key design insight: These are legitimately different abstractions
- response_t = "command reply" (intentionally fixed-size for simplicity)
- event_t = "sensor observation" (intentionally flexible for extensibility)
Option A: Create device_output_t tagged union encompassing both response and event variants
Option B: Extend response_t with event variant (adding event_t fields to union)
Option C: Extend event_t with response metadata (adding status/error fields to struct)
Option D: Create shared formatter interface accepting both types polymorphically
Output Pipeline Analysis
Current: Separate paths for responses (send_response) and events (stream_formatter)
Common ground: Both eventually produce serial output (JSONL or other formats)
Opportunity: Unified formatter could accept either response_t or event_t
Challenge: response_t is response-specific (status, error_code), event_t is event-specific (hit1/2/3)
Field Mapping Consistency
Response field types: int32_t, uint32_t, const char* (string), bool (is_string)
Event field types: uint16_t (hits), int (sensorValue), float (sensors), uint32_t (timestamps), double (GNSS)
Type alignment challenge: Response uses int32_t, Event uses uint16_t/float/double
Opportunity: Standardize on common numeric types (e.g., int32_t for all values)
Builder vs Struct Trade-offs
response_t uses builder pattern: send_response(response_int("key", 42))
event_t populated imperatively: event.hit1 = 100; event.hit2 = 85;
Consistency opportunity: Could use builders for both (response_event(), event_builder)
Or: Document clear boundaries (response_t for replies, event_t for observations)

Learnings¶

Structural Differences Are Intentional
response_t: Command reply with fixed reply slots (0-4 pairs)
event_t: Sensor observation with variable optional fields
These solve different problems (confirmations vs measurements)
Forcing unification could reduce clarity and add overhead
JSONL as Common Output Format
Both responses and events can be serialized to JSONL
Responses: {"type":"response","status":"ok|error",...}
Events: Format varies (SSV in default, JSONL when STREAM_FORMAT=3)
Not all events use JSONL (SSV is default for simplicity/performance)
Preprocessor Flexibility in event_t
Optional fields via #if ENABLE_* directives provide true zero-cost abstraction
response_t has no such flexibility (all builders/variants always compiled)
This asymmetry is appropriate: responses are rare, events are frequent
Type System Differences
response_t: Uniform types (int32_t, uint32_t, const char*)
event_t: Diverse types (uint16_t, float, double, bool) per sensor
Unification would require type widening (int32_t) or heterogeneous union
Builder Pattern Benefits for response_t
Type-safe construction: response_pairs() enforces pair structure
Eliminates JSON boilerplate in handlers
Compiler catches misuse (wrong number of args)
event_t doesn't need this (struct field assignment is clear)

Next Steps¶

Compare with REFACTORING_ROADMAP.md v1.12.0+ plans
Check if response/event unification is already planned
Determine priority relative to other architectural work
Align design with existing roadmap vision
Decide: Unify or Maintain Separation?
If unifying: Choose approach (tagged union, polymorphic formatter, or type traits)
If keeping separate: Document clear contract (when to use response_t vs event_t)
Option: Unify at serialization layer only (single formatter accepting both)
If Consolidating at Formatter Layer
Create device_output_formatter.h accepting response_t or event_t
Implement format dispatch: format_output(const void* output, output_type_t type)
Support all existing formats (JSONL, SSV, TSV, CSV)
Maintain current send_response() as thin wrapper for compatibility
If Consolidating Structures
Design device_output_t union with response and event variants
Create builders for both: output_response(), output_event()
Update all handlers to construct device_output_t instead of response_t
Implement single send_output() dispatcher
Document Design Decision
Create docs/architecture/response-event-unification.md documenting chosen approach
Explain trade-offs and rationale for separation or consolidation
Provide migration guide if restructuring is implemented