Skip to content

Phase 6: Payload Pointer Pattern Implementation (1.13.12)

Commits Since 1.13.11: 4 commits Date: 2025-12-11 Status: Implemented ✅ Build Status: SUCCESS

Overview

Phase 6 implements a more memory-efficient and type-flexible response payload pattern. Instead of embedding command-specific fields directly in the command_response_t struct, handlers now set a pointer to a static JsonDocument containing the payload. This allows:

  • Zero type restrictions on payload fields (no predefined struct members)
  • More efficient memory (pointer only = 8 bytes vs. multiple struct fields)
  • Greater flexibility for future commands without struct changes
  • Cleaner separation between Layer 1 (handler) and Layer 3 (serialization)

Key Changes

1. Data Structure Modifications

command_response_t (include/command_queue.h)

Before (Phase 0.5): 

typedef struct {
  char version[32];           // GET_VERSION
  uint32_t uptime_ms;         // GET_STATUS
  char mac_address[18];       // GET_STATUS
  uint16_t poll_count;        // GET_STATUS
  uint16_t deadtime_ms;       // GET_STATUS
  uint8_t channel;            // GET_THRESHOLD
  uint16_t threshold;         // GET_THRESHOLD
  // ... more fields per command type
} command_response_t;

After (Phase 6): 

typedef struct {
  const JsonDocument* payload;  // Pointer to static JsonDocument
} command_response_t;

Benefits: - Single field handles all command types (no need to add fields for new commands) - 8 bytes per response (pointer) vs. ~200+ bytes (typed fields) - Type-flexible: handlers define their own JSON structure

device_response_t (include/device_response.h)

Same change applied to the output struct: 

typedef struct {
  const JsonDocument* payload;  // Command-specific response payload
} device_response_t;

2. Handler Pattern (Phase 6: Payload Pointer)

All handlers now follow this unified pattern:

command_response_t handle_get_version(const command_t& cmd) {
  // Validation
  if (cmd.arg_count != 0) {
    return error_response(DEVICE_CODE_INVALID_ARG, "takes no arguments");
  }

  // Business logic
  const char* version = Command::getInstance().get_version();

  // Phase 6 Pattern: Create static payload
  command_response_t response = success_response();

  static JsonDocument payload;
  payload.clear();
  payload["version"] = version;  // Define any fields needed

  response.payload = &payload;   // Set pointer
  return response;
}

Key Points: - static JsonDocument ensures pointer validity through layers - Handler defines structure in JSON (no type restrictions) - error_response() automatically sets payload = nullptr - No type casting or serialization in handler code

3. Files Modified

File Changes Purpose
include/command_queue.h Replaced typed fields with JsonDocument* payload Unified payload structure
include/device_response.h Added JsonDocument* payload field Layer 3 serialization input
src/device_response.cpp Updated send() to merge payload with envelope Phase 6 serialization logic
src/command/version.cpp Implemented Phase 6 pattern reference implementation GET_VERSION handler
src/command/status.cpp Migrated to Phase 6 pattern GET_STATUS handler
src/command/threshold.cpp Migrated to Phase 6 pattern GET_THRESHOLD/SET_THRESHOLD
src/command/deadtime.cpp Migrated to Phase 6 pattern GET_DEADTIME/SET_DEADTIME
src/command/poll_count.cpp Migrated to Phase 6 pattern GET_POLL_COUNT/SET_POLL_COUNT
src/command/uptime.cpp Migrated to Phase 6 pattern GET_UPTIME handler
src/command/mac_address.cpp Migrated to Phase 6 pattern GET_MAC_ADDRESS handler

4. Commit Sequence

a6fae02: feat(device-response): implement Phase 6 payload pointer pattern

  • Initial implementation of JsonDocument* payload in command and device response types
  • Updated device_response.cpp to merge payload with JSON envelope
  • Documented Phase 6 pattern in header comments

659d36c: refactor(handlers): migrate to Phase 6 payload pointer pattern for version and threshold

  • Refactored version.cpp (reference implementation with full documentation)
  • Refactored threshold.cpp to Phase 6 pattern
  • Updated inline comments and handler documentation

ca8e10c: refactor(handlers): migrate remaining handlers to Phase 6 payload pointer pattern

  • Migrated status.cpp, deadtime.cpp, poll_count.cpp, uptime.cpp, mac_address.cpp
  • Unified all handlers to new pattern

a4c6845: fix(build): resolve ArduinoJson type compatibility errors

  • Added missing #include <ArduinoJson.h> in command_queue.h
  • Fixed JsonDocument const iteration compatibility with GCC 8.4.0

Build Status: All commits pass build verification (25.3% flash, 30.7% RAM)

Architecture Layers (Updated)

Phase 6 Message Flow

Layer 1 (Handler):
  └─> Create static JsonDocument with payload
  └─> Set response.payload = &payload
  └─> Return command_response_t

    ↓ (Dispatcher copies pointer)

Layer 2 (CommandQueue::execute):
  └─> Call DeviceResponse::from_command(command_response_t)
  └─> Copy payload pointer to device_response_t
  └─> Return device_response_t

    ↓ (Serializer merges payload)

Layer 3 (DeviceResponse::send):
  └─> Construct JSON envelope (type, status, sent_at, error fields)
  └─> Check if payload != nullptr
  └─> Merge payload fields into envelope
  └─> Serialize to JSON Lines
  └─> Output to Serial

Implementation Details

Static JsonDocument Lifetime

Each handler file contains a static JsonDocument variable:

// In version.cpp
static JsonDocument payload;

Why Static: - Persists across function calls (pointer remains valid) - Single instance per handler (no heap allocation) - No destructor concerns (JsonDocument cleans up on clear()) - Thread-safe on single-threaded ESP32

Payload Pointer Ownership

Entity Role Ownership
Handler Creates and populates Maintains (static lifetime)
CommandQueue Passes pointer through References (doesn't own)
DeviceResponse::send() Dereferences and merges References (doesn't own)

Null Safety: Layer 3 checks payload != nullptr before dereferencing

JSON Merging Algorithm (Layer 3)

// In device_response.cpp
if (dr.payload) {
  for (JsonPairConst pair : JsonObjectConst(dr.payload->as<JsonObject>())) {
    envelope[pair.key()] = pair.value();  // Merge payload fields
  }
}

This approach: - Iterates through all payload fields - Copies each field into the envelope - Works with any command type (no hardcoded field names) - Handles nested objects and arrays via ArduinoJson

Type Safety & Flexibility

Before Phase 6 (Type-Strict)

// Struct enforces specific types
uint16_t threshold;           // Must be uint16_t
char mac_address[18];         // Fixed size array
uint32_t uptime_ms;          // Only uint32_t allowed

After Phase 6 (Type-Flexible)

// JsonDocument accepts any type
payload["threshold"] = 512;           // int, uint16_t, uint32_t, etc.
payload["mac_address"] = "A4:CF:12";  // String of any length
payload["uptime_ms"] = 123456;        // Any numeric type
payload["nested"] = nestedDoc;        // Even nested objects

Benefit: New commands don't require struct field additions

Memory Usage

Comparison: Phase 0.5 vs Phase 6

Metric Phase 0.5 Phase 6 Savings
command_response_t size ~230 bytes 8 bytes 96.5% reduction
device_response_t size ~230 bytes 8 bytes 96.5% reduction
Per-response overhead 230 bytes 8 bytes 222 bytes
Flash (measured) ~330KB ~331KB +1KB (includes JSON code)
RAM (measured) 30.7% 30.7% Stable

Analysis: - Struct reduction: 230 → 8 bytes (pointer only) - JSON merging code: ~1KB (shared across all handlers) - Net: Smaller code + more flexible = Phase 6 wins

Backward Compatibility

Configuration

  • ENABLE_DEVICE_RESPONSE=1 (Phase 6 implementation)
  • ENABLE_DEVICE_RESPONSE=0 (falls back to legacy types)

When Phase 6 is disabled, system uses: - Original typed command_response_t struct - Legacy runtime_config module - No JsonDocument usage (ArduinoJson not required)

Error Response Handling

Error responses automatically use null payload:

return error_response(DEVICE_CODE_INVALID_ARG, "Invalid channel");
// Internally sets response.payload = nullptr
// Layer 3 checks: if (payload) { merge(); }
// Result: Only envelope fields serialized

Output Example: 

{"type":"response","status":"error","error_code":1,"error_message":"Invalid channel"}

Testing & Build Verification

Current Build Status

  • RAM: 30.7% (100,604 / 327,680 bytes)
  • Flash: 25.3% (331,393 / 1,310,720 bytes)
  • All 4 commits pass build verification

Manual Test Cases

GET_VERSION Success: 

Input:  GET_VERSION
Output: {"type":"response","status":"ok","sent_at":12345,"version":"v1.14.0"}

GET_THRESHOLD with Error: 

Input:  GET_THRESHOLD invalid_channel
Output: {"type":"response","status":"error","error_code":1,"error_message":"..."}

GET_STATUS Success (multiple payload fields): 

Input:  GET_STATUS
Output: {"type":"response","status":"ok","sent_at":12345,"uptime_ms":123456,"poll_count":100,"deadtime_ms":0,"mac_address":"A4:CF:12:7F:9E:34"}

Next Phase: Phase 7 (Planned)

Proposed enhancements: 1. Error Response Payloads: Include diagnostic info in error responses 2. Streaming Responses: Multi-message responses for large datasets 3. Client-Side Type Validation: JSON Schema-based validation of received payloads

Documentation Updates

Handler Development Guide

New pattern documentation in: - include/command_queue.h - Detailed Layer 1 instructions - src/command/version.cpp - Reference implementation - CLAUDE.md (command class documentation)

Key Principle: Separation of Concerns

  • Layer 1: Define payload structure (JSON format)
  • Layer 2: Transport data (copy pointers)
  • Layer 3: Serialize output (merge and output)

Handlers never call JSON serialization functions—Layer 3 owns that responsibility.

References