• Date Created: 2025-12-11
• Last Modified: 2025-12-11

Progress Log: Phase 6 Payload Serialization Strategy

Task Description

Define implementation strategy for Phase 6: completing the 3-layer architecture by implementing payload field serialization in DeviceResponse::send(). The challenge is handling multiple command types, each with different payload structures, using a single send() method without code duplication.

Problem: Command-Specific Payloads

Each command returns different payload fields in JSON:

• GET_VERSION: version (1 field)
• GET_STATUS: uptime_ms, mac_address, poll_count, deadtime_ms (4 fields)
• GET_THRESHOLD/SET_THRESHOLD: channel, threshold (2 fields as pair)
• GET_UPTIME: uptime_ms (1 field)
• GET_MAC_ADDRESS: mac_address (1 field)
• GET_POLL_COUNT: poll_count (1 field)
• SET_POLL_COUNT: poll_count (1 field)
• GET_DEADTIME: deadtime_ms (1 field)
• SET_DEADTIME: deadtime_ms (1 field)
• Other commands with their own payloads

Challenge: Implement universal serialization without hardcoding each command's format.

Solution: Three-Layer Approach with Selective Serialization

Layer 1: Handlers (No Conditionals)

Handlers populate ALL possible payload fields. Only the fields relevant to the command are populated.

Example: GET_VERSION

command_response_t handle_get_version(const command_t& cmd) {
  if (cmd.arg_count != 0) {
    return error_response(DEVICE_CODE_INVALID_ARG, "GET_VERSION takes no arguments");
  }

  const char* version = Command::getInstance().get_version();

  command_response_t response = success_response();
  strncpy(response.version, version, sizeof(response.version) - 1);
  response.version[sizeof(response.version) - 1] = '\0';

  // Other fields (uptime_ms, mac_address, poll_count, etc.) remain 0/empty
  return response;
}

Principle: Handler doesn't care about serialization. Just populate typed fields.

Layer 2: Conversion (Pure Data Copy)

Copy all fields from command_response_t → device_response_t. No conditionals.

device_response_t DeviceResponse::from_command(const command_response_t& cmd) {
  device_response_t response = {};

  // Envelope
  response.type = cmd.type;
  response.status = cmd.status;
  response.sent_at = cmd.sent_at;
  response.error_code = cmd.error_code;
  strncpy(response.error_message, cmd.error_message,
          sizeof(response.error_message) - 1);
  response.error_message[sizeof(response.error_message) - 1] = '\0';

  // Payload - flat copy, no conditionals
  strncpy(response.version, cmd.version, sizeof(response.version) - 1);
  response.version[sizeof(response.version) - 1] = '\0';
  response.uptime_ms = cmd.uptime_ms;
  strncpy(response.mac_address, cmd.mac_address, sizeof(response.mac_address) - 1);
  response.mac_address[sizeof(response.mac_address) - 1] = '\0';
  response.poll_count = cmd.poll_count;
  response.deadtime_ms = cmd.deadtime_ms;
  response.channel = cmd.channel;
  response.threshold = cmd.threshold;

  return response;
}

Principle: Pure data copy, deterministic, no side effects.

Layer 3: Serialization (Selective Based on Non-Zero/Non-Empty)

Only serialize fields that are populated (non-zero/non-empty).

void DeviceResponse::send(const device_response_t& response) {
  JsonDocument doc;

  // === Envelope (always) ===
  doc["type"] = (response.type == DEVICE_TYPE_RESPONSE) ? "response" : "event";
  doc["status"] = (response.status == DEVICE_STATUS_OK) ? "ok" : "error";
  doc["sent_at"] = response.sent_at;

  // === Error (when status=error) ===
  if (response.status == DEVICE_STATUS_ERROR) {
    doc["error_code"] = response.error_code;
    doc["error_message"] = response.error_message;
  }

  // === Payload (when status=ok, selective) ===
  if (response.status == DEVICE_STATUS_OK) {
    // String fields: check non-empty
    if (response.version[0] != '\0') {
      doc["version"] = response.version;
    }
    if (response.mac_address[0] != '\0') {
      doc["mac_address"] = response.mac_address;
    }

    // Integer fields: check > 0
    if (response.uptime_ms > 0) {
      doc["uptime_ms"] = response.uptime_ms;
    }
    if (response.poll_count > 0) {
      doc["poll_count"] = response.poll_count;
    }
    if (response.deadtime_ms >= 0) {  // 0 might be valid, check with context
      doc["deadtime_ms"] = response.deadtime_ms;
    }

    // Paired fields: channel enables threshold
    if (response.channel > 0) {  // Channels 1-3, 0 = not set
      doc["channel"] = response.channel;
      doc["threshold"] = response.threshold;
    }
  }

  serializeJson(doc, Serial);
  Serial.println();
}

Principle: Only serialize non-zero/non-empty fields. Saves bandwidth, matches JSON schema.

Concrete Examples

Example 1: GET_VERSION

Handler Output (Layer 1):

version = "v1.14.0"
uptime_ms = 0 (not set)
mac_address = "" (empty)
poll_count = 0 (not set)
channel = 0 (not set)

Conversion (Layer 2):

response.version = "v1.14.0"
response.uptime_ms = 0
response.mac_address = ""
response.poll_count = 0
response.channel = 0

Serialized Output (Layer 3):

{"type":"response","status":"ok","sent_at":12345,"version":"v1.14.0"}

Why: Only version is non-empty, so only it's included in JSON.

Example 2: GET_THRESHOLD 1

Handler Output (Layer 1):

channel = 1
threshold = 512
version = "" (not set)
uptime_ms = 0 (not set)
mac_address = "" (not set)
poll_count = 0 (not set)

Conversion (Layer 2):

response.channel = 1
response.threshold = 512
response.version = ""
response.uptime_ms = 0
response.mac_address = ""
response.poll_count = 0

Serialized Output (Layer 3):

{"type":"response","status":"ok","sent_at":12345,"channel":1,"threshold":512}

Why: channel > 0, so both channel and threshold are included. Other fields are 0/empty, so excluded.

Example 3: GET_STATUS

Handler Output (Layer 1):

uptime_ms = 5000
mac_address = "A4:CF:12:7F:9E:34"
poll_count = 100
deadtime_ms = 0
version = "" (not set)
channel = 0 (not set)

Conversion (Layer 2):

response.uptime_ms = 5000
response.mac_address = "A4:CF:12:7F:9E:34"
response.poll_count = 100
response.deadtime_ms = 0
response.version = ""
response.channel = 0

Serialized Output (Layer 3):

{"type":"response","status":"ok","sent_at":12345,"uptime_ms":5000,"mac_address":"A4:CF:12:7F:9E:34","poll_count":100,"deadtime_ms":0}

Why: uptime_ms, mac_address, poll_count, and deadtime_ms are all set (non-zero or non-empty). version and channel remain 0/empty, so excluded.

Example 4: Error Response (Any Command)

Handler Output (Layer 1):

status = ERROR
error_code = DEVICE_CODE_OUT_OF_RANGE (2)
error_message = "Invalid threshold (must be 0-1023)"
version = "" (not populated on error)
channel = 0 (not populated on error)
(all other fields 0/empty)

Conversion (Layer 2):

response.status = DEVICE_STATUS_ERROR
response.error_code = 2
response.error_message = "Invalid threshold (must be 0-1023)"
(payload fields copied as-is, but not serialized due to status=error)

Serialized Output (Layer 3):

{"type":"response","status":"error","sent_at":12345,"error_code":2,"error_message":"Invalid threshold (must be 0-1023)"}

Why: When status=error, only envelope + error fields are serialized. Payload fields are NOT included (error case, no successful data to return).

Field Selection Logic

String Fields: Non-Empty Check

if (response.version[0] != '\0') {
  doc["version"] = response.version;
}

- Empty string ("") = not set, don't include - Non-empty = valid data, include

Integer Fields: Non-Zero Check

if (response.uptime_ms > 0) {
  doc["uptime_ms"] = response.uptime_ms;
}

- 0 = not set, don't include - > 0 = valid data, include

Special Case: Valid Zero Values

if (response.poll_count > 0) {
  doc["poll_count"] = response.poll_count;  // 0 = not set
}

// But for fields where 0 IS valid (like deadtime_ms):
// Include if command that uses it was called
// Strategy: Use channel presence to determine if threshold-related command
if (response.channel > 0) {
  doc["channel"] = response.channel;
  doc["threshold"] = response.threshold;  // Always included with channel
}

Paired Fields: Keep Together

// Channel and threshold always appear together
if (response.channel > 0) {  // If channel is set
  doc["channel"] = response.channel;
  doc["threshold"] = response.threshold;  // Automatically included
}

Device Response Structure Update

// include/device_response.h:96-108 (UPDATED)
typedef struct {
  // Envelope (always present)
  device_response_type_t type;
  device_response_status_t status;
  uint32_t sent_at;
  device_response_code_t error_code;
  char error_message[256];

  // ===== Payload Fields (conditional serialization) =====

  // GET_VERSION
  char version[32];

  // GET_STATUS, GET_UPTIME, related
  uint32_t uptime_ms;
  char mac_address[18];
  uint16_t poll_count;
  uint16_t deadtime_ms;

  // GET_THRESHOLD, SET_THRESHOLD
  uint8_t channel;
  uint16_t threshold;

  // Reserved for future commands/features
  // ... additional fields as needed per spec
} device_response_t;

Implementation Checklist for Phase 6

Prerequisites

• Review current implementation (from_command currently envelope-only)
• Confirm all payload fields in command_response_t
• Review JSON schema docs/schemas/device-response.json

Step 1: Update device_response_t Structure

• Add all payload fields (version, uptime_ms, mac_address, poll_count, deadtime_ms, channel, threshold)
• Document each field's purpose and valid range
• Add comments about which commands use which fields

Step 2: Implement from_command() Payload Copying

• Copy all string fields (version, mac_address) with null termination
• Copy all integer fields (uptime_ms, poll_count, deadtime_ms, channel, threshold)
• Test that copy preserves all values
• Verify no conditionals at Layer 2

Step 3: Implement send() Selective Serialization

• String fields: if (field[0] != '\0')
• Integer fields: if (field > 0)
• Paired fields: if (channel > 0) include both channel and threshold
• Error path: serialize error_code and error_message when status=error
• Success path: serialize all non-zero/non-empty payload fields when status=ok

Step 4: Test Each Command

• GET_VERSION → only version field
• GET_STATUS → all status fields
• GET_THRESHOLD 1 → channel + threshold
• SET_THRESHOLD 1 512 → channel + threshold
• GET_UPTIME → only uptime_ms
• GET_MAC_ADDRESS → only mac_address
• GET_POLL_COUNT → only poll_count
• Error cases → only error_code + error_message

Step 5: Validate Against Schema

• Each response validates against docs/schemas/device-response.json
• No unexpected fields in JSON output
• All required fields present

Step 6: Update Tests & Documentation

• Update task.md Phase 6 completion notes
• Update CLAUDE.md with new payload serialization strategy
• Add examples to device_response.cpp comments

Design Principles

1. No Conditionals at Layer 1

Handlers should NOT check what fields to populate. Just populate everything relevant.

✅ CORRECT: Populate all handler-relevant fields

command_response_t response = success_response();
response.channel = ch;
response.threshold = val;
return response;

❌ WRONG: Conditional population

if (cmd == "GET_VERSION") response.version = ...;
if (cmd == "GET_STATUS") response.uptime_ms = ...;

2. No Conditionals at Layer 2

Conversion should be pure data copy. Copy everything.

✅ CORRECT: Flat copy

response.version = cmd.version;
response.channel = cmd.channel;
response.threshold = cmd.threshold;

❌ WRONG: Conditional copy

if (cmd.version[0]) response.version = cmd.version;
if (cmd.channel > 0) response.channel = cmd.channel;

3. Selective Serialization at Layer 3

Only serialize what's populated. Use presence (non-zero/non-empty) as indicator.

✅ CORRECT: Selective serialization

if (response.version[0] != '\0') doc["version"] = response.version;
if (response.channel > 0) {
  doc["channel"] = response.channel;
  doc["threshold"] = response.threshold;
}

Benefits

✅ Clean Layer Separation: Each layer has single, clear responsibility ✅ Flexible Payload Handling: Different commands → different payloads, all handled by single send() ✅ No Code Duplication: Single send() method, no per-command JSON format logic ✅ Schema Compliant: Output automatically matches JSON schema ✅ Extensible: New commands → add fields to struct + add serialization condition (no handler/layer2 changes) ✅ Testable: Each layer can be tested independently

Related Documents

• Design: /docs/progress/entries/2025-12-10-device-response-unification-analysis.md (lines 1459-1531)
• Analysis: /docs/progress/entries/2025-12-11-design-vs-implementation-analysis.md (found Phase 6 deferral)
• Schema: docs/schemas/device-response.json (validation reference)
• Tasks: specs/029-command-response-refactor/tasks.md (Phase 6 user story)