DO NOT MERGE: Hue instrumented capture

drivers/SmartThings/philips-hue/INSTRUMENTATION_README.md

@@ -0,0 +1,166 @@
+# Hue Driver Instrumentation - Capture Branch
+
+## Overview
+
+This branch contains a heavily instrumented version of the Philips Hue driver designed to capture comprehensive real-world behavior for analysis and refactoring.
+
+## What's Been Added
+
+### Core Logging Infrastructure
+
+1. **`capture_logger.lua`** - Structured JSON logging module
+   - Logs all network traffic (REST requests/responses, SSE events)
+   - Logs all IPC communication (commands, capability events, lifecycle)
+   - Logs state changes (device fields, datastore, discovery cache)
+   - Generates correlation IDs for request→response tracking
+   - High-resolution timestamps (milliseconds)
+   - Configurable sanitization of sensitive data
+
+2. **`capture_device_wrapper.lua`** - Automatic device event capture
+   - Wraps `device:emit_event()` to capture all capability events
+   - Wraps `device:set_field()` to capture state changes
+   - Wraps device creation/deletion
+   - Wraps online/offline status changes
+
+### Instrumented Modules
+
+- **`lunchbox/rest.lua`** - REST client with request/response logging
+- **`lunchbox/sse/eventsource.lua`** - SSE client with event logging
+- **`handlers/commands.lua`** - Command handlers with IPC logging
+- **`handlers/lifecycle_handlers/init.lua`** - Lifecycle event logging
+- **`init.lua`** - Driver initialization with wrapper integration
+
+## Log Format
+
+All capture logs are prefixed with `[CAPTURE]` and contain JSON objects with these fields:
+
+- `type` - Log category (NETWORK_OUT, NETWORK_IN, IPC_IN, IPC_OUT, etc.)
+- `subtype` - Specific event type (REST_REQUEST, SSE_EVENT, COMMAND, etc.)
+- `timestamp` - Milliseconds since epoch
+- Event-specific fields
+
+See `CAPTURE_TEST_PLAN.md` Appendix for detailed log format examples.
+
+## Configuration
+
+Edit `capture_logger.lua` to configure:
+
+```lua
+M.enabled = true              -- Enable/disable capture logging
+M.log_to_hub = true          -- Send logs to hub (visible in hub logs)
+M.include_sensitive = false  -- Log API keys/tokens (CAUTION!)
+```
+
+## Usage
+
+### Building & Installing
+
+```bash
+# Package driver
+cd ~/Projects/SmartThingsEdgeDrivers
+./tools/package_driver.sh drivers/SmartThings/philips-hue
+
+# Install to hub
+smartthings edge:drivers:install
+```
+
+### Capturing Logs
+
+```bash
+# Stream logs from hub
+smartthings edge:drivers:logcat --hub-address=<HUB_IP> | tee hue_capture.log
+
+# Extract only capture logs
+grep '\[CAPTURE\]' hue_capture.log > capture_only.log
+
+# Pretty-print JSON
+jq . capture_only.log > capture_pretty.json
+```
+
+### Test Plan
+
+See **`CAPTURE_TEST_PLAN.md`** for comprehensive testing scenarios including:
+- Initial discovery and pairing
+- Device operations (lights, buttons, sensors)
+- Network interruptions and recovery
+- Error handling
+- Long-running stability tests
+
+## Performance Impact
+
+⚠️ **This instrumented driver has significant performance overhead:**
+- Every network request/response is logged
+- Every capability event is logged
+- All logs are JSON-encoded
+- High volume of log data
+
+**Do not use in production!** This is for behavior capture only.
+
+## Log Analysis
+
+After capturing logs, analyze them to:
+1. Identify all REST API endpoints used
+2. Document all SSE event types
+3. Map command→API call→event sequences
+4. Find error patterns and edge cases
+5. Generate integration test scenarios
+
+## File Summary
+
+### New Files
+- `src/capture_logger.lua` - Core logging infrastructure (467 lines)
+- `src/capture_device_wrapper.lua` - Device event wrapper (151 lines)
+- `CAPTURE_TEST_PLAN.md` - Comprehensive test plan (900+ lines)
+- `INSTRUMENTATION_README.md` - This file
+
+### Modified Files
+- `src/init.lua` - Initialize capture logging
+- `src/lunchbox/rest.lua` - REST client instrumentation
+- `src/lunchbox/sse/eventsource.lua` - SSE client instrumentation
+- `src/handlers/commands.lua` - Command logging
+- `src/handlers/lifecycle_handlers/init.lua` - Lifecycle logging
+
+## Next Steps
+
+1. **Execute Test Plan** - Follow `CAPTURE_TEST_PLAN.md` scenarios
+2. **Collect Logs** - Organize by scenario with annotations
+3. **Analyze Data** - Parse JSON, identify patterns, map behavior
+4. **Build Tests** - Create integration tests based on captured behavior
+5. **Refactor** - Simplify driver while maintaining captured behavior
+
+## Troubleshooting
+
+### Logs Not Appearing
+
+- Check `capture_logger.M.enabled = true`
+- Verify hub logs are streaming
+- Look for "[CAPTURE]" prefix in logs
+
+### Too Much Log Volume
+
+- Disable capture temporarily: `M.enabled = false`
+- Run specific scenarios in isolation
+- Filter logs by subtype
+
+### Driver Instability
+
+- Check for errors in capture modules
+- Verify JSON encoding doesn't fail
+- Add error handling in critical paths
+
+## Reverting to Normal Driver
+
+To switch back to non-instrumented driver:
+
+```bash
+git checkout main  # or your production branch
+./tools/package_driver.sh drivers/SmartThings/philips-hue
+smartthings edge:drivers:install
+```
+
+---
+
+**Branch:** `hue-instrumented-capture`  
+**Purpose:** Behavior capture for refactoring  
+**Status:** Ready for testing  
+**Created:** 2026-08-21

drivers/SmartThings/philips-hue/src/capture_device_wrapper.lua

@@ -0,0 +1,156 @@
+--[[
+  Device Wrapper for Capture Logging
+
+  This module wraps device methods to automatically capture IPC communication:
+  - emit_event: Captures all outgoing capability events
+  - online/offline: Captures device status changes
+
+  Usage:
+    Call capture_device_wrapper.wrap_driver(driver) in driver initialization
+]]
+
+local capture_logger = require "capture_logger"
+local log = require "log"
+
+local M = {}
+
+-- Store original emit_event method
+local original_emit_event = nil
+
+-- Wrapped emit_event that logs before calling original
+local function wrapped_emit_event(device, event)
+  -- CAPTURE: Log outgoing capability event
+  if event and event.capability and event.attribute then
+    capture_logger.log_capability_event(
+      device.id,
+      event.capability,
+      event.attribute.NAME or event.attribute,
+      event.attribute.value,
+      event.component or "main",
+      nil  -- state_change_id (could be added later if needed)
+    )
+  end
+
+  -- Call original emit_event
+  return original_emit_event(device, event)
+end
+
+-- Wrap device online/offline methods
+local original_try_update_metadata = nil
+
+local function wrapped_try_update_metadata(device_api, device_id, update_tbl)
+  -- Check if this is an online/offline change
+  if update_tbl and update_tbl.online ~= nil then
+    capture_logger.log_device_status(
+      device_id,
+      update_tbl.online,
+      "Metadata update"
+    )
+  end
+
+  return original_try_update_metadata(device_api, device_id, update_tbl)
+end
+
+-- Wrap create_device to capture device creation
+local original_create_device = nil
+
+local function wrapped_create_device(device_api, device_create_tbl)
+  capture_logger.log_device_create(
+    device_create_tbl.parentDeviceId or "unknown",
+    device_create_tbl
+  )
+
+  return original_create_device(device_api, device_create_tbl)
+end
+
+-- Wrap delete_device to capture device deletion
+local original_delete_device = nil
+
+local function wrapped_delete_device(device_api, device_id)
+  capture_logger.log_device_delete(device_id)
+
+  return original_delete_device(device_api, device_id)
+end
+
+-- Wrap the driver to automatically capture device events
+function M.wrap_driver(driver)
+  log.info("[CAPTURE] Wrapping driver for event capture")
+
+  -- Wrap device emit_event method
+  -- This needs to be done for all devices, so we wrap it in the driver's device_added callback
+  local original_device_added = driver.lifecycle_handlers.added
+
+  driver.lifecycle_handlers.added = function(driver, device, ...)
+    -- Wrap this device's emit_event if not already wrapped
+    if device.emit_event and device.emit_event ~= wrapped_emit_event then
+      if not original_emit_event then
+        original_emit_event = device.emit_event
+      end
+      device.emit_event = wrapped_emit_event
+    end
+
+    -- Call original added handler
+    if original_device_added then
+      return original_device_added(driver, device, ...)
+    end
+  end
+
+  -- Wrap device API methods for online/offline tracking
+  if driver.device_api and driver.device_api.try_update_metadata then
+    if not original_try_update_metadata then
+      original_try_update_metadata = driver.device_api.try_update_metadata
+    end
+    driver.device_api.try_update_metadata = wrapped_try_update_metadata
+  end
+
+  -- Wrap device creation
+  if driver.device_api and driver.device_api.create_device then
+    if not original_create_device then
+      original_create_device = driver.device_api.create_device
+    end
+    driver.device_api.create_device = wrapped_create_device
+  end
+
+  -- Wrap device deletion
+  if driver.device_api and driver.device_api.delete_device then
+    if not original_delete_device then
+      original_delete_device = driver.device_api.delete_device
+    end
+    driver.device_api.delete_device = wrapped_delete_device
+  end
+
+  log.info("[CAPTURE] Driver wrapping complete")
+end
+
+-- Wrap device set_field to capture state changes
+function M.wrap_device_set_field(device)
+  if device._capture_wrapped then
+    return  -- Already wrapped
+  end
+
+  local original_set_field = device.set_field
+
+  device.set_field = function(dev, field, value, opts)
+    -- Get old value before setting
+    local old_value = dev:get_field(field)
+
+    -- Call original
+    local result = original_set_field(dev, field, value, opts)
+
+    -- CAPTURE: Log field change
+    if old_value ~= value then
+      capture_logger.log_field_change(
+        dev.id,
+        field,
+        old_value,
+        value
+      )
+    end
+
+    return result
+  end
+
+  device._capture_wrapped = true
+end
+
+return M