Read Protobuf and MessagePack bodies

CHANGELOG.md

@@ -11,6 +11,7 @@ board can be cross-checked against what actually shipped.
 
 ### Added
 
+- Protobuf and MessagePack bodies are readable. MessagePack needs nothing and is shown as indented text, and a Protobuf body with no schema is shown by field number, guessed the way `protoc --decode_raw` guesses. `FlutwireProtobufDecoder` takes the descriptor set that `protoc` writes and shows the fields by name and type, with enums by name, maps as objects and packed runs as lists. A gRPC or gRPC-web call finds its message types from the method in its address, lists the messages of a stream and shows the trailers. The decoders run when a body is opened, so the capture is untouched. The example app has a body of each (M22)
 - WebSockets and server-sent event streams. A WebSocket is one call that stays open, and it lists its messages with the ping, pong and close frames, which way each went, how big it was and when it crossed, compressed messages included. An event stream lists its events as they complete, with their names and ids, and keeps the raw stream as the body of the response. Both get a Messages or Events tab, a badge in the list, a line in the overview, entries in the HAR and text exports, and their messages are kept on disk with the call. The latest 1000 messages are kept, with a megabyte of payload at most, and the tab says how many came before (M16)
 - Any call can be opened in an editor, changed and sent again. The editor starts from the method, address, header fields and body that were recorded. A value the redaction masked was never kept, so it is flagged and not sent unless it is typed again. What goes out is recorded as a new call that names the one it repeats, marked in the list and linked from its overview. It needs `dart:io`, so the web does not offer it (M15)
 - A line above the list that says when Flutwire may be missing calls, with the reason and what to change: a client or adapter that does not use `dart:io`, an adapter that ran in another isolate, or something that replaced the capture's `HttpOverrides`. It stays out of sight until Flutwire has seen one of them happen, and it names the client or the isolate. `Flutwire.coverage` lists the notes, and an adapter that stands down for the capture reports through `Flutwire.watchStandDown`. The example app has a probe for each case (M24)

packages/flutwire/README.md

@@ -55,6 +55,23 @@ A WebSocket opened through `dart:io` is one call that stays open. Its Messages t
 
 `dart:io` makes one client for all WebSockets the first time one is opened, so a WebSocket opened before `Flutwire.init` is not seen: call `init` first thing in `main`. On the web the browser opens WebSockets itself, and none is recorded there.
 
+## Protobuf and MessagePack
+
+A MessagePack body is shown as indented text with no setup, and a Protobuf body is shown by field number. To see Protobuf with names and types, give the decoder the descriptor set that `protoc` writes with `--descriptor_set_out=api.desc --include_imports`:
+
+```dart
+WidgetsFlutterBinding.ensureInitialized();
+final descriptors = (await rootBundle.load('assets/api.desc')).buffer.asUint8List();
+
+await Flutwire.init(
+  config: FlutwireConfig(
+    decoders: [FlutwireProtobufDecoder(descriptorSet: descriptors)],
+  ),
+);
+```
+
+A gRPC or gRPC-web call finds its message types from the method in its address. For any other Protobuf body, say which message it is with `messageType: (call, {required isRequest}) => 'demo.Person'`, and a body whose type is not known is shown by field number. The decoders run when a body is opened, so what the app sends and receives is not touched.
+
 ## Sending a call again
 
 Open a call and tap the replay button in the top bar. The method, the address, the header fields and the body can all be changed, and Send makes the request. What goes out is recorded as a new call that names the one it repeats.
@@ -76,7 +93,7 @@ await Flutwire.init(
 );
 ```
 
-Authorization headers and the values of cookies are masked by default; the names and attributes of cookies stay readable. A masked value is replaced when the call is recorded, so it never reaches memory, disk, a notification or an export. The other options are documented on `FlutwireConfig`: how much of a body is kept, where the history is stored, what the notification lists, the language, the look, and decoders for formats the inspector does not read, such as Protobuf.
+Authorization headers and the values of cookies are masked by default; the names and attributes of cookies stay readable. A masked value is replaced when the call is recorded, so it never reaches memory, disk, a notification or an export. The other options are documented on `FlutwireConfig`: how much of a body is kept, where the history is stored, what the notification lists, the language, the look, and decoders for formats the inspector does not read on its own.
 
 ## Clients that bypass `dart:io`
 

packages/flutwire/example/lib/main.dart

@@ -176,6 +176,8 @@ final List<(String, List<Probe>)> groups = [
               client.get(Uri.parse('$_bin/cookies/set?theme=dark&session=abc')),
         ),
       ),
+      Probe('MessagePack', () => _canned(_messagePack, 'application/msgpack')),
+      Probe('Protobuf', () => _canned(_protobuf, 'application/x-protobuf')),
     ],
   ),
   (
@@ -215,6 +217,33 @@ final List<(String, List<Probe>)> groups = [
   ),
 ];
 
+/// `{"club": "Orion", "members": 3, "tags": ["a", "b"], "founded": 1999.5}`.
+const List<int> _messagePack = [
+  0x84, //
+  0xa4, 0x63, 0x6c, 0x75, 0x62, 0xa5, 0x4f, 0x72, 0x69, 0x6f, 0x6e,
+  0xa7, 0x6d, 0x65, 0x6d, 0x62, 0x65, 0x72, 0x73, 0x03,
+  0xa4, 0x74, 0x61, 0x67, 0x73, 0x92, 0xa1, 0x61, 0xa1, 0x62,
+  0xa7, 0x66, 0x6f, 0x75, 0x6e, 0x64, 0x65, 0x64,
+  0xcb, 0x40, 0x9f, 0x3e, 0x00, 0x00, 0x00, 0x00, 0x00,
+];
+
+/// A message of field 1 as "Orion", field 2 as 3 and field 3 as a message of
+/// its own that holds "Vega" in field 1.
+const List<int> _protobuf = [
+  0x0a, 0x05, 0x4f, 0x72, 0x69, 0x6f, 0x6e, //
+  0x10, 0x03,
+  0x1a, 0x06, 0x0a, 0x04, 0x56, 0x65, 0x67, 0x61,
+];
+
+/// Answers with [body] and the media type [type], so that the viewer has
+/// something to decode. The client records the call itself.
+Future<void> _canned(List<int> body, String type) => _safe(
+  () => FlutwireClient(
+    _Canned(body, type),
+    force: true,
+  ).get(Uri.parse('$_todos/club')),
+);
+
 /// Opens a WebSocket to an echo server, sends a text and a binary message, and
 /// closes it, so the inspector has a conversation to list. `dart:io` makes one
 /// client for every WebSocket the first time it opens one, so this works
@@ -294,6 +323,22 @@ Future<void> _fromAnotherIsolate() async {
   }
 }
 
+/// A client that answers every request with the same body.
+class _Canned extends http.BaseClient {
+  _Canned(this.body, this.type);
+
+  final List<int> body;
+  final String type;
+
+  @override
+  Future<http.StreamedResponse> send(http.BaseRequest request) async =>
+      http.StreamedResponse(
+        Stream.value(body),
+        200,
+        headers: {'content-type': type},
+      );
+}
+
 /// A client with a transport of its own, as `cronet_http` has: it never
 /// reaches dart:io, so the capture cannot see what it sends.
 class _OffTheWire extends http.BaseClient {

packages/flutwire/lib/flutwire.dart

@@ -8,6 +8,8 @@ export 'src/capture/recorder.dart' show FlutwireCallRecorder;
 export 'src/capture/redaction.dart';
 export 'src/config.dart';
 export 'src/coverage.dart' show FlutwireCoverageNote, FlutwireGap;
+export 'src/decode/messagepack.dart' show FlutwireMessagePackDecoder;
+export 'src/decode/protobuf_decoder.dart' show FlutwireProtobufDecoder;
 export 'src/engine.dart' show FlutwireStandDown;
 export 'src/export/export.dart' show FlutwireExportFormat, callToJson;
 export 'src/flutwire.dart' show Flutwire;

packages/flutwire/lib/src/capture/recorder.dart

@@ -186,7 +186,7 @@ final class FlutwireCallRecorder {
         : FlutwireConversationKind.webSocket;
     final cap = _host.config.maxBodyBytes;
     var bytes = payload is Uint8List ? payload : Uint8List.fromList(payload);
-    if (bytes.length > cap) bytes = Uint8List.fromList(bytes.sublist(0, cap));
+    if (bytes.length > cap) bytes = bytes.sublist(0, cap);
     if (kind == FlutwireMessageKind.text || kind == FlutwireMessageKind.event) {
       bytes = _host.config.redaction.applyToBody(bytes, contentType: null);
     }

packages/flutwire/lib/src/capture/wire/wire_io.dart

@@ -5,9 +5,9 @@ import 'dart:io';
 import '../../engine.dart';
 import '../../model/call.dart';
 import '../../model/headers.dart';
-import '../messages/websocket_reader.dart';
+import '../messages/websocket_reader_io.dart';
 import '../recorder.dart';
-import 'wire_socket.dart';
+import 'wire_socket_io.dart';
 
 /// Records every call made through `dart:io`, with no setup in the client.
 ///

packages/flutwire/lib/src/capture/wire/wire_socket.dart → packages/flutwire/lib/src/capture/wire/wire_socket_io.dart

@@ -4,7 +4,7 @@ import 'dart:io';
 import 'dart:typed_data';
 
 import '../../model/message.dart';
-import '../messages/websocket_reader.dart';
+import '../messages/websocket_reader_io.dart';
 import '../recorder.dart';
 
 /// A socket that was an HTTP connection and became a WebSocket. It hands the

packages/flutwire/lib/src/decode/built_in.dart

@@ -0,0 +1,11 @@
+import '../config.dart';
+import 'messagepack.dart';
+import 'protobuf_decoder.dart';
+
+/// The decoders every inspector has, which the viewer asks after the ones the
+/// app registered. Neither needs a schema: MessagePack describes itself, and a
+/// Protobuf body with none is shown by field number.
+const List<FlutwireBodyDecoder> builtInDecoders = [
+  FlutwireMessagePackDecoder(),
+  FlutwireProtobufDecoder.raw(),
+];

packages/flutwire/lib/src/decode/decoded_text.dart

@@ -0,0 +1,119 @@
+import 'dart:convert';
+import 'dart:typed_data';
+
+/// Binary content in a value a decoder read. It is written as hex, the way the
+/// inspector writes bytes everywhere else.
+final class DecodedBytes {
+  /// Wraps [bytes].
+  const DecodedBytes(this.bytes);
+
+  /// The bytes.
+  final Uint8List bytes;
+
+  /// How many bytes are written out before the text says there are more.
+  static const int shown = 64;
+
+  @override
+  String toString() {
+    final head = bytes.length <= shown ? bytes : bytes.sublist(0, shown);
+    final hex = [
+      for (final byte in head) byte.toRadixString(16).padLeft(2, '0'),
+    ].join(' ');
+    return bytes.length > shown ? '$hex … (${bytes.length} bytes)' : hex;
+  }
+}
+
+/// Writes a value a decoder read as indented text that reads like JSON.
+///
+/// It takes `null`, `bool`, `int`, `BigInt`, `double`, `String`, `DateTime`,
+/// [DecodedBytes], lists, and maps with string keys, and keeps the order of a
+/// map. It is not JSON where JSON cannot say it: a number too big for a double
+/// is written out in full, `NaN` and the infinities are strings, and bytes are
+/// a string of hex.
+String renderDecoded(Object? value) {
+  final out = StringBuffer();
+  _write(out, value, 0);
+  return out.toString();
+}
+
+/// How deep a value is followed before the text gives up and writes `…`.
+const int _maxDepth = 64;
+
+/// How long a list of plain values may be and still go on one line.
+const int _inlineWidth = 72;
+
+void _write(StringBuffer out, Object? value, int depth) {
+  if (depth > _maxDepth) {
+    out.write('"…"');
+    return;
+  }
+  switch (value) {
+    case null:
+      out.write('null');
+    case final bool flag:
+      out.write(flag);
+    case final int number:
+      out.write(number);
+    case final BigInt number:
+      out.write(number);
+    case final double number:
+      out.write(number.isFinite ? number : jsonEncode(number.toString()));
+    case final String text:
+      out.write(jsonEncode(text));
+    case final DateTime time:
+      out.write(jsonEncode(time.toUtc().toIso8601String()));
+    case final DecodedBytes bytes:
+      out.write(jsonEncode(bytes.toString()));
+    case final List<Object?> list:
+      _writeList(out, list, depth);
+    case final Map<String, Object?> map:
+      _writeMap(out, map, depth);
+    default:
+      out.write(jsonEncode('$value'));
+  }
+}
+
+bool _isPlain(Object? value) => value is! List<Object?> && value is! Map;
+
+void _writeList(StringBuffer out, List<Object?> list, int depth) {
+  if (list.isEmpty) {
+    out.write('[]');
+    return;
+  }
+  if (list.every(_isPlain)) {
+    final line = StringBuffer('[');
+    for (var i = 0; i < list.length; i++) {
+      if (i > 0) line.write(', ');
+      _write(line, list[i], depth + 1);
+    }
+    line.write(']');
+    if (line.length <= _inlineWidth) {
+      out.write(line);
+      return;
+    }
+  }
+  final inner = '  ' * (depth + 1);
+  out.writeln('[');
+  for (var i = 0; i < list.length; i++) {
+    out.write(inner);
+    _write(out, list[i], depth + 1);
+    out.writeln(i == list.length - 1 ? '' : ',');
+  }
+  out.write('${'  ' * depth}]');
+}
+
+void _writeMap(StringBuffer out, Map<String, Object?> map, int depth) {
+  if (map.isEmpty) {
+    out.write('{}');
+    return;
+  }
+  final inner = '  ' * (depth + 1);
+  out.writeln('{');
+  var left = map.length;
+  for (final entry in map.entries) {
+    out.write('$inner${jsonEncode(entry.key)}: ');
+    _write(out, entry.value, depth + 1);
+    out.writeln(--left == 0 ? '' : ',');
+  }
+  out.write('${'  ' * depth}}');
+}