ROS Bag Pipelines (bagpipe)

Quick Start (Demo)

We provide a demo script that automatically downloads test bagfiles and runs the conversion logic for you. The script also opens the converted file in Foxglove if it is installed.

# Run the demo (downloads data -> converts -> verifies)
./run_demo.sh

Prefer to test manually? You can download the sample split-bag dataset from Google Drive here.

---

TL;DR

bash ./scripts/install.sh

# Basic conversion (Single File)
bagpipe convert /path/to/single_ros1.bag

# Series conversion (Folder Input)
bagpipe convert /path/to/ros1_bag_folder --series

# What am I running, and is it current?
bagpipe info
bagpipe update

The command used to be convert_bag. That name still works as an alias, but bagpipe is the one to use — it is not limited to conversion.

This repository provides a containerized solution for two primary tasks:

1. ROSbags Conversion: A utility to convert ROS 1 (.bag) files to ROS 2 (.mcap) format. Features include:

• Plugin System: Extendable architecture to modify data on the fly, inject timestamps, or split topics using Python plugins.
• Automatic Type Repair: Migrates custom messages without source code. Automatically fixes ROS1 vs ROS2 incompatibilities.
• Smart Schema Registry: Handles both standard ROS definitions and custom message definitions generated by plugins.
• Split-Bag Handling: Automatically detects split bags and injects /tf_static into every chunk. Every output file is self-contained.
• Crash-Safe: Implements graceful signal handling. Hit Ctrl+C anytime, and your MCAP file will still be valid and playable.
• Zero Dependencies: Dockerized solution. No need to install ROS 1 or build custom message packages locally.

2. ROS Noetic Development: A persistent environment for developing, building, and executing ROS 1 (Noetic) packages.

Prerequisites

• Docker: Ensure Docker and Docker Compose are installed.
  • basic installaiton: https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository
• Permissions: The current user must have permission to run Docker commands (e.g., be in the docker group).
  • post-installation steps: https://docs.docker.com/engine/install/linux-postinstall

---

Part 1: Bag Converter

The converter runs as an ephemeral container. It mounts the target data directory, processes the files, and terminates.

Installation

Run the provided installation script. It links the bagpipe command into ~/.local/bin, installs shell completion, and pulls the docker image.

./scripts/install.sh

Note: Ensure ~/.local/bin is in your system $PATH. The installer adds it to ~/.bashrc if it is missing.

Commands

bagpipe can be executed from any location. Tab completion covers the command names, the flags of each command, and .bag files and folders.

Command	Purpose
bagpipe convert <input> [opts]	Convert ROS 1 .bag files to ROS 2 .mcap
bagpipe plugins	List available plugins and whether they are configured
bagpipe shell	Open a shell in the container, in the current directory
bagpipe info	Version, paths, image digest and update status
bagpipe update	Pull the latest code and container image
bagpipe help	Command list

run <pipeline> <input> and inspect <bag> are reserved for the general pipeline work described in the roadmap; they are recognized but not yet implemented.

Usage

1. Single File Conversion

Converts a single .bag file. The output .mcap file is created in the same directory as the input.

bagpipe convert /path/to/recording.bag

2. Series Conversion (Split Bags)

Use the --series flag when processing split bag files (e.g., _0.bag, _1.bag). This mode enables Static TF Injection (ensuring all parts have TF data) and generates metadata.yaml for seamless playback.

Note: You must pass the folder containing the sequence, not individual files.

# Process all bags in a folder
bagpipe convert /path/to/data_folder --series

---

Staying up to date

bagpipe update is the single update action. It matters that it is a single action, because the tool has two halves that must move together: the container image, and the Python code in src/, which is bind-mounted from this checkout at run time. Pulling only the image leaves you running old code.

bagpipe update      # git pull --ff-only + docker compose pull + relink

When your checkout falls behind main, invocations print a one-line notice:

! bagpipe is out of date (3 new commits are available on github/main).
  update with: bagpipe update

The check is designed so that the invocation which pays the network cost is never the invocation that prints the warning:

• The foreground work is reading one small cache file plus a git rev-parse — around 5 ms, no network. For scale, the docker compose run round trip that follows it is roughly 350 ms.
• The comparison against the remote runs at most once every 24 h, in a detached background job that nothing waits on. It writes a pre-rendered message that a later invocation displays.
• Being offline, or having an SSH key that needs a passphrase, is a silent no-op — the background job uses BatchMode and a timeout, so it can never hang or prompt.

Variable	Effect
BAGPIPE_NO_UPDATE_CHECK=1	Disable the notice entirely
BAGPIPE_CHECK_INTERVAL=<sec>	Change the 24 h check interval
BAGPIPE_UPSTREAM=<remote>	Compare against a specific git remote
BAGPIPE_IMAGE_TAG=<tag>	Run a specific container image tag
BAGPIPE_DEBUG=1	Print the computed mounts and container command

---

Plugin System

The converter features a modular plugin architecture allowing users to manipulate messages during the conversion process (e.g., parsing raw strings, debayering images, or anonymizing data).

There are two plugin layers:

• System plugins (always on): loaded from src/system_plugins/, configured in src/system_plugins.yaml, intended for compatibility and safety.
• User plugins (optional): loaded from src/plugins/ when --with-plugins is used, intended for project-specific transformations.

Architecture

The system uses a hook-based architecture:

1. System Loader: plugin_manager.py scans src/system_plugins/ and loads system plugins by default.
2. User Loader: plugin_manager.py scans src/plugins/ for optional user plugins.
3. Configuration: It reads src/system_plugins.yaml for system plugins and src/plugins.yaml for user plugins.
4. Execution: System plugins run first (pre-conversion guardrails), then user plugins run on converted ROS2 messages.

Configuration

Plugins are managed via src/plugins.yaml. You can pass arbitrary parameters to plugins here.

plugins:
  # Example: Image processing
  DebayerPlugin:
    target_topic: "camera_front"
    jpeg_quality: 95

  # Example: Custom parsing logic
  RobustelFixPlugin:
    input_filter: "robustel"
    output_topic: "/robustel/parsed"

How to Develop a Plugin (Click to Expand)

You can add custom logic by adding a Python script to the src/plugins/ directory.

1. Reference Example See src/plugins/debayer.py or src/plugins/robustel_fix.py for working examples.

2. Plugin Signature All plugins must inherit from BasePlugin. The process method must return a list of emissions and a boolean modification flag.

from typing import Any, List, Tuple, Optional
from plugin_manager import BasePlugin

class MyPlugin(BasePlugin):
    def process(self, topic: str, msg: Any, msg_type: str, timestamp: int) -> Tuple[List[Tuple[str, Any, str, Optional[str]]], bool]:
        """
        Args:
            timestamp: Nanoseconds from the bag file.
            
        Returns:
            Tuple(Emissions_List, Modified_Flag)
            
            Emissions_List format: 
            [ (Topic, Message, Type_String, Definition_Override) ]
        """
        
        # Access config from YAML
        my_param = self.config.get('my_param', 'default')

        # Example: Pass through unchanged
        return [(topic, msg, msg_type, None)], False

3. Hot-Reloading Because the src folder is mounted into the container, you do not need to rebuild the Docker image to test new plugins. Simply edit the Python file and run bagpipe convert. Use bagpipe plugins to confirm your class was discovered and picked up its YAML configuration.

---

Advanced Configuration

bagpipe convert forwards every argument to the internal Python converter verbatim, so bagpipe convert --help is always the authoritative option list:

• --out-dir <path>: Forces a specific output directory.
• --series: Treats the input folder as a split sequence.
• --split-size <size>: Splits output files by size (e.g. 3G, 500M).
• --with-plugins: Enables the plugin system (reads src/plugins.yaml).
• --dry-run: Validates input files, write permissions, and configuration without processing data.
• --skip-topics <topic1> <topic2> ...: A list of topics to exclude from the conversion (blacklist).

How paths reach the container (Click to Expand)

Host paths are bind-mounted into the container at their own path, so a file at /media/disk/rec.bag on the host is at /media/disk/rec.bag inside the container too. Two things follow from that:

1. Arguments are never rewritten. The wrapper passes "$@" through untouched, so it does not need to know the converter's flag list — adding a new flag to src/convert.py requires no wrapper change. (The previous wrapper relativized every path against a single /data mount, which meant hardcoding which flags take values.)
2. Reported paths are real. The conversion summary prints host paths you can copy and paste, rather than container-internal /data/... paths.

The wrapper decides what to mount by taking the arguments that resolve to something existing on disk, mapping each to its nearest existing directory, and collapsing nested paths. An argument whose path has no existing component at all is not a path — this is what stops topic names like /tf_static from being treated as mount points. Mount targets that would shadow the container's own filesystem are refused rather than silently breaking the container.

Run any command with BAGPIPE_DEBUG=1 to see the computed mounts.

Feature: Auto-Generating ROS 2 Message Definitions (Click to Expand)

The conversion tool embeds message definitions directly into the .mcap file. Modern visualization tools like Foxglove Studio read these embedded schemas automatically.

However, if you want to replay the bag using CLI tools (ros2 bag play) or inspect topics (ros2 topic echo), your local ROS 2 environment needs the compiled message packages installed.

Usage:

1. Run the extractor (in your ROS 2 environment):

python3 additional/extract_mcap_msgs.py /path/to/my_data.mcap --out-dir src/

2. Build the packages:

colcon build
source install/setup.bash

3. Check the packages:

ros2 interface list | grep <YourCustomMessage>

---

Part 2: ROS 1 Development Environment (Click to Expand)

The ros_dev service provides a full ROS Noetic desktop environment with GUI support.

Configuration

Map your host directories to the container by editing docker-compose.yml:

Host Path	Container Path	Description
~/repos	/home/dev/repos	Source code repositories.
./catkin_ws	/home/dev/catkin_ws	The active Catkin workspace.

Workflow

1. Start the Service:

docker compose up -d ros_dev

2. Access the Shell:

docker exec -it ros_pet_container bash

3. GUI Visualization: If your host supports X11 forwarding, you can run GUI tools (rviz) directly.

---

Troubleshooting

• bagpipe: command not found: Add ~/.local/bin to your $PATH, or re-run ./scripts/install.sh.
• refusing to mount <path>: the input or output path resolves to a directory that would shadow the container's own filesystem (e.g. /usr/lib, or a bare /home). Pass a more specific path.
• Tab completion not working: it is installed to ~/.local/share/bash-completion/completions/; start a new shell, and make sure the bash-completion package is installed.
• Changes to code not appearing: The src folder is mounted. Changes apply immediately. If you add system dependencies (pip/apt), run docker compose build converter.
• Custom Messages not showing in Foxglove: Ensure the plugin emits a standard type (e.g., std_msgs/String) or provides a valid definition override in the return tuple.

---

Improvement Proposals (TODO) (Click to Expand)

Pending Improvements

• bagpipe run <pipeline>: Named, configurable pipelines over bags, so processing is not tied to the conversion step. The verb is reserved in the dispatcher and the wrapper no longer rewrites arguments, so this needs only a Python-side pipeline runner.
• bagpipe inspect <bag>: Summarize topics, types, message counts and duration without converting. Also reserved in the dispatcher.
• MCAP-to-MCAP Tooling: Generalize the architecture to support MCAP-to-MCAP manipulation. This would allow using the plugin system (filtering, anonymization) on native ROS 2 data, not just during conversion.

Completed

• Unified CLI: bagpipe with subcommands, tab completion, and convert_bag kept as a compatibility alias.

• Self-description and updates: bagpipe info and bagpipe update, plus a background-refreshed "out of date" notice.

• Identity path mounts: host paths mount at their own paths, removing the argument-rewriting layer that was coupled to the converter's flag list.

• Auto-splitting: Series conversion with static TF injection is implemented.

• Plugin System V2: Parametric configuration, 1-to-N message expansion, and timestamp injection.

• Topic Filtering: Added --skip-topics argument to blacklist unwanted data.

• Automated Image Publishing: CI/CD scripts (scripts/build-image.sh, scripts/push-image.sh) are in place.

• Pre-Flight Checks: Implement a --dry-run mode to validate paths and disk space.

• UX: Added tqdm progress bars and detailed summary reports.

---

Maintainer Guide

To update the system dependencies (Dockerfile):

1. Login: echo $GITHUB_TOKEN | docker login ghcr.io -u USER --password-stdin
2. Build: ./scripts/build-image.sh
3. Push: ./scripts/push-image.sh

Bump the VERSION file when the user-facing behavior of the tool changes; it is what bagpipe info and bagpipe --version report.

The update notice compares the local checkout against the main branch of the git remote that the current branch tracks (github by default here). Pushing to that remote is what makes other users see the notice, so land wrapper changes there rather than only on a fork.