Skip to content

Fix data race on AbstractMetaObjectBase's owning-loader list - #236

Open
thomasmoore-torc wants to merge 1 commit into
ros:rollingfrom
thomasmoore-torc:fix/isowned-by-race-and-createinstance-early-unlock
Open

thomasmoore-torc wants to merge 1 commit into
ros:rollingfrom
thomasmoore-torc:fix/isowned-by-race-and-createinstance-early-unlock

Conversation

@thomasmoore-torc

Copy link
Copy Markdown

Problem

AbstractMetaObjectBase::addOwningClassLoader() / removeOwningClassLoader()
/ isOwnedBy() (and the other accessors of associated_class_loaders_) take
no lock of their own. createInstance() in class_loader_core.hpp releases
getPluginBaseToFactoryMapMapMutex() before reading
factory->isOwnedBy(loader), so a concurrent addOwningClassLoader() /
removeOwningClassLoader() on another thread (e.g. from loadLibrary()'s
already-loaded branch, or from onPluginDeletion()) can race with that read —
and, since it's a std::vector, with any other concurrent read/write of the
same metaobject's owner list.

I hit this with ThreadSanitizer as a data race on std::vector<ClassLoader *>'s
push_back()/erase() (via the vector's internal reallocation). It's
straightforward to trigger by constructing many independent ClassLoader
instances for the same library concurrently — e.g. one rosbag2_cpp::Writer
per thread, each internally constructing its own pluginlib::ClassLoader for
the serialization format converter, all for the first time, simultaneously.

This is separate from #234 (loadLibrary()/unloadLibrary() overlap) and
#231 (has_unmanaged_instance_been_created_) — both already merged, and
both fix different unsynchronized globals in this same area. I checked the
current rolling HEAD before writing this: createInstance() still releases
the mutex before the isOwnedBy() checks, and meta_object.cpp's accessors
are still fully unguarded.

Fix

  • Give AbstractMetaObjectBaseImpl its own recursive_mutex guarding
    associated_class_loaders_, so every accessor (addOwningClassLoader,
    removeOwningClassLoader, isOwnedBy, isOwnedByAnybody,
    getAssociatedClassLoadersCount, getAssociatedClassLoader) is
    self-contained and safe regardless of what lock, if any, the caller happens
    to be holding.
  • Also extend createInstance()'s existing
    getPluginBaseToFactoryMapMapMutex() scope to cover the isOwnedBy()
    checks themselves: between releasing that mutex (previously done right
    after the map lookup) and reading isOwnedBy(), the factory pointer
    looked up under it could in principle be invalidated by a concurrent
    destroyMetaObjectsForLibrary() on another thread. The mutex-per-vector
    fix above closes the vector-corruption race on its own; this closes the
    separate use-after-free risk on the factory pointer itself.

Testing

Added ClassLoaderTest.threadSafetyMultipleLoadersPerLibrary to
test/utest.cpp. Unlike the existing ClassLoaderTest.threadSafety (one
ClassLoader shared by all STRESS_TEST_NUM_THREADS threads, so
isOwnedBy() is always queried with the loader that already, and
permanently, owns the metaobject), this constructs a separate
ClassLoader per thread for the same library, so construction/destruction
concurrently mutates associated_class_loaders_ while other threads
concurrently read it via isOwnedBy() in createInstance().

I verified the underlying fix (byte-for-byte equivalent transformation,
ported here to current rolling) in a downstream Bazel-based ROS 2 build
(torc-forks/rules_ros2) with
ThreadSanitizer: this specific race (the ClassLoader* vector
push_back/_M_realloc_insert/erase) is reported on every run before the
fix and gone after, alongside the class_loader#234/#231 fixes for the two
other races in this area.

I was not able to run the full colcon build --packages-select class_loader --cmake-args -DBUILD_TESTING=ON && colcon test in my environment (no full
ROS 2 workspace available), but I syntax-checked the changed files directly
with g++ -fsyntax-only -std=c++20 against this package's own headers plus
console_bridge/rcpputils (both clean), and reasoned through the new
test's logic against the existing threadSafety test's pattern. I'd
appreciate a reviewer running the full suite, including the new test, to
confirm - matching the verification style of #234/#233.

Generative AI

This fix was investigated, written, and iteratively verified (ThreadSanitizer

  • stress-testing, in the downstream build referenced above) by Claude
    (Anthropic), operating as Claude Code, at the direction of and reviewed by
    me. The regression test and this PR description were also written by Claude;
    I reviewed both before submitting.

AbstractMetaObjectBase::addOwningClassLoader()/removeOwningClassLoader()/
isOwnedBy() (and the other accessors of associated_class_loaders_) took no
lock of their own. createInstance() in class_loader_core.hpp released
getPluginBaseToFactoryMapMapMutex() before reading factory->isOwnedBy(loader),
so a concurrent addOwningClassLoader()/removeOwningClassLoader() on another
thread (e.g. from loadLibrary()'s already-loaded branch, or from
onPluginDeletion()) could race with that read - and, since it's a std::vector,
with any other concurrent read/write of the same metaobject's owner list.

Reported by ThreadSanitizer as a data race on std::vector<ClassLoader *>'s
push_back()/erase() (via vector's internal reallocation), hit by constructing
many independent ClassLoader instances for the same library concurrently -
e.g. one rosbag2_cpp::Writer per thread, each internally constructing its own
pluginlib::ClassLoader for the serialization format converter.

Give AbstractMetaObjectBaseImpl its own recursive_mutex guarding
associated_class_loaders_, so every accessor is self-contained and safe
regardless of what lock (if any) the caller happens to be holding. Also
extend createInstance()'s existing getPluginBaseToFactoryMapMapMutex() scope
to cover the isOwnedBy() checks themselves, since between releasing that
mutex and reading isOwnedBy(), the factory pointer looked up under it could
in principle be invalidated by a concurrent destroyMetaObjectsForLibrary()
on another thread.

This is separate from ros#234 (loadLibrary()/unloadLibrary() overlap) and ros#231
(has_unmanaged_instance_been_created_): those fixed different unsynchronized
globals in this same area.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@thomasmoore-torc

Copy link
Copy Markdown
Author

For reference, tracking the two related PRs mentioned above, both already merged:

This PR fixes a third, still-unaddressed race in the same area (AbstractMetaObjectBase's owning-loader list).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant