Fix data race on AbstractMetaObjectBase's owning-loader list - #236
Open
thomasmoore-torc wants to merge 1 commit into
Open
thomasmoore-torc wants to merge 1 commit into
thomasmoore-torc wants to merge 1 commit into
Conversation
AbstractMetaObjectBase::addOwningClassLoader()/removeOwningClassLoader()/ isOwnedBy() (and the other accessors of associated_class_loaders_) took no lock of their own. createInstance() in class_loader_core.hpp released getPluginBaseToFactoryMapMapMutex() before reading factory->isOwnedBy(loader), so a concurrent addOwningClassLoader()/removeOwningClassLoader() on another thread (e.g. from loadLibrary()'s already-loaded branch, or from onPluginDeletion()) could race with that read - and, since it's a std::vector, with any other concurrent read/write of the same metaobject's owner list. Reported by ThreadSanitizer as a data race on std::vector<ClassLoader *>'s push_back()/erase() (via vector's internal reallocation), hit by constructing many independent ClassLoader instances for the same library concurrently - e.g. one rosbag2_cpp::Writer per thread, each internally constructing its own pluginlib::ClassLoader for the serialization format converter. Give AbstractMetaObjectBaseImpl its own recursive_mutex guarding associated_class_loaders_, so every accessor is self-contained and safe regardless of what lock (if any) the caller happens to be holding. Also extend createInstance()'s existing getPluginBaseToFactoryMapMapMutex() scope to cover the isOwnedBy() checks themselves, since between releasing that mutex and reading isOwnedBy(), the factory pointer looked up under it could in principle be invalidated by a concurrent destroyMetaObjectsForLibrary() on another thread. This is separate from ros#234 (loadLibrary()/unloadLibrary() overlap) and ros#231 (has_unmanaged_instance_been_created_): those fixed different unsynchronized globals in this same area. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Author
|
For reference, tracking the two related PRs mentioned above, both already merged:
This PR fixes a third, still-unaddressed race in the same area (AbstractMetaObjectBase's owning-loader list). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
AbstractMetaObjectBase::addOwningClassLoader()/removeOwningClassLoader()/
isOwnedBy()(and the other accessors ofassociated_class_loaders_) takeno lock of their own.
createInstance()inclass_loader_core.hppreleasesgetPluginBaseToFactoryMapMapMutex()before readingfactory->isOwnedBy(loader), so a concurrentaddOwningClassLoader()/removeOwningClassLoader()on another thread (e.g. fromloadLibrary()'salready-loaded branch, or from
onPluginDeletion()) can race with that read —and, since it's a
std::vector, with any other concurrent read/write of thesame metaobject's owner list.
I hit this with ThreadSanitizer as a data race on
std::vector<ClassLoader *>'spush_back()/erase()(via the vector's internal reallocation). It'sstraightforward to trigger by constructing many independent
ClassLoaderinstances for the same library concurrently — e.g. one
rosbag2_cpp::Writerper thread, each internally constructing its own
pluginlib::ClassLoaderforthe serialization format converter, all for the first time, simultaneously.
This is separate from #234 (
loadLibrary()/unloadLibrary()overlap) and#231 (
has_unmanaged_instance_been_created_) — both already merged, andboth fix different unsynchronized globals in this same area. I checked the
current
rollingHEAD before writing this:createInstance()still releasesthe mutex before the
isOwnedBy()checks, andmeta_object.cpp's accessorsare still fully unguarded.
Fix
AbstractMetaObjectBaseImplits ownrecursive_mutexguardingassociated_class_loaders_, so every accessor (addOwningClassLoader,removeOwningClassLoader,isOwnedBy,isOwnedByAnybody,getAssociatedClassLoadersCount,getAssociatedClassLoader) isself-contained and safe regardless of what lock, if any, the caller happens
to be holding.
createInstance()'s existinggetPluginBaseToFactoryMapMapMutex()scope to cover theisOwnedBy()checks themselves: between releasing that mutex (previously done right
after the map lookup) and reading
isOwnedBy(), thefactorypointerlooked up under it could in principle be invalidated by a concurrent
destroyMetaObjectsForLibrary()on another thread. The mutex-per-vectorfix above closes the vector-corruption race on its own; this closes the
separate use-after-free risk on the
factorypointer itself.Testing
Added
ClassLoaderTest.threadSafetyMultipleLoadersPerLibrarytotest/utest.cpp. Unlike the existingClassLoaderTest.threadSafety(oneClassLoadershared by allSTRESS_TEST_NUM_THREADSthreads, soisOwnedBy()is always queried with the loader that already, andpermanently, owns the metaobject), this constructs a separate
ClassLoaderper thread for the same library, so construction/destructionconcurrently mutates
associated_class_loaders_while other threadsconcurrently read it via
isOwnedBy()increateInstance().I verified the underlying fix (byte-for-byte equivalent transformation,
ported here to current
rolling) in a downstream Bazel-based ROS 2 build(torc-forks/rules_ros2) with
ThreadSanitizer: this specific race (the
ClassLoader*vectorpush_back/_M_realloc_insert/erase) is reported on every run before thefix and gone after, alongside the class_loader#234/#231 fixes for the two
other races in this area.
I was not able to run the full
colcon build --packages-select class_loader --cmake-args -DBUILD_TESTING=ON && colcon testin my environment (no fullROS 2 workspace available), but I syntax-checked the changed files directly
with
g++ -fsyntax-only -std=c++20against this package's own headers plusconsole_bridge/rcpputils(both clean), and reasoned through the newtest's logic against the existing
threadSafetytest's pattern. I'dappreciate a reviewer running the full suite, including the new test, to
confirm - matching the verification style of #234/#233.
Generative AI
This fix was investigated, written, and iteratively verified (ThreadSanitizer
(Anthropic), operating as Claude Code, at the direction of and reviewed by
me. The regression test and this PR description were also written by Claude;
I reviewed both before submitting.