Fix rmw_fastrtps SHM crash on Windows with secured large messages (Issue #561)

[Aaravanand00]

Description

On Windows, FastDDS uses Shared Memory (SHM) transport by default for intra-host communication. When DDS-Security is enabled, the security overhead causes the SHM segment to overflow when sending large messages (e.g. UnboundedSequences), resulting in an Access Violation crash (0xC0000005) in the publisher process. Since the publisher crashes, the subscriber waits indefinitely and the test times out.

This PR fixes the crash by supplying a FastDDS XML profile (fastdds_no_shm.xml) that disables SHM and forces UDPv4 transport with enlarged send/receive buffers (1 MB) to accommodate the security-layer overhead on large messages.

Key changes :-

• Added test_security_files/fastdds_no_shm.xml: FastDDS profile that disables built-in SHM and uses only UDPv4 transport.
• Updated test_secure_publisher_subscriber.py.in: Injects FASTRTPS_DEFAULT_PROFILES_FILE env var for rmw_fastrtps_cpp / rmw_fastrtps_dynamic_cpp processes.
• Updated CMakeLists.txt: Computes the native path to the XML profile on WIN32; set to empty string on Linux/macOS so there is zero behavioural change on non-Windows platforms.

Fixes #561

Is this user-facing behavior change?

No. This is a test infrastructure fix. The actual security tests remain unchanged no tests are skipped or removed.

Did you use Generative AI?

Yes, Ai Agent was used to assist in root cause analysis and implementation.

Additional Information

The fix applies only when WIN32 is defined at CMake configure time, so Linux and macOS CI is completely unaffected.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[Aaravanand00]

Hi @fujitatomoya looking into issue #561 the publisher was crashing on Windows (exit code 0xC0000005) because FastDDS's default Shared Memory transport overflows when handling large encrypted messages. Fixed it by injecting a FastDDS XML profile that switches to UDP transport only applies on Windows, no tests skipped, Linux/macOS completely unaffected. Please let me know if any changes was needed....

[Aaravanand00]

@fujitatomoya ptal

[fujitatomoya]

@Aaravanand00 thanks for creating PR.

a couple of comments and did you actually reproduce the issue on windows and check if this PR addresses the issue?

CC: @MiguelCompany

[MiguelCompany]

I'd like to note that issue #561 is another instance of Windows process killed with SIGTERM does not correctly clean up resources.

If we make every run of a test involving a Fast DDS rmw call fastdds shm clean before actually running the test, we would probably fix all the issues of this kind.

[Aaravanand00]

@fujitatomoya @MiguelCompany

Updated the pr to remove the xml profile file completely and simplify the fix by directly setting FASTDDS_BUILTIN_TRANSPORTS=UDPv4 in the test environment for rmw_fastrtps_cpp. ptal

BTW I recently introduced myself in the ROS community and was recommended by VMB to join the ROS Issue Triage ("Waffle") meeting to help with triaging.
Will you also be attending the upcoming Waffle meeting???

[MiguelCompany]

The reason for the crash is not due to the large messages, but due to accumulating corrupted files because processes are being killed with SIGTERM. It just happens on those messages because they are the last ones being tested.

Suggested change

	# On Windows, FastDDS SHM transport causes an Access Violation (0xC0000005) when
	# sending large messages (e.g. UnboundedSequences) with security enabled.
	# Force UDPv4 transport to prevent the crash. (Issue #561)
	# On Windows, FastDDS SHM transport may cause an Access Violation (0xC0000005) when
	# previous tests kill a process using SIGTERM. (Issue #561)
	# Force UDPv4 transport to prevent the crash till we write an isolation plugin that cleans
	# those files before a test is run.