How to Measure Network Throughput in a Multi-Robot Scenario

[MikeDegany]

Describe the the feature you would like

I've connected multiple TurtleBots to a central computer to do mapping and navigation. To measure data throughput under various scenarios—like when a single robot is connected, or multiple robots are in operation—I'm looking for a tool or command within ROS or Linux that can show the speed and volume of data transfer between the robots, or between the robots and the central computer. Any suggestions?

Motivation and impact

My apologies if this question isn't strictly about TurtleBots or is more general regarding ROS2. However, asking here ensures we can account for any specialized tools or limitations specific to TurtleBots.
This tool will be very helpful for networks with many robots, where having a strong network is really important.

Other notes

No response

[Jayden-F]

Hey @MikeDegany, are you noticing issues with the network and robots? I experienced quadratic network traffic (quadratic with the number of nodes on the network) with cyclone dds and namespacing multiple robots.

[MikeDegany]

Hey @MikeDegany, are you noticing issues with the network and robots? I experienced quadratic network traffic (quadratic with the number of nodes on the network) with cyclone dds and namespacing multiple robots.

@Jayden-F
I've also had many network problems, but somehow, I managed to get my three turtlebots to do slam and navigation at the same time. How can you check if your network traffic grows quadratically? When we say "nodes" here, do we mean robots?"

[hilary-luo]

Hey @MikeDegany. I don't have anything Turtlebot4 specific to recommend. eProsima does have a network monitoring tool that you could try but it does require building Fast DDS from source. I haven't tried this out so if you do try it let me know how it goes. There was a ROScon talk about it: http://download.ros.org/downloads/roscon/2022/ROS%202%20network%20monitoring.pdf (I imagine there is also a video record of the talk itself).

Beyond ROS, I think there are a few other tools available, just not ones I am particularly familiar with.

That being said, keep an eye out for when we release support for discovery server with multiple robots, that should help optimize your network. (I am hoping it will be soon but can't promise any dates). And try the latest humble firmware!

[MikeDegany]

Dear @hilary-luo, that's true. eProsima provides a tool named FastDDS Monitor, but you must install the ROS2 from source and build it with -DFASTDDS_STATISTICS on. It didn't work really smooth for me ( I haven't had enough time to address the issue tbh), so I went for Vulcanexus which is a ROS2 with analyzing tools activated inherently. I could setup a talker-chatter-listener on the PC and see the graphs on FastDDS Monitor, however it stopped drawing throughput or latency graphs when I increased the talker node frequency a lot. Btw, I couldn't see the same analysis for topics published by the robots which means the ROS2 on Turtlebots must be build with -DFASTDDS_STATISTICS=ON.
So, for now I am trying to make some network analyzing nodes and topics and measure some network metrics on published messages. I you have any idea or know any similar work out there, I would genuinely appreciate it if you could let me know about them.

[hilary-luo]

@MikeDegany It's been a while so I am not sure if this is still on your radar but over the past few months, I have found that wireshark / tshark have been my goto tools for monitoring this sort of network traffic. I use tshark on my robots to capture and then pull it off and analyze it later. On my PC I use wireshark live. Both during live recordings in wireshark and when later analyzing the tshark captures in wireshark, I am using a few different tools. I am primarily using the Statistics -> I/O Graphs option to display graphs of the traffic. You can add different lines to the graph to display different things. This allows me to track number of packets as well as bandwidth and then I use the display filters on these graphs to analyze how much of that traffic is going to each address as well as the type of messages being sent. You can go so far as to break this down using GUIDs and/or ports to pinpoint particular participants. I also use the IPv4 statistics to look at traffic to different addresses and ports. Finally, when I find something suspicious that I want to better understand I will go to those particular messages and analyze the messages themselves. If you capture participant discovery within the wireshark capture then it will actually connect the messages with the discovery information, making it much easier to discern which messages are connected to which topics without having to track the GUIDs yourself.

Of course one limitation with this approach is that it does not snoop on the entire network, it simply records the data going through a given device and I compensate for that by capturing this data on both my PC and the raspberry pi.