optimize get_rdma_nics.sh to support nsys profile#8099
Conversation
PaddlePaddle-bot
left a comment
There was a problem hiding this comment.
🤖 Paddle-CI-Agent | pr_review |
2026-07-13 21:32:07
📋 Review 摘要
PR 概述:缓存 ibdev2netdev 输出以减少 RDMA NIC 枚举期间的重复 fork 调用。
变更范围:fastdeploy/cache_manager/transfer_factory/get_rdma_nics.sh
影响面 Tag:[KVCache] [PD Disaggregation]
问题
| 级别 | 文件 | 概述 |
|---|---|---|
| 🟡 建议 | fastdeploy/cache_manager/transfer_factory/get_rdma_nics.sh:31 |
同名脚本副本未同步缓存逻辑,部分入口仍会多次调用 ibdev2netdev |
📝 PR 规范检查
标题缺少官方 Tag。描述结构包含 checklist.md §D2 要求的 Motivation、Modifications、Usage or Command、Accuracy Tests、Checklist 章节。
标题建议(可直接复制):
[Optimization] optimize get_rdma_nics.sh to support nsys profile
总体评价
本次修改能覆盖 Python runtime 通过 fastdeploy.cache_manager.transfer_factory.utils.get_rdma_nics() 调用的 package 脚本,但仓库内仍有旧副本未同步,建议补齐后再合入以避免 profile 场景入口不一致。
| gpu_first=true | ||
| xpu_first=true | ||
| cpu_first=true | ||
| # Cache ibdev2netdev output once to avoid repeated subprocess forks in the loop. |
There was a problem hiding this comment.
🟡 建议 这次只缓存了 package 资源脚本,但仓库里还有同名脚本副本仍保留旧的多次 ibdev2netdev 调用。
fastdeploy/cache_manager/transfer_factory/utils.py 会调用本文件;但 scripts/get_rdma_nics.sh 和 tests/e2e/utils/get_rdma_nics.sh 仍在 gpu / xpu / cpu_ib 分支里直接执行 ibdev2netdev | awk,e2e 用例也会调用后者。在 nsys profile 场景下这些入口仍可能卡在多次 fork 上,和本 PR 的优化目标不一致。
建议修复方式:将同样的 _IBDEV2NETDEV_CACHE 缓存逻辑同步到这两个副本,或改成单一脚本来源,避免三份实现继续漂移。
|
root seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #8099 +/- ##
==========================================
Coverage ? 67.50%
==========================================
Files ? 475
Lines ? 66962
Branches ? 10329
==========================================
Hits ? 45200
Misses ? 18884
Partials ? 2878
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Thanks for your contribution! |
Motivation
原有get_rdma_nics.sh实现调用多次ibdev2netdev,每次调用ibdev2netdev耗时大概0.4s。nsys profile需要用ptrace attach fork出来的进程,会增加调用耗时,由于ibdev2netdev内部存在多次fork调用,所以单次调用耗时从0.4提升到几十秒。get_rdma_nics.sh中多次调用,导致nsys profile启动进程时一直被卡在ibdev2netdev的调用中。
所以为避免nsys profile hang住,需要减少这个脚本中对于ibdev2netdev的调用。
Modifications
将ibdev2netdev的输出放入cache 变量 _IBDEV2NETDEV_CACHE中,避免后续多次调用ibdev2netdev。
Usage or Command
无
Accuracy Tests
无
Checklist
[Feature]
[FDConfig],[APIServer],[Engine],[Scheduler],[PD Disaggregation],[Executor],[Graph Optimization],[Speculative Decoding],[RL],[Models],[Quantization],[Loader],[OP],[KVCache],[DataProcessor],[BugFix],[Docs],[CI],[Optimization],[Feature],[Benchmark],[Others],[XPU],[HPU],[GCU],[DCU],[Iluvatar],[Metax]]pre-commitbefore commit.releasebranch, make sure the PR has been submitted to thedevelopbranch, then cherry-pick it to thereleasebranch with the[Cherry-Pick]PR tag.