Skip to content

optimize get_rdma_nics.sh to support nsys profile#8099

Open
wangyuwen1999 wants to merge 1 commit into
PaddlePaddle:developfrom
wangyuwen1999:develop
Open

optimize get_rdma_nics.sh to support nsys profile#8099
wangyuwen1999 wants to merge 1 commit into
PaddlePaddle:developfrom
wangyuwen1999:develop

Conversation

@wangyuwen1999

Copy link
Copy Markdown
Contributor

Motivation

💡 If this PR is a Cherry Pick, the PR title needs to follow the format by adding the [Cherry-Pick] label at the very beginning and appending the original PR ID at the end. For example, [Cherry-Pick][CI] Add check trigger and logic(#5191)

💡 如若此PR是Cherry Pick,PR标题需遵循格式,在最开始加上[Cherry-Pick]标签,以及最后面加上原PR ID,例如[Cherry-Pick][CI] Add check trigger and logic(#5191)

原有get_rdma_nics.sh实现调用多次ibdev2netdev,每次调用ibdev2netdev耗时大概0.4s。nsys profile需要用ptrace attach fork出来的进程,会增加调用耗时,由于ibdev2netdev内部存在多次fork调用,所以单次调用耗时从0.4提升到几十秒。get_rdma_nics.sh中多次调用,导致nsys profile启动进程时一直被卡在ibdev2netdev的调用中。

所以为避免nsys profile hang住,需要减少这个脚本中对于ibdev2netdev的调用。

Modifications

将ibdev2netdev的输出放入cache 变量 _IBDEV2NETDEV_CACHE中,避免后续多次调用ibdev2netdev。

Usage or Command

Accuracy Tests

Checklist

[Feature]

  • Add at least a tag in the PR title.
    • Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
    • You can add new tags based on the PR content, but the semantics must be clear.
  • Format your code, run pre-commit before commit.
  • Add unit tests. Please write the reason in this PR if no unit tests.
  • Provide accuracy results.
  • If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

@PaddlePaddle-bot PaddlePaddle-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Paddle-CI-Agent | pr_review | 2026-07-13 21:32:07

📋 Review 摘要

PR 概述:缓存 ibdev2netdev 输出以减少 RDMA NIC 枚举期间的重复 fork 调用。
变更范围fastdeploy/cache_manager/transfer_factory/get_rdma_nics.sh
影响面 Tag[KVCache] [PD Disaggregation]

问题

级别 文件 概述
🟡 建议 fastdeploy/cache_manager/transfer_factory/get_rdma_nics.sh:31 同名脚本副本未同步缓存逻辑,部分入口仍会多次调用 ibdev2netdev

📝 PR 规范检查

标题缺少官方 Tag。描述结构包含 checklist.md §D2 要求的 MotivationModificationsUsage or CommandAccuracy TestsChecklist 章节。

标题建议(可直接复制):

  • [Optimization] optimize get_rdma_nics.sh to support nsys profile

总体评价

本次修改能覆盖 Python runtime 通过 fastdeploy.cache_manager.transfer_factory.utils.get_rdma_nics() 调用的 package 脚本,但仓库内仍有旧副本未同步,建议补齐后再合入以避免 profile 场景入口不一致。

gpu_first=true
xpu_first=true
cpu_first=true
# Cache ibdev2netdev output once to avoid repeated subprocess forks in the loop.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 建议 这次只缓存了 package 资源脚本,但仓库里还有同名脚本副本仍保留旧的多次 ibdev2netdev 调用。

fastdeploy/cache_manager/transfer_factory/utils.py 会调用本文件;但 scripts/get_rdma_nics.shtests/e2e/utils/get_rdma_nics.sh 仍在 gpu / xpu / cpu_ib 分支里直接执行 ibdev2netdev | awk,e2e 用例也会调用后者。在 nsys profile 场景下这些入口仍可能卡在多次 fork 上,和本 PR 的优化目标不一致。

建议修复方式:将同样的 _IBDEV2NETDEV_CACHE 缓存逻辑同步到这两个副本,或改成单一脚本来源,避免三份实现继续漂移。

@liyonghua0910 liyonghua0910 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


root seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@codecov-commenter

codecov-commenter commented Jul 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (develop@9a48fab). Learn more about missing BASE report.

Additional details and impacted files
@@            Coverage Diff             @@
##             develop    #8099   +/-   ##
==========================================
  Coverage           ?   67.50%           
==========================================
  Files              ?      475           
  Lines              ?    66962           
  Branches           ?    10329           
==========================================
  Hits               ?    45200           
  Misses             ?    18884           
  Partials           ?     2878           
Flag Coverage Δ
GPU 77.53% <ø> (?)
XPU 6.95% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@paddle-bot

paddle-bot Bot commented Jul 14, 2026

Copy link
Copy Markdown

Thanks for your contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants