[DLRMv4] Restrict the dlrmv4 learning rate to a band around the scaled reference - #473
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
|
@chriscai-amd shouldn't open division have the same constraint as closed? |
cf19554 to
81b4883
Compare
Both divisions left opt_base_learning_rate unconstrained apart from positivity, so a submission could report any learning rate while still being scored against RCPs collected on a single recipe. The reference sweep follows a linear scaling law -- 1e-6 at global batch 8192, 2e-6 at 16384, 4e-6 at 32768, i.e. 1e-6 * GBS / 8192 -- and convergence is only comparable near it. Confine the learning rate to half to 1.5x that value, deriving the bounds in the global_batch_size POST block where the batch size is already parsed. Because the rules execute in log order, the checks guard on the bounds being resolved so a log that reports the learning rate before the batch size fails with the state dump rather than raising inside the comparison. The band applies to the open division too. That division is meant to showcase algorithmic changes, not to let submitters buy faster convergence with a learning rate the closed division would reject. Because the open rules do not pin the embedding rate to the dense one, opt_sparse_base_learning_rate is banded independently there; in the closed division the existing equality rule already covers it. Co-authored-by: Cursor <cursoragent@cursor.com>
thanks, @ShriyaRishab , updated the PR to restrict both the closed and open divisions |
3 participants
The closed division left opt_base_learning_rate unconstrained apart from positivity, so a submission could report any learning rate while still being scored against RCPs collected on a single recipe. The reference sweep follows a linear scaling law -- 1e-6 at global batch 8192, 2e-6 at 16384, 4e-6 at 32768, i.e. 1e-6 * GBS / 8192 -- and convergence is only comparable near it.
Confine the learning rate to half to 1.5x that value, deriving the bounds in the global_batch_size POST block where the batch size is already parsed. Because the rules execute in log order, the check guards on the bounds being resolved so a log that reports the learning rate before the batch size fails with the state dump rather than raising inside the comparison. The sparse learning rate is already pinned to the dense one, so the band applies to both channels.
The open division is deliberately untouched: it exists to permit recipe changes, and the RCP checker only runs on closed submissions.