Skip to content

[BUG] Skip None buffers in AutoTP checkpoint loading - #8607

Draft
adarshsm wants to merge 1 commit into
deepspeedai:masterfrom
adarshsm:fix/8514-autotp-none-buffer
Draft

adarshsm wants to merge 1 commit into
deepspeedai:masterfrom
adarshsm:fix/8514-autotp-none-buffer

Conversation

@adarshsm

Copy link
Copy Markdown

Bug

Loading.load_buffer() in deepspeed/module_inject/auto_tp.py iterates every key in module._buffers and unconditionally does module._buffers[name].data.is_meta. PyTorch lets a buffer be registered as None — e.g. nn.InstanceNorm1d with the default track_running_stats=False registers running_mean, running_var, and num_batches_tracked as None. Loading a checkpoint into a model containing such a module crashes with:

AttributeError: 'NoneType' object has no attribute 'data'

Repro from #8514 (CPU-only, no GPU or transformers needed):

import torch, torch.nn as nn
from deepspeed.module_inject.replace_module import replace_module

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.norm = nn.InstanceNorm1d(4)
        self.fc = nn.Linear(4, 4)

torch.save(Net().state_dict(), "/tmp/ckpt.pt")
noop = lambda child, policy, layer_id, prefix='', state_dict=None: child

replace_module(Net(), nn.Linear, noop, None, checkpoint=None)             # returns
replace_module(Net(), nn.Linear, noop, None, checkpoint="/tmp/ckpt.pt")   # raises

Fix

Skip a buffer when it is None before touching .data — there is nothing to cast to a concrete tensor or copy a checkpoint value into for an unset buffer.

Testing

Added tests/unit/module_inject/test_load_buffer.py:

  • test_load_buffer_skips_unset_buffers — an nn.InstanceNorm1d (all-None buffers) goes through Loading.load_buffer without raising. This fails with the reported AttributeError on the code before this change.
  • test_load_buffer_still_loads_present_buffers — a normal nn.BatchNorm1d buffer is still copied from state_dict as before.

Ran pytest tests/unit/module_inject/test_load_buffer.py — both pass after the fix, and test_load_buffer_skips_unset_buffers reproduces the reported crash when run against the pre-fix code. Also ran flake8 and yapf --style .style.yapf --diff over both changed files with no findings.

Fixes #8514

Loading.load_buffer() iterated every key in module._buffers and
unconditionally read module._buffers[name].data, but PyTorch modules
can register a buffer as None (e.g. nn.InstanceNorm1d's
running_mean/running_var/num_batches_tracked when
track_running_stats=False). Loading a checkpoint into such a module
raised AttributeError: 'NoneType' object has no attribute 'data'.

Skip buffers that are None before touching .data; there is nothing to
cast to a concrete tensor or copy a checkpoint value into.

Fixes deepspeedai#8514

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: adarshsm <24850536+adarshsm@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] AutoTP checkpoint load raises AttributeError on a model containing nn.InstanceNorm1d

1 participant