Pinned Loading

1. Single-file training script for a GP... Single-file training script for a GPT-style decoder trained on TinyStories using GPT-2 BPE (tiktoken). Implements a modern pre-norm stack (RMSNorm + causal SDPA + GELU MLP) with weight tying, bf16 mixed precision (when available), warmup + cosine LR decay, gradient clipping, checkpointing (best + periodic), and a built-in sample generation at the end.

   1

   """

   2

   i16 — Train a GPT-style decoder on TinyStories (tiktoken GPT-2 BPE).

   3

   4

   Usage:

   5

     python i16_train.py --data TinyStoriesV2-GPT4-train.txt --out checkpoints