Neural Network from Scratch

A feedforward neural network implemented using only Python and NumPy. No TensorFlow. No PyTorch. Just matrix math, backpropagation, and gradient descent.

Architecture

    INPUT          HIDDEN LAYERS          OUTPUT
   (n inputs)     (configurable)        (n outputs)

     x₁ ───┬────── h₁ ──┐
            ╳             ╳──── ... ──── ŷ₁
     x₂ ───┼────── h₂ ──┤
            ╳             ╳──── ... ──── ŷ₂
     x₃ ───┼────── h₃ ──┤
            ╳             ╳──── ... ──── ŷₘ
     ...    └────── h₄ ──┘
            
   Every connection has a learnable weight.
   Every neuron has a learnable bias.

Project Structure

neural-network-from-scratch/
├── neural_net.py       # Reusable NeuralNetwork class + demos
├── nn_study_notes.py   # Step-by-step walkthrough + experiments
└── README.md

Files

File	Purpose
neural_net.py	The main implementation — a reusable NeuralNetwork class with 3 demos
nn_study_notes.py	Annotated study notes building a network step-by-step, plus experiments

Features

• Configurable architecture — pass any list of layer sizes (e.g. [2, 8, 4, 1])
• Multiple activations — sigmoid, ReLU, tanh, softmax
• Multiple loss functions — MSE and cross-entropy
• Xavier initialization — helps training converge faster than plain random weights
• Momentum — optional momentum for gradient descent, helps on harder problems
• Mini-batch training & Learning rate decay — optional batch_size slicing and lr_decay learning rate schedule
• Early stopping — stop training automatically when loss drops below a threshold
• Model summary — Keras-style summary() showing layer shapes and parameter counts
• Save / Load — persist and restore trained models with np.savez
• Three demos — XOR, circle classification, and spiral classification

Quick Start

# Run all three demos
python neural_net.py

# Run the annotated study notes + experiments
python nn_study_notes.py

Demo Output (XOR)

==================================================
  XOR  –  the classic nonlinear benchmark
==================================================

  epoch      1/10000   loss: 0.268292
  epoch   1000/10000   loss: 0.007248
  ...
  ✓ converged at epoch 6453 (loss 0.000100 < tol 0.0001)

Predictions:
  [0 0]  →  0.0124  ✓
  [0 1]  →  0.9867  ✓
  [1 0]  →  0.9869  ✓
  [1 1]  →  0.0144  ✓

Use as a Library

from neural_net import NeuralNetwork
import numpy as np

# 1. Create a network
nn = NeuralNetwork(
    layer_sizes=[2, 8, 4, 1],
    activation="sigmoid",       # or "relu", "tanh", "softmax"
    loss="mse",                 # or "cross_entropy"
    seed=42
)

# 2. Train it
nn.train(
    X_train, y_train,
    epochs=5000,
    lr=0.5,
    momentum=0.9,               # optional, default 0.0
    batch_size=32,              # optional mini-batch size
    lr_decay=0.0001,            # optional learning rate decay
    tol=0.001,                  # optional early stopping
    verbose=True
)

# 3. Predict
predictions = nn.predict(X_test)

# 4. Inspect
nn.summary()
print(nn.loss_history[-1])      # final loss value

# 5. Save & load
nn.save("my_model.npz")
nn2 = NeuralNetwork.load("my_model.npz")

How It Works

Forward pass: Input flows through the network layer by layer. Each layer does a weighted sum of its inputs, adds a bias, and passes the result through an activation function.

z = X · W + b
a = activation(z)

Backward pass (backpropagation): Starting from the output, we compute how much each weight contributed to the error using the chain rule. Then we nudge every weight in the direction that reduces the loss.

Gradient descent: The nudge size is controlled by the learning rate. Too big and training oscillates. Too small and it takes forever. Momentum helps by accumulating velocity from past gradients, letting the optimizer "roll through" flat spots.

Early stopping: Instead of always running for the full number of epochs, we can stop as soon as the loss drops below a threshold — saving time on problems that converge quickly.

Things I Learned Building This

1. You need a hidden layer for XOR — a single-layer perceptron literally can't solve it (Minsky & Papert, 1969)
2. Weight initialization matters — all-zeros means all neurons learn the same thing (the symmetry problem)
3. Xavier initialization is a simple fix — scale initial weights by sqrt(6 / (fan_in + fan_out)) to keep gradients healthy
4. The learning rate is touchy — 5.0 diverged, 0.01 barely moved, 0.5–1.0 worked for XOR
5. Backprop is just the chain rule applied repeatedly — scarier in theory than in code
6. Shape bugs are the #1 time sink — print .shape after every operation
7. Momentum makes a real difference — the circle and spiral problems trained noticeably faster with momentum=0.9
8. MSE works fine for small problems, but cross-entropy is better for classification at scale
9. The spiral problem really needs depth — a [2, 4, 1] network struggles, but [2, 16, 16, 1] handles it

What's Next

Ideas for future extensions:

• Mini-batch training — update weights on random subsets instead of the full dataset
• Learning rate scheduling — reduce the learning rate over time for finer convergence
• Dropout — randomly disable neurons during training to prevent overfitting
• Batch normalization — normalize layer inputs to stabilize and speed up training
• Matplotlib visualizations — plot loss curves, decision boundaries, and weight distributions
• Convolutional layers — extend to image classification (MNIST digits)

Requirements

• Python 3.8+
• NumPy