Skip to content

Search every L-BFGS step with a zoom line search by default - #170

Merged
jessegrabowski merged 6 commits into
pymc-devs:mainfrom
jessegrabowski:lbfgs-default-line-search
Oct 6, 2026
Merged

jessegrabowski merged 6 commits into
pymc-devs:mainfrom
jessegrabowski:lbfgs-default-line-search

Conversation

@jessegrabowski

@jessegrabowski jessegrabowski commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

lbfgs_updates, the L-BFGS rule, now takes an optional line_search. With one set, it searches along the step -learning_rate * H g and moves by the step size the search accepts, so learning_rate becomes the first trial. The lbfgs alias defaults to zoom_line_search(), and on Rosenbrock it reproduces the first 12 losses of optax.lbfgs() to 1e-10.

With the search on, lbfgs() raises when it is given precomputed gradients, when it is placed after another transform in a chain, or when the loss draws random numbers. The search evaluates the loss at several trial points, so it needs the loss itself, and the loss has to be deterministic. line_search=None restores the fixed step.

On mlx, every search runs all max_steps trials, because mx.compile cannot stop a loop early. Each step there costs that many loss and gradient evaluations.

LineSearch and zoom_line_search are now exported from pytensor_ml.optim.

Closes #58


📚 Documentation preview 📚: https://pytensor-ml--170.org.readthedocs.build/en/170/

@jessegrabowski
jessegrabowski merged commit fc23fc9 into pymc-devs:main Oct 6, 2026
22 of 39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement LBFGS optimizer

1 participant