Least Squares: Levenberg-Marquardt

The Levenberg-Marquardt method is one of the most common algorithms for nonlinear least squares [1],[2]. It is popular because it balances two useful behaviors:

• It can behave like Gauss-Newton when the local approximation is reliable.
• It can behave more like gradient descent when the step needs to be cautious.

That balance makes it especially useful for real experimental fitting, where the first parameter guess is often imperfect.

The Gauss-Newton problem

In Gauss-Newton, each iteration solves [3],[4]:

$${\mathbf{J}}^T\mathbf{J}\mathbf{\delta }={\mathbf{J}}^T\mathbf{r}$$

and then updates:

$${\mathbf{\beta }}^{(k+1)}={\mathbf{\beta }}^{(k)}+\mathbf{\delta }$$

This can be very fast when the current point is already close to the answer. But if the starting point is poor, the Gauss-Newton step can be too aggressive.

Adding damping

A common Levenberg-style damping form modifies the system by adding a diagonal stabilizing term:

$$({\mathbf{J}}^T\mathbf{J}+\lambda \mathbf{I})\mathbf{\delta }={\mathbf{J}}^T\mathbf{r}$$

Here, $\lambda$ controls the damping and $\mathbf{I}$ is the identity matrix [1],[2]. Many descriptions of the Marquardt variant use $\lambda \mathrm{diag}({\mathbf{J}}^T\mathbf{J})$ instead of $\lambda \mathbf{I}$. The idea is the same for this article: add damping so the step becomes safer when the local approximation is unreliable.

When $\lambda$ is small, the method behaves like Gauss-Newton. When $\lambda$ is large, the method becomes more cautious and behaves more like gradient descent.

Interactive example

This panel uses the same kind of nonlinear curve fitting problem as Gauss-Newton, but now the damping parameter changes as the fit improves. When the model is uncertain, the method behaves cautiously. When the local approximation starts working, it becomes more confident.

The algorithmic rhythm

A typical Levenberg-Marquardt iteration looks like this:

1. Start from current parameters.
2. Compute residuals and the Jacobian.
3. Solve the damped system for $\mathbf{\delta }$.
4. Try the new parameters.
5. If the fit improves, accept the step and reduce damping.
6. If the fit worsens, reject or shrink the step and increase damping.

This makes the method adaptive. It can move quickly when the path is clear, but slow down when the model surface becomes tricky.

Where it appears in chemistry

Levenberg-Marquardt is commonly used for nonlinear curve fitting:

• Exponential decay models.
• Michaelis-Menten kinetics.
• Spectral peak fitting.
• Binding curves.
• Calibration models with nonlinear response.

In all these cases, the model parameters have physical meaning, and a stable fitting method matters. A small change in starting values should not make the model behave wildly.

What it gives you

Levenberg-Marquardt does not guarantee perfection. Nonlinear least squares can still have local minima, and the starting point still matters. But compared with plain Gauss-Newton, it is usually more forgiving.

If Gauss-Newton is the confident version of nonlinear least squares, Levenberg-Marquardt is the version that knows when to be careful.

References