Spectral Derivatives

The mathematics of finite differences, the discrete cousin of differentiation, goes back to Isaac Newton, who developed difference formulas in the late 17th century to interpolate between tabulated values. Through the 18th century mathematicians such as Brook Taylor and Leonhard Euler turned the calculus of finite differences into a branch of mathematics of its own. For a long time, though, these tools lived in numerical tables and astronomy. Derivative spectra were first recorded in the 1950s with analogue electronics: Allen Giese and C. Stacy French (1955) used a derivative spectrophotometer to reveal weak bands hidden under strong ones [8]. When spectrometers began to produce digital output in the 1960s, the derivative could be computed numerically instead, and the problem became how to do that without drowning the result in noise.

A key step came in 1964, when Abraham Savitzky and Marcel Golay published their paper on smoothing and differentiation by simplified least squares [1]. It is usually cited for smoothing, but it also showed that smoothed derivatives fall out of local polynomial fitting for free: evaluate the derivative of the fitted polynomial instead of the polynomial itself, and you get a derivative and a smoothing in a single convolution. This solved a serious practical problem, because plain finite differences amplify noise so much that they are often useless on real data.

Derivative spectroscopy was well established in UV-Vis spectrophotometry by the late 1970s [4],[5], and in near-infrared (NIR) spectroscopy it became a routine tool. In their 1984 study of protein in ground wheat, Karl Norris and Phil Williams examined how particle size distorts NIR reflectance spectra and found that derivative-based treatments, in particular second derivatives with ratios between wavelengths, greatly reduced the particle-size effects [2]. Norris also popularised the "gap" or "segment" derivative described below. Today, derivatives are among the most widely used preprocessing steps in chemometrics, applied routinely to NIR, mid-IR, Raman and UV-Vis spectra [3].

Derivatives solve three recurring problems.

Baseline removal. Many spectra sit on offsets or slopes caused by scattering, instrument drift or sample positioning. Differentiation removes them:

• A constant offset has zero slope, so the first derivative eliminates it.
• A straight-line baseline has zero curvature, so the second derivative eliminates it.
• In general, the $q$-th derivative (the derivative of order $q$) removes any polynomial baseline of degree $q-1$.

Resolving overlapping bands. Broad, overlapping bands are common, especially in NIR, where overtone and combination bands are intrinsically wide. The second derivative turns each band into a narrower negative peak, so overlapping components can be seen separately.

Enhancing subtle features. Small shoulders and inflection points that are almost invisible in the spectrum become clear features in derivative spectra, which helps to detect minor components or small differences between similar samples.

The panel below shows all three effects. Three samples share the same chemistry but sit on different baselines. Switch between the spectrum and its derivatives, and between a plain finite difference, a Savitzky-Golay derivative and a gap-segment derivative.

Things to notice:

• In the spectrum, the three samples differ mostly by their baselines.
• The first derivative removes the offsets, but the samples with different slopes still differ by a constant amount.
• The second derivative makes the three curves practically identical: offsets and slopes are both gone.
• With a finite difference, the second derivative is buried in noise even at a modest noise level. The Savitzky-Golay derivative recovers it; widening its window cleans it further, until the bands themselves start to be smoothed away.
• The gap-segment derivative (described below, here with gap and segment equal to a third of the window) gives a derivative very similar to the Savitzky-Golay one.

The first derivative is the rate of change, the slope, at each point. For a spectrum sampled at equal intervals $\Delta x$, the simplest estimate is the central difference

which uses the points on either side of $i$.

• Band maxima become zero crossings. At the top of a band the slope is zero, changing from positive to negative.
• Rising flanks become positive, falling flanks negative.
• Constant offsets vanish. A flat offset has zero slope everywhere.
• The band shape becomes antisymmetric: a positive lobe followed by a negative lobe, with extremes at the inflection points of the original band.

A band maximum is where $dy/dx=0$ and the derivative changes from positive to negative. Zero crossings of a (smoothed) first derivative are therefore a convenient way to locate band positions, especially when bands sit on a sloping baseline that makes the maximum of the raw spectrum shift.

The second derivative is the curvature. Its finite difference estimate is

• Bands become negative peaks. At a maximum the curve bends downwards, so the curvature is negative. The minimum of the second derivative sits at the band centre.
• Bands become narrower. For a Gaussian band, the negative peak of the second derivative has a full width at half maximum of about 53 % of the original band. (The fourth derivative narrows further, to about 41 %.)
• Side lobes appear. Each negative peak is flanked by two small positive lobes, which can overlap with neighbouring bands in crowded spectra.
• Straight-line baselines vanish. A line $a+bx$ has zero curvature, so offsets and slopes both disappear.

Narrow bands win. For a Gaussian band of height $h$ and width $\sigma$, the amplitude of the $q$-th derivative scales as $h/{\sigma }^q$. A band twice as narrow gives a second derivative four times larger. This is why derivatives suppress broad backgrounds, such as fluorescence or scattering, and emphasise sharp features.

Because bands point downwards in the second derivative, many authors plot −1 times the second derivative so that bands point up as in the original spectrum. Always check the axis label before interpreting a published second-derivative spectrum.

The narrowing is what makes second derivatives useful for resolving overlapping bands. Move the two bands below closer together and count the maxima.

For two equal Gaussian bands, the spectrum shows a single hump as soon as their centres are closer than about 0.85 times their FWHM. The inverted second derivative still shows two peaks down to about 0.63 FWHM, and the fourth derivative down to about 0.51 FWHM. The panel starts with two equal bands, so you can reproduce these numbers to within one step of the slider; with unequal bands all three limits move to larger separations. This is a real gain in apparent resolution, but it is paid for in noise: in real data you can only use it if the signal-to-noise ratio is high enough [4]. Add a little noise in the panel and watch the derivative, the fourth derivative above all, grow spurious maxima while the spectrum still looks clean.

There are three common ways to compute spectral derivatives.

The formulas above compute derivatives directly from neighbouring points. They are simple, but they keep all the noise and amplify it: each differentiation acts as a high-pass filter, letting fast changes through and suppressing slow ones. On real spectra, plain finite differences are often unusable without prior smoothing.

The Savitzky-Golay method fits a polynomial of order $p$ to a window of $w$ points by least squares and evaluates the derivative of that polynomial at the centre [1],[6]. Because the fit averages over the whole window, the result is far less noisy than a finite difference.

As for smoothing, the operation reduces to fixed weights, which depend on $w$, $p$ and the derivative order $q$:

For example, a 5-point quadratic fit gives the first-derivative weights $\frac{1}{10}[-2,-1,0,1,2]$, and a 7-point quadratic fit gives the second-derivative weights $\frac{1}{42}[5,0,-3,-4,-3,0,5]$.

Key advantage: smoothing and differentiation happen in one step.

Parameter requirements:

• The polynomial order must be at least the derivative order: a straight line has no curvature, so $p\ge q$. In practice $p=2$ or 3 is used for both first and second derivatives.
• Larger windows give smoother derivatives but broaden features.
• Typical settings: $p=2$ or 3, with a window chosen from the number of points that span a band (see Choosing Savitzky-Golay derivative parameters below).

The gap derivative, popularised by Karl Norris [9], takes the difference between two points separated by a gap of $g$ data points ($g$ even, so that both points are equally far from $i$):

The wider gap gives some built-in smoothing, because the difference is taken over a longer interval where the signal changes more than the noise. Often each end point is replaced by the average of a short segment of $s$ points before taking the difference. This version is known as the gap-segment or Norris-Williams derivative, and it has three settings: derivative order, segment size and gap size. Be careful when you copy settings from a paper or a program: some define the gap as the distance between the centres of the two segments, others as the number of points skipped between them, and the same numbers then give different filters.

When to use gap derivatives:

• When you want a simple derivative with modest smoothing.
• In NIR applications, where gap-segment derivatives have a long tradition and are built into much commercial software.
• When you want to reproduce a published method that used them.

This is the single most important practical issue in derivative spectroscopy. Picture a signal made of a single wave, $\sin (\omega x)$. Its derivative is $\omega \cos (\omega x)$: the same wave, multiplied by its frequency $\omega$. Broad bands are made of slow waves (small $\omega$) and are barely amplified. Noise is made of fast waves (large $\omega$) and is amplified strongly. A second derivative multiplies by ${\omega }^2$, so the effect is even stronger.

Consequences:

• First derivative: noticeably rougher than the spectrum.
• Second derivative: without adequate smoothing, dominated by noise.
• Third and higher derivatives: rarely used, because noise becomes unmanageable (the fourth derivative is occasionally used in UV-Vis on very clean data).

For independent noise with standard deviation ${\sigma }_0$, error propagation gives the noise in simple finite differences [11]. For the forward difference $(y_{i+1}-y_i)/\Delta x$:

For the central difference $(y_{i+1}-y_{i-1})/(2\Delta x)$, it is ${\sigma }_0/(\sqrt{2}\Delta x)$. For the second difference:

because the weights 1, -2, 1 have squares summing to 6. Measured per data point (that is, before dividing by $\Delta x$ or $(\Delta x{)}^2$), the noise only grows by a modest factor, at most $\sqrt{6}$ here, but the signal in a derivative is small when bands span many points: the slope of a band 20 points wide is far smaller per point than its height. The ratio of derivative signal to derivative noise therefore drops quickly with derivative order, and it drops further when the spectrum is sampled more finely (smaller $\Delta x$ compared with the band width).

1. Smooth first, then differentiate. Apply a smoother (moving average, Gaussian, Whittaker) and differentiate the result. It works, but you have two sets of parameters to tune.
2. Use Savitzky-Golay derivatives. One operation, with the window size and polynomial order jointly controlling the trade-off between noise and resolution.

First derivative when:

• You need to remove constant offsets.
• You want to emphasise slopes and inflection points.
• You build PLS or PCR models and want to remove additive baseline effects while keeping noise amplification moderate.

Second derivative when:

• You need to remove both offsets and linear slopes.
• You want to resolve overlapping bands or find hidden shoulders.
• Your signal-to-noise ratio can tolerate the extra amplification.

Third derivative and above are rarely worth it. If the second derivative is not enough, consider other preprocessing instead.

Window size w. Larger windows give smoother derivatives but broaden features. Starting points:

• First derivatives: $w=9$ to 15.
• Second derivatives: $w=11$ to 21, since more smoothing is needed.
• Adjust to the number of points across your narrowest band and to the noise level. Zimmermann and Kohler show, for infrared spectra, how the window and order trade resolution against noise and how to optimise them [10].

Polynomial order p. At least the derivative order:

• First derivative: $p=2$ (for the first derivative, $p=1$ and 2 give the same weights, as do 3 and 4).
• Second derivative: $p=2$ or 3 (identical weights for the second derivative), or 4 for sharper bands with a longer window.

Rule of thumb: use the lowest polynomial order that follows your band shapes, with a window just large enough to tame the noise, and validate by inspection.

For gap-segment derivatives, the gap plays a role similar to the Savitzky-Golay window:

• Smaller gap: more detail, more noise.
• Larger gap: smoother, but narrow features may be lost.
• Values of a few to about twenty points are common for NIR data with 1 to 2 nm spacing; choose them relative to your band widths.

1. Visual inspection. Do the derivative features match real spectral features, or is the curve dominated by noise?
2. Zero crossings. For first derivatives, do they align with known band positions?
3. Band count. For second derivatives, does the number of resolved bands match what the chemistry suggests? Be suspicious of extra bands produced by noise or side lobes.
4. Model performance. If the derivatives feed a PLS model, compare cross-validated errors for different orders and windows [7].

Derivatives are probably most important in NIR, where they serve as a primary baseline correction. NIR spectra consist of broad, overlapping overtone and combination bands, and scattering from solid samples adds offsets and slopes that vary from sample to sample [3]:

• The first derivative removes sample-to-sample offsets.
• The second derivative removes offsets and linear slopes, and sharpens the broad bands.
• Savitzky-Golay and gap-segment derivatives are among the most common NIR preprocessing steps [3].

Electronic absorption bands are broad and often merge. Second (and sometimes fourth) derivatives help to reveal the number and position of underlying components and to quantify one compound in the presence of another with an overlapping spectrum [4],[5]. Because differentiation is linear, the derivative amplitude is still proportional to concentration, just like the absorbance in the Beer-Lambert law. A classic trick is the zero-crossing method: measure the analyte at a wavelength where the derivative of the interfering compound is zero, so that the interferent contributes nothing there. Typical uses include:

• Pharmaceutical analysis, determining drugs in mixtures with overlapping UV spectra.
• Removing background absorption or turbidity that varies smoothly with wavelength.
• Protein studies, where derivatives of near-UV spectra reveal the environment of aromatic amino acids.

Raman spectra often sit on a broad fluorescence background. Since that background is smooth while Raman bands are sharp, a second derivative suppresses the fluorescence and keeps the bands. In practice, dedicated baseline correction methods such as AsLS are usually preferred, because they keep the familiar band shapes.

Mid-IR spectra have sharper bands and smaller baseline variation than NIR, so derivatives are used less routinely. Second derivatives remain useful for:

• Resolving overlapping bands in the fingerprint region.
• Detecting minor components in mixtures.
• Analysing protein secondary structure from the components of the amide I band.

import numpy as np
from scipy.signal import savgol_filter
import matplotlib.pyplot as plt

# Generate a synthetic spectrum with overlapping peaks on a sloping baseline
rng = np.random.default_rng(42)
x = np.linspace(800, 2500, 850)  # NIR wavelengths (nm)

# True chemical signal: overlapping absorption bands
signal = (0.4 * np.exp(-((x - 1200)**2) / 8000) +
          0.35 * np.exp(-((x - 1400)**2) / 6000) +
          0.5 * np.exp(-((x - 1700)**2) / 10000) +
          0.25 * np.exp(-((x - 1950)**2) / 5000))

# Add a linear baseline (simulating scattering)
baseline = 0.3 + 0.0002 * x

# Add noise
noise = rng.normal(0, 0.012, len(x))
spectrum = signal + baseline + noise

# Compute SG derivatives
# First derivative: window=15, polynomial order=2, derivative order=1
dx = x[1] - x[0]  # sample spacing, so the SG derivatives are per nm like np.gradient
first_deriv = savgol_filter(spectrum, window_length=15, polyorder=2, deriv=1, delta=dx)

# Second derivative: window=21, polynomial order=3, derivative order=2
second_deriv = savgol_filter(spectrum, window_length=21, polyorder=3, deriv=2, delta=dx)

# For comparison: simple central finite differences, as in the formulas above (noisy!)
fd_first = np.r_[np.nan, (spectrum[2:] - spectrum[:-2]) / (2 * dx), np.nan]
fd_second = np.r_[np.nan, np.diff(spectrum, 2), np.nan] / dx**2

# Plot results
fig, axes = plt.subplots(4, 1, figsize=(12, 14))

# Original spectrum
axes[0].plot(x, spectrum, linewidth=1.5, color='#2563eb')
axes[0].set_ylabel('Absorbance')
axes[0].set_title('Original NIR Spectrum (with baseline + noise)')
axes[0].grid(True, alpha=0.3)

# First derivative comparison
axes[1].plot(x, fd_first, alpha=0.4, color='gray', label='Finite difference')
axes[1].plot(x, first_deriv, linewidth=2, color='#dc2626',
             label='SG (w=15, p=2)')
axes[1].axhline(0, color='black', linestyle='--', alpha=0.3)
axes[1].set_ylabel('dA/dx')
axes[1].set_title('First Derivative')
axes[1].legend()
axes[1].grid(True, alpha=0.3)

# Second derivative comparison
axes[2].plot(x, fd_second, alpha=0.4, color='gray', label='Finite difference')
axes[2].plot(x, second_deriv, linewidth=2, color='#059669',
             label='SG (w=21, p=3)')
axes[2].axhline(0, color='black', linestyle='--', alpha=0.3)
axes[2].set_ylabel('d²A/dx²')
axes[2].set_title('Second Derivative')
axes[2].legend()
axes[2].grid(True, alpha=0.3)

# Effect of window size on second derivative
for w in [11, 17, 25]:
    d2 = savgol_filter(spectrum, window_length=w, polyorder=3, deriv=2, delta=dx)
    axes[3].plot(x, d2, linewidth=1.5, label=f'w={w}')
axes[3].axhline(0, color='black', linestyle='--', alpha=0.3)
axes[3].set_xlabel('Wavelength (nm)')
axes[3].set_ylabel('d²A/dx²')
axes[3].set_title('Second Derivative: Effect of Window Size')
axes[3].legend()
axes[3].grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

% Spectral derivatives with Savitzky-Golay
rng(42);
x = linspace(800, 2500, 850);  % NIR wavelengths (nm)

% Synthetic spectrum: overlapping bands + linear baseline + noise
signal = 0.4 * exp(-((x - 1200).^2) / 8000) + ...
         0.35 * exp(-((x - 1400).^2) / 6000) + ...
         0.5 * exp(-((x - 1700).^2) / 10000) + ...
         0.25 * exp(-((x - 1950).^2) / 5000);
baseline = 0.3 + 0.0002 * x;
noise = 0.012 * randn(size(x));
spectrum = signal + baseline + noise;

% Savitzky-Golay derivatives
% sgolay returns the filters; column q+1 of g gives the q-th derivative divided by q!,
% and conv flips the filter, hence the factor factorial(q)/(-dx)^q (see MATLAB docs)
% First derivative (window=15, order=2)
[~, g15] = sgolay(2, 15);
dx = x(2) - x(1);
first_deriv_sg = conv(spectrum, -g15(:,2), 'same') / dx;          % factorial(1)/(-dx)^1

% Second derivative (window=21, order=3)
[~, g21] = sgolay(3, 21);
second_deriv_sg = conv(spectrum, 2 * g21(:,3), 'same') / dx^2;   % factorial(2)/(-dx)^2

% Simple central finite differences, as in the formulas above (for comparison)
fd_first = [NaN, (spectrum(3:end) - spectrum(1:end-2)) / (2 * dx), NaN];
fd_second = [NaN, diff(spectrum, 2), NaN] / dx^2;

% Plot results
figure('Position', [100 100 1000 900]);

subplot(4,1,1);
plot(x, spectrum, 'LineWidth', 1.5, 'Color', [0.15 0.39 0.93]);
ylabel('Absorbance');
title('Original NIR Spectrum');
grid on;

subplot(4,1,2);
plot(x, fd_first, 'Color', [0.7 0.7 0.7], 'DisplayName', 'Finite diff');
hold on;
plot(x, first_deriv_sg, 'LineWidth', 2, 'Color', [0.86 0.15 0.15], ...
     'DisplayName', 'SG (w=15, p=2)');
yline(0, '--', 'Color', [0.6 0.6 0.6], 'HandleVisibility', 'off');
ylabel('dA/dx');
title('First Derivative');
legend('Location', 'best');
grid on;

subplot(4,1,3);
plot(x, fd_second, 'Color', [0.7 0.7 0.7], 'DisplayName', 'Finite diff');
hold on;
plot(x, second_deriv_sg, 'LineWidth', 2, 'Color', [0.02 0.59 0.40], ...
     'DisplayName', 'SG (w=21, p=3)');
yline(0, '--', 'Color', [0.6 0.6 0.6], 'HandleVisibility', 'off');
ylabel('d²A/dx²');
title('Second Derivative');
legend('Location', 'best');
grid on;

subplot(4,1,4);
for w = [11, 17, 25]
    [~, gw] = sgolay(3, w);
    d2 = conv(spectrum, 2 * gw(:,3), 'same') / dx^2;
    plot(x, d2, 'LineWidth', 1.5, 'DisplayName', sprintf('w=%d', w));
    hold on;
end
yline(0, '--', 'Color', [0.6 0.6 0.6], 'HandleVisibility', 'off');
xlabel('Wavelength (nm)');
ylabel('d²A/dx²');
title('Second Derivative: Effect of Window Size');
legend('Location', 'best');
grid on;

library(signal)  # For sgolayfilt

# Generate synthetic NIR spectrum
set.seed(42)
x <- seq(800, 2500, length.out = 850)

# Overlapping absorption bands + linear baseline + noise
signal <- 0.4 * exp(-((x - 1200)^2) / 8000) +
          0.35 * exp(-((x - 1400)^2) / 6000) +
          0.5 * exp(-((x - 1700)^2) / 10000) +
          0.25 * exp(-((x - 1950)^2) / 5000)
baseline <- 0.3 + 0.0002 * x
noise <- rnorm(length(x), mean = 0, sd = 0.012)
spectrum <- signal + baseline + noise

dx <- x[2] - x[1]

# SG first derivative (window=15, polynomial=2, derivative=1)
first_deriv_sg <- sgolayfilt(spectrum, p = 2, n = 15, m = 1, ts = dx)

# SG second derivative (window=21, polynomial=3, derivative=2)
second_deriv_sg <- sgolayfilt(spectrum, p = 3, n = 21, m = 2, ts = dx)

# Simple central finite differences, as in the formulas above (for comparison)
n <- length(spectrum)
fd_first <- c(NA, (spectrum[3:n] - spectrum[1:(n - 2)]) / (2 * dx), NA)
fd_second <- c(NA, diff(spectrum, differences = 2) / dx^2, NA)

# Plot results
par(mfrow = c(4, 1), mar = c(4, 4, 3, 1))

# Original spectrum
plot(x, spectrum, type = 'l', lwd = 1.5, col = '#2563eb',
     xlab = '', ylab = 'Absorbance',
     main = 'Original NIR Spectrum')
grid()

# First derivative
plot(x, fd_first, type = 'l', col = 'gray70',
     xlab = '', ylab = 'dA/dx',
     main = 'First Derivative')
lines(x, first_deriv_sg, lwd = 2, col = '#dc2626')
abline(h = 0, lty = 2, col = 'black')
legend('topright', legend = c('Finite diff', 'SG (w=15, p=2)'),
       col = c('gray70', '#dc2626'), lwd = c(1, 2), cex = 0.9)
grid()

# Second derivative
plot(x, fd_second, type = 'l', col = 'gray70',
     xlab = '', ylab = expression(d^2*A/dx^2),
     main = 'Second Derivative')
lines(x, second_deriv_sg, lwd = 2, col = '#059669')
abline(h = 0, lty = 2, col = 'black')
legend('topright', legend = c('Finite diff', 'SG (w=21, p=3)'),
       col = c('gray70', '#059669'), lwd = c(1, 2), cex = 0.9)
grid()

# Window size comparison
windows <- c(11, 17, 25)
colors <- c('#2563eb', '#dc2626', '#059669')
d2_first <- sgolayfilt(spectrum, p = 3, n = windows[1], m = 2, ts = dx)
plot(x, d2_first, type = 'l', lwd = 1.5, col = colors[1],
     xlab = 'Wavelength (nm)', ylab = expression(d^2*A/dx^2),
     main = 'Second Derivative: Effect of Window Size')
for (i in 2:length(windows)) {
  d2 <- sgolayfilt(spectrum, p = 3, n = windows[i], m = 2, ts = dx)
  lines(x, d2, lwd = 1.5, col = colors[i])
}
abline(h = 0, lty = 2, col = 'black')
legend('topright', legend = paste0('w=', windows),
       col = colors, lwd = 1.5, cex = 0.9)
grid()

• The $q$-th derivative removes polynomial baselines of degree $q-1$: offsets with the first derivative, offsets and slopes with the second.
• Second derivatives turn bands into narrower negative peaks, which helps to resolve overlapping bands.
• Derivative amplitude scales as $1/{\text{width}}^q$, so derivatives favour sharp bands over broad backgrounds, and it stays proportional to concentration.
• Differentiation amplifies noise, more so for each order. Never differentiate raw data with plain finite differences.
• Savitzky-Golay derivatives smooth and differentiate in one step and are the standard choice; gap-segment derivatives are a common alternative in NIR.
• Derivatives do not remove multiplicative scatter, and they do not preserve absolute intensities.