Glossary term

Linearity Error

Engineering definition of linearity error covering nonlinearity, best-fit straight line, full-scale error, calibration residuals, sensor validation and release evidence.

Branch: Engineering Physics
Glossary type: metric
Content: Glossary term
Updated: Jun 26, 2026
Revision: v1.0.0 · reviewed

Definition

metric

Linearity error is the deviation between a measurement system output and the straight-line relation used to represent it over a stated range.

Linearity error measures nonlinearity after a reference line is chosen, such as an endpoint line, zero-based line or best-fit straight line. It is different from bias because the error changes with input rather than acting as one constant offset. It is important for sensors, ADCs, calibration curves, optical detectors, load cells, wind-tunnel balances and biomedical instruments.

Linearity error is the deviation between a measurement system and the straight-line relation used to represent it over a stated range. It answers a practical question: is a linear calibration model good enough for the engineering decision?

The term is only meaningful after the reference line is defined. Endpoint linearity, zero-based linearity and best-fit straight-line linearity can produce different numbers from the same calibration data.

Linear Reference

For a linear calibration model:

\hat{x}_i=a s_i+b

where (s_i) is sensor output, (a) is slope and (b) is offset. The linearity error at a calibration point can be written as:

e_{L,i}=x_i-\hat{x}_i

where (x_i) is the reference value at that point.

Maximum Linearity Error

The maximum absolute linearity error over the tested range is:

e_{L,max}=\max |e_{L,i}|

It is often reported as percent of full scale:

\displaystyle \%FS_L=100\frac{e_{L,max}}{FS}

where (FS) is the full-scale range or span used for the specification.

Worked Example

A pressure sensor has a 0 to 10 bar range. A linear fit predicts reference points with residuals:

-0.02,\ 0.04,\ 0.07,\ -0.03,\ -0.06\ \text{bar}

The maximum absolute residual is:

e_{L,max}=0.07\ \text{bar}

With (FS=10\ \text{bar}):

\displaystyle \%FS_L=100\frac{0.07}{10}=0.70\%

If the release requirement is less than 0.5 percent full scale, this linear model is not adequate without a narrower range, nonlinear model, correction table or larger uncertainty allowance.

Endpoint Versus Best Fit

Endpoint linearity forces the reference line through the endpoints. Best-fit straight-line linearity chooses the line that minimizes squared residuals:

S=\sum_{i=1}^{n}(x_i-\hat{x}_i)^2

Endpoint linearity is intuitive when the endpoints define calibration span. Best-fit linearity can make the maximum residual smaller, but it may leave endpoint errors that matter for release. The method must be stated before comparing specifications.

Point Selection

Linearity cannot be proven by two endpoints. Points should be dense enough to expose curvature where the device physics, electronics or installation are most likely to depart from a straight line. That may mean more points near zero, near saturation, near a valve transition, near optical detector overload or near a biomedical decision threshold.

Independent verification points matter because a model can look good on the same points used to fit it. If the calibration data define the line, a separate verification sweep gives stronger evidence that the line is valid for future measurements.

Relation to Bias

Bias can be a constant offset. Linearity error changes across the range. A simple error model is:

e(x)=b_0+b_1x+e_{NL}(x)

where (b_0) is offset bias, (b_1x) is gain-related error and (e_{NL}(x)) is nonlinear residual error. Treating nonlinear error as one bias correction can improve one part of the range while making another part worse.

Relation to ADC Linearity

In mixed-signal systems, integral nonlinearity and differential nonlinearity describe code-level ADC behavior. Those metrics are related, but a complete measurement system can also have sensor, amplifier, reference, input-settling, saturation or firmware-scaling nonlinearity.

For engineering release, the useful question is not only whether the ADC is linear. It is whether the complete measurement chain is linear enough for the converted engineering quantity.

Model Decision

When linearity error is too large, the options are not all equivalent. A narrower range may be acceptable for monitoring. A piecewise or polynomial calibration may be useful if the mechanism is stable and verified. A design change may be required if nonlinearity comes from saturation, common-mode limits, mechanical binding, optical overload or flow-regime change.

The selected response should match consequence. A lab demonstrator can often use a documented correction. A safety, compliance, clinical or production-release measurement usually needs independent verification and a stated residual uncertainty after correction.

Evidence for Validation

A useful linearity review states the reference line method, range, calibration points, independent verification points, residuals, full-scale basis, direction of approach, environmental condition, uncertainty, repeatability and whether the tested setup matches installed use.

Linearity should be checked at enough points to reveal curvature. A two-point calibration cannot prove linearity between endpoints. For hysteretic or saturating devices, increasing and decreasing sweeps may need separate checks.

Limits and Common Mistakes

Common mistakes include quoting a linearity percentage without saying endpoint or best-fit method, using percent of full scale when the decision is near zero, testing only endpoints, hiding curvature inside a single uncertainty term and assuming a high (R^2) value proves engineering accuracy.

Another mistake is improving the calibration fit while ignoring the physical cause. Saturation, common-mode limits, optical detector overload, bridge nonlinearity, fixture deflection or flow-regime change may need design changes rather than another polynomial.

REF

Disciplines