Quantization

Quantization replaces a value with the nearest or otherwise selected value from a finite set of levels. In an analog-to-digital converter, a voltage range is divided into codes. With b bits, an ideal converter has:

possible output levels. The step size, often called one least significant bit, depends on reference range and coding scheme. Quantization error is the difference between the true input and the represented value.

Signal impact

For an ideal uniform quantizer with a sufficiently busy signal, quantization error is often modeled as noise distributed within plus or minus half a step. This model is useful but not universal. Small periodic signals, clipping, missing codes, nonlinear converters, and deterministic control loops can produce structured errors, limit cycles, or distortion rather than benign white noise.

Quantization interacts with sampling. Sampling sets when measurements are taken; quantization sets how finely each sampled value is represented. Anti-alias filtering can protect frequency content, but it does not remove quantization error. Increasing bit depth, scaling the input range, dithering, oversampling, and filtering can improve effective resolution.

Engineering use

In embedded systems, quantization affects sensor resolution, actuator commands, fixed-point controllers, lookup tables, digital filters, and communication payloads. In neural networks, reducing weights and activations to lower precision can shrink memory and accelerate inference, but may reduce accuracy or change failure modes.

Common mistakes

A common mistake is quoting bit depth without reference range, noise, linearity, and effective number of bits. Another is scaling a signal poorly so most codes are unused, wasting resolution. A good review states range, step size, rounding rule, saturation behavior, signed or unsigned format, noise floor, calibration, and whether quantization effects were tested in closed-loop operation.