Skip to content

Statistical tests are not incoming inspection

Statistical test suites detect structure in output. They cannot detect predictability, because a well-built pseudorandom generator has no detectable structure and is entirely predictable to anyone holding its seed. That is not a flaw in the tests — it is what they are for. Somebody still has to inspect the goods coming in.

What the suites actually do

NIST STS, Dieharder, TestU01 and their relatives look for deviation from what a uniform random stream would produce: bias, correlation, periodicity, unexpected patterns. They are well designed and worth running.

What they answer is "does this output have a detectable statistical defect". A failure is informative — something is wrong. A pass tells you no defect was detected by those tests at that sample size.

Why a pass proves nothing about security

Take AES in counter mode with a fixed key. Every suite passes, at any sample size you care to run. The stream is completely determined. There is no test in any suite that distinguishes it from a physical source, because being indistinguishable from random is precisely the design goal of the cipher.

So statistical testing cannot separate a good physical source from a good deterministic one. Since the entire security difference between them lies in unpredictability, the tests are silent on the only question that matters.

The incoming inspection analogy

A factory does not establish that a shipment of parts is sound by looking at the finished product. It inspects the parts as they arrive, because defects can be masked by everything downstream.

Conditioning is a masking step of exactly this kind, and a very effective one. It makes under-entropic input indistinguishable from good input. Testing after conditioning is inspecting the finished product and concluding the parts were fine.

What to do instead

Test the raw source output, before conditioning, where structure is still visible. Estimate min-entropy from those samples using the SP 800-90B estimators. Measure the physical quantities the entropy model depends on, continuously, and alarm on departure.

Then keep running the statistical suites, because they still catch the catastrophic case cheaply. Just stop treating a pass as evidence of anything more than the absence of gross defect.