Statistical tests are not incoming inspection
Statistical test suites detect structure in output. They cannot detect predictability, because a well-built pseudorandom generator has no detectable structure and is entirely predictable to anyone holding its seed. That is not a flaw in the tests — it is what they are for. Somebody still has to inspect the goods coming in.
What the suites actually do
NIST STS, Dieharder, TestU01 and their relatives look for deviation from what a uniform random stream would produce: bias, correlation, periodicity, unexpected patterns. They are well designed and worth running.
What they answer is "does this output have a detectable statistical defect". A failure is informative — something is wrong. A pass tells you no defect was detected by those tests at that sample size.
Why a pass proves nothing about security
Take AES in counter mode with a fixed key. Every suite passes, at any sample size you care to run. The stream is completely determined. There is no test in any suite that distinguishes it from a physical source, because being indistinguishable from random is precisely the design goal of the cipher.
So statistical testing cannot separate a good physical source from a good deterministic one. Since the entire security difference between them lies in unpredictability, the tests are silent on the only question that matters.
The incoming inspection analogy
A factory does not establish that a shipment of parts is sound by looking at the finished product. It inspects the parts as they arrive, because defects can be masked by everything downstream.
Conditioning is a masking step of exactly this kind, and a very effective one. It makes under-entropic input indistinguishable from good input. Testing after conditioning is inspecting the finished product and concluding the parts were fine.
What to do instead
Test the raw source output, before conditioning, where structure is still visible. Estimate min-entropy from those samples using the SP 800-90B estimators. Measure the physical quantities the entropy model depends on, continuously, and alarm on departure.
Then keep running the statistical suites, because they still catch the catastrophic case cheaply. Just stop treating a pass as evidence of anything more than the absence of gross defect.
Keep reading
- Assurance and measurement
What is entropy assurance?
Entropy assurance is the combination of three things: a physical model of where unpredictability originates, measurement of the quantities that model depends on while the system runs, and evidence a third party can inspect. Any two without the third leaves a gap somebody will eventually ask about.
- Assurance and measurement
Health tests at runtime, what continuous monitoring looks like
SP 800-90B mandates two continuous health tests — repetition count and adaptive proportion — which detect a source that has stopped or stuck. Detecting a source that has merely degraded requires monitoring the physical quantities the entropy model depends on, at a cadence fast enough to act on.
- Assurance and measurement
What happens to an entropy source when voltage or temperature drifts
Entropy sources drift with temperature, supply voltage and age, and they can be pushed deliberately. Drift rarely shows in the output statistics — it shows in the physical quantities the entropy model depends on, which is why a certificate issued at evaluation cannot speak for a device in service years later.