Tests must document what they are testing for
Wednesday, July 19th, 2023 16:27While I worked at Google I spent a lot of time doing testing and code review. One of the things I learned from this is:
A test must document what it is testing.
This doesn’t have to be an explicit comment (it might be adequately explained by the name of the test, or the code in it), but either way, it should be obvious to maintainers what the goal of the test is — what property the code under test should have. “This test should continue to pass” is not a sufficient explanation of the goal. Why? Consider these scenarios:
A test failed. Does that mean…
…the test successfully detected the loss of a desired behavior? (This is the ideal case, that we hope to enable by writing and running tests.)
…the test incidentally depended on something that is changing but is not significant, such as ordering of something unordered-but-deterministic, or the exact sequence of some method calls? (Ideally, test assertions will be written precisely so that they accept everything that meets requirements, but this is not always feasible or correctly anticipated.)
…the test incidentally depended on something that is being intentionally changed, but it wasn’t intending to test that part of the system, which has its own tests?
A test no longer compiles because you deleted a function (or other declaration/item/symbol) it used. Does that mean…
…the test demonstrates a reason why you should not delete that function?
…the test needs to be rewritten to test the thing it is about, without incidentally using that function?
…the function is necessary to perform this test, so it should continue to exist but be private?
If tests specify the property they are looking for, then maintainers can quickly decide how to respond to a test failure — whether to count it a bug in the code under test, or to change or delete the test.
If they do not, then maintainers will waste time keeping the code conformant to obsolete requirements, or digging through revision history to determine the original intent.