Tests must document what they are testing for

Wed, 19 Jul 2023 23:40:09 GMT

While I worked at Google I spent a lot of time doing testing and code review. One of the things I learned from this is:

A test must document what it is testing.

This doesn’t have to be an explicit comment (it might be adequately explained by the name of the test, or the code in it), but either way, it should be obvious to maintainers what the goal of the test is — what property the code under test should have. “This test should continue to pass” is not a sufficient explanation of the goal. Why? Consider these scenarios:

A test failed. Does that mean…
- …the test successfully detected the loss of a desired behavior? (This is the ideal case, that we hope to enable by writing and running tests.)
- …the test incidentally depended on something that is changing but is not significant, such as ordering of something unordered-but-deterministic, or the exact sequence of some method calls? (Ideally, test assertions will be written precisely so that they accept everything that meets requirements, but this is not always feasible or correctly anticipated.)
- …the test incidentally depended on something that is being intentionally changed, but it wasn’t intending to test that part of the system, which has its own tests?
A test no longer compiles because you deleted a function (or other declaration/item/symbol) it used. Does that mean…
- …the test demonstrates a reason why you should not delete that function?
- …the test needs to be rewritten to test the thing it is about, without incidentally using that function?
- …the function is necessary to perform this test, so it should continue to exist but be private?

If tests specify the property they are looking for, then maintainers can quickly decide how to respond to a test failure — whether to count it a bug in the code under test, or to change or delete the test.

If they do not, then maintainers will waste time keeping the code conformant to obsolete requirements, or digging through revision history to determine the original intent.

comments

end and beginning

Sat, 06 May 2023 21:47:54 GMT

One month ago, April 7, 2023, I left my job at Google.

At the time, I was working on Earth Engine — “a planetary-scale platform for Earth science data & analysis”, or if you're looking at it through my lens, a cloud computing service which interprets parallel, pure-functional programs interleaved with image processing and database queries. (It’s a really interesting system, and if you'd like to check it out yourself, it's free for non-commercial use. Play with big data and make interesting pictures!)

One of the elements of the plan I made for the future after this choice was that, this year, I would get to take some time to pursue my personal projects and interests unencumbered by “having a day job” or intellectual property contracts — and I have been doing that.

Now, as you can see from the fact that my blog is as silent the last month as it has been for the last two years, that didn't instantly bring back all the energy for blogging I had in 2012. But, step one is to break the awkward silence. Step two is to post about stuff I've been doing. Maybe someday I'll get to step “update my website”.

comments

Kevin Reid's blog

Tests must document what they are testing for

end and beginning