Friday, July 17, 2015

Pouring concrete

If there were 10 commandments of automated testing, number 1 must surely be "Thou shalt test what, not how".

Imagine if I wrote a sort routine, and further imagine I wrote a bubble sort in particular.
In a misguided attempt to test it, I decided I would make sure it did the correct swaps. So I come up with test data, and figure out what items get swapped, and then I modify my sort to pass in a "swap" delegate, so that I can intercept this and test that the correct swaps occur.

I run my tests, and I see that is it good. Each call to swap happens just as I expect, and I know exactly what my code is doing.

Only it is not good.

As my data sets increase, bubble sort struggles, and then I read about Quicksort, or Radix sort, and I decide to replace my bubble sort.

All my tests fail. They all need to be fixed by figuring out what swaps will now occur.
My tests have poured concrete on the implementation, and made it impossible to change without dynamiting the unit tests.

Let's rewind and put in the rule that tests must test results, not actions.
If I sort data, it must be sorted. I don't care how it sorts it. I just want it sorted.

My tests have less detail, and for sure, there could be some hidden bug, that still leaves my data sorted, but does not operate in manner I expected. But my data is sorted. Now I don't need that swap delegate. I pass in my data, I get back the sorted result, and I confirm that the sorted result is indeed sorted.

When I replace my bubble sort with a New and Better Sort, lo and behold, all my tests still run, and hopefully pass.

Test that the data is sorted, not how the data gets to be sorted.

Sadly many test tools facilitate the how testing. The ability to write A.CallTo(()=>swap(a,b)).MustHaveHappened; or some such thing is nothing other than pouring concrete deep and heavy all over the implementation. Avoid the temptation, the road to hell is paved with concrete.