What decisions are you going to make with that information?

One of my beliefs about building reporting tools is that the information that is included in the report should be driven by the questions that you want to be able to answer when using the tool. If you don't include all the information that is needed to answer, then the tool won't be useful. If you add unneeded information you are A) doing extra work when you build it and B) cluttering up the page and distracting people from what really matters.

I was talking with colleague this week about our integrated build validation test (BVT) information page page. We have multiple BVT "tools", each of which potentially gets run against multiple platforms for each build. Currently we report pass/fail at the tool level. If any platform fails a given tool, that tool reports as "fail". If all the platforms for a given tool pass then that tool reports a pass. My colleague doesn't like this. He thinks we should report all results. He thinks it's important to know that the slam BVT (one of the tools) passed on five platforms but failed on a sixth. In general, I think he's correct. We do want to know that it's working on some platforms but not others. But on this particular web page, that information doesn't seem relevant and I've never quite understood why he feels so strongly about it.

As we were digging into this it seemed that his core concern was that we were "making decisions" with incomplete information.

Certainly that sounds like a bad idea. But is it really? Not all information is relevant to a decision. In court, the most important issue to be decided is sometimes whether or not a particular piece of information can be put before a jury. If the judge rules that it is irrelevant to the question the jury is being asked to answer, then you are not allowed to present it. Even though that leads to decisions being made with incomplete information.

In this instance, what decisions are we making? The decisions we might make based on the BVT info page are "do we move forward with this build now, do we not move forward this build, or do we need to investigate and get more information before we decide?"

The BVT info page will show that either A) all of the BVT tests have passed against this build, or B) at least one test is failed. If all of the tests have passed, we pretty much always decide to move forward with the build. If most or all of the BVT tools have failed, we usually decide not to move forward with the build. Finally, if one or two of the BVT tools report failure we conclude that we need further investigation before we make the decision.

So how would knowing that for a given tool some platforms past and other platforms didn't change our decision? As far as I can tell it wouldn't. If it all passes, this problem doesn't apply. If most of the tools of failed on at least one platform that strongly suggests the build isn't solid enough to go forward. And if a particular tool is failing on we are going to want to investigate. And if the tool is failing on some platforms but not others, we are still going to want to investigate it.

For some decisions, like whether or not to send it off to developer to look at or have a tester look at it, or deciding which developer or tester to have investigate the issue, knowing that the tool failed on some platforms and passed on others is significant. But for the decision is actually being made from this particular report, I don't think it is. So including that information would just clutter up the report without adding value.