Logic and Software Testing

Summary: Formal logic is what runs computers, but it is only a part of the logic used by a software tester. In this installment of his ongoing series on philosophy and software testing, Rick Scott explains.


QSM
Finally, we come around to a branch of philosophy in this series that most people will immediately associate with software. Logic is what runs computers, right? After all, they are logical machines. However, the software we are testing often seems to behave in a way that is anything but logical. So, what exactly is logic, and how is it relevant to software testers?

What Is Logic?

We bandy the word “logic” around a great deal, but our mental definition of it tends to be a bit amorphous. While we have a good general idea of what logic is, we tend to conflate it with “thinking" and “reason.”

Thinking is, broadly, any mental process. Reason is a kind of thinking—specifically, a mental process of arriving at an answer to a question or of deciding amongst choices. Logic tries to enumerate rules for reasoning—rules that allow us to reason in an orderly manner and help to ensure our conclusions are sound. As such, logic is invaluable for ensuring that we make robust decisions, that we are systematic in our consideration of difficult issues, and that we can perceive the flaws in erroneous arguments before they mislead us.

Formal Logic

Formal logic (also known as “mathematical logic”) is the flavor of logic that comes to mind when we speak about computers. It’s based on “propositions” or “Boolean variables” that can be either true or false. These are combined with logical connectives, such as AND (true when all constituent propositions are true) and OR (true when any constituent proposition is true). This, along with conditionals (IF ... THEN), is the foundation of how computers “make choices” or “reason” at both the hardware and software levels. For example:

IF the user is logged in,
AND the user has the correct permissions,
THEN show the user the configuration page.

IF this exception is uncatchable,
OR we haven't provided a way to handle it,
THEN crash.

Testers make use of formal logic in ways beyond understanding the machines we work with, particularly when it comes to testing strategy. Whether we explicitly say so or not, we often plan our test steps and allocate our time using formal logic: IF performance is slow AND it’s slow in browsers other than Internet Explorer 6, THEN I'll spend more time on performance testing; otherwise, I'll devote more time to inspecting the new UI. In those blessed (albeit infrequent) scenarios where we can enumerate all the possible inputs and outputs of a test scenario, we can use truth tables to make sure we don't miss anything and sometimes tools such as Boolean algebra or Karnaugh maps to separate the inputs that should affect the outcome from the ones that shouldn't.

Informal Logic

As software professionals, half of our job is interfacing with technology and half of it is interfacing with people. While it is laudable to have found an important bug buried deep in a application’s dark recesses, it does no good for the users of the software if you can’t convince management or the development team that it is worthwhile to fix it.

Informal logic (sometimes called "persuasive logic") is how we form arguments and attempt to reason with each other in our everyday lives. As opposed to the mathematical structure of formal logic, it deals with argument and reasoning in natural language. An argument consists of one or more premises, a line of reasoning, and a conclusion reached thereby. Informal logic outlines what constitutes a sound premise and what constitutes valid reasoning so that the conclusions we reach are both justified and defensible.

As one of the core skills involved in bug advocacy, informal logic is invaluable to testers. It lets you argue why something is in fact a bug and why that bug is important enough to fix. In a broader context, informal logic is the tool that lets you advocate for any of the myriad ideas and initiatives that you come up with. It may be clear to you that the test team needs more resources or that one feature should be prioritized over another, but neither course of action is likely to happen unless you can present a convincing argument for it to someone in management.

Logical Fallacies: Learning by Counterexample

While studying the attributes of a sound argument seems like the most straightforward way to get a grasp of informal logic, learning about common logical fallacies is a more accessible route to take. The language used to describe them is colourful (as opposed to the rather dry theory often associated with formal logic), and you can usually come up with instances of their occurring in your life or workplace.

To illustrate, let’s see what specious arguments we can muster to rally support for a statement that is patently false. I’m going to make the claim that pigs can fly. When someone points out that they can’t, I can try to discredit him or her personally: “Do you have a degree in biology? And, didn’t you just tell a lie the other day?” This is called ad hominem (Latin, “to the man”). I’m attacking the person presenting the argument instead of refuting the argument’s substance. Alternatively, I can reply with the following: “Why do you think animals can't fly? Birds fly all the time.” This is a straw man. I’ve misrepresented my opponent’s argument, then attacked that misrepresentation instead of the actual argument.

When it comes to affirming my own dubious position, I have many options:

* Argumentum ad populum (“appeal to the people”)—“Ten thousand people believe that pigs can fly, so surely they can.” This is claiming that something is true because many people think it is.
* Argumentum ad verecundiam (“appeal to authority”)—“Professor Bloggins says pigs can fly.”
* Argumentum ad consequentiam (“appeal to the consequences”)—“A world where pigs can fly would be an awesome place, so I'll believe that they can.”
* Argumentum ad baculum (“appeal to force” or, literally, “appeal to the stick”)—“If you don’t agree that pigs can fly, I’ll punch you in the nose.”
* Argumentum ad nauseam—Finally, ridiculous though my case may be, if I continue pressing it long enough, everyone else will get sick of arguing with me.

Setting aside the arguments of questionable relevance we’ve made in favor of flying pigs, the varied types of faulty reasoning also constitute fallacies of their own.

Offering premises that give no support to one’s conclusion is called a non sequitur (“It does not follow”). “His dissertation was excellent, since he served doughnuts at the seminar where he presented it.” Doughnuts are good in and of themselves, but they have nothing to do with whether the dissertation was good or poor.

Affirming the consequent is the fallacy that arises from the mistaken belief that “if X, then Y” also means “if Y, then X.” “When it snows, it’s cold outside. It’s cold outside, therefore it must be snowing.”

Finally, though begging the question is often taken to mean “"raising the question” nowadays, it refers to a circular argument in which the conclusion appears as one of the premises. In other words, one assumes at the outset what one is supposed to be proving. “Bob is always right.” Why? “Because Bob says so, and he’s never wrong.”

Knowledge of logical fallacies is a powerful tool for both improving your own reasoning and examining the arguments of others. Many of them seem ridiculous or absurd on their faces, but people actually do formulate and fall for arguments that use them! There are many more named logical fallacies. If this whets your appetite, Wikipedia’s list makes a good jumping-off point, and an introductory text on informal logic makes an even better one.

Conclusion

Testers need to reason with both computers and other people, and logic is at the heart of both. Whether analyzing the flow of a program, making a case for fixing an involved bug, or evaluating next quarter’s development plan, a solid grounding in logic will serve testers well.