11.1.1.1. Test Cases and Suites¶

OK, if the goal is to execute a program (or part of one) with the intent of finding an error, then let’s run it! But what input value(s) or user interaction sequence(s) should we use?

As an example, suppose you are testing something very simple, like a single function called isEven() that takes an integer value and returns true if the integer is even, or false if it is odd.

Remember that just running the program doesn’t really count as testing; instead, we must be trying to demonstrate failures of some kind. That means carefully planning out exactly how we are going to run the program–what input values or situations we will use, and what specific behavior or result we will be looking for.

A test case is a single scenario under which the software will be run to attempt to demonstrate a failure, and typically has three basic parts.

1. First, a test case must define all of the input values, conditions, or variables used in the test. For example, we can imagine a scenario where we wish to test isEven() on the input value 2.

2. Second, a test case must define a procedure for exercising the software under test. In the case of isEven(), the procedure can be very simple: call the method with the given input value, and store the boolean result. If we were testing a larger software part or an entire application, however, we may need to perform a longer, more complicated series of actions to exercise it, and this procedure is part of the test case.

3. Third, a test case must define the expected behavior that should result when the software is exercised in this way. For example, if we call isEven() with a value of 2, we would expect it to produce the result true.

Further, a test case is worked out (and written down) before the test is carried out. That is, we figure out what we will do and what result we expect it to produce before running the test. That differentiates testing from simply “running the code to see what happens”.

Of course, it will probably take more than one test case to determine that some required behavior is fully satisfied. For example, an implementation of isEven() that always returns true, regardless of the input value, will pass the example test case described above. As a result, programmers typically design collections of test cases that all work together. A collection of test cases designed for exercising the same piece of software is called a test suite.

For example, one might design five test cases for isEven(): one test for a positive even integer, one for a positive odd integer, one for a negative even integer, one for a negative odd integer, and one for zero. Another person might come up with a different test suite for this method, but this group would have a good chance of successfully demonstrating most of the simple bugs that could occur in the method in question.

11.1.1.2. Good Test Cases¶

Even when testing a single method like isEven(), there are a huge number of test cases to choose from. For 32-bit integers, there are 2³² possible input values we could try. So why would we pick any one of these values over another to use in a test case? What makes a test case a “good” test case?

According to Glenford Myers, a good test case is one that:

• Has a high probability of finding an error. Since our goal is to demonstrate faults, we should pick test cases that we think have the greatest chance of succeeding.

• Is not redundant. If we already have a test case for the value 2, we don’t need another one. Further, why not use some other integer, like 147652? Well, is there any reason to think this test case will have a high probability of finding an error? If we already have other test cases that cover even numbers, why test this number as well? If it simply retests behavior that has already been adequately demonstrated, then it is redundant–it exercises behavior already tested by other test cases in our test suite.

• Is most “effective”. In other words, if there are several redundant test cases you might consider that all cover the same behavior, pick the test case that has the highest probability of finding errors. For example, if we want to test isEven() on an even negative integer, which value should we pick? Will any random value do? In this case, it might be better to pick the largest negative integer (-2³¹), since it will also check that the method works on an extreme boundary of the integer range.

• Is neither too simple nor too complex. Suppose you are testing a group of methods on a class, rather than just focusing on isEven(). You could have a test case that called every single method in the class in some order (and maybe call some methods more than once!). Such a test case might cover a lot of ground–all of the methods would have to work correctly on at least some test values in order for this test case to pass. However, if the test reveals a defect, where is it? It could be in any method, almost anywhere. This is an example of a test case that is too complex, since it involves too many behaviors all in the same series of actions. It would be better to use many smaller test cases, each focused on examining a single behavior, so that the test results would be more meaningful. Similarly, a test case that simply constructed the object and did nothing else might be too simple to reveal any defects. Strive for narrowly focused test cases that are still likely to reveal defects in the behavior they are exercising.

11.1.1.3. Software Testing Approaches¶

Clearly, exhaustive testing, where every possible execution sequence is tested, is impractical, even for very simple pieces of code like isEven(). There are just too many possibilities to try, and it takes too long. As a result, we need a way to select an appropriate set of test cases (a test suite) that has a high likelihood of revealing the most probable defects. Many different testing methods have been developed for devising test cases, all with the goal of helping you pick an effective test suite.

The two biggest groups of testing approaches are black-box testing approaches and white-box testing approaches, although others exist as well.

Testing activitives are also characterized according to the nature of the units being tested, and the focus of what you are trying to verify. Some of the most common testing activities are unit testing, integration testing, and system testing.

Black-box testing or functional testing is a term used for a family of testing approaches where the test cases are derived from a specification, an interface definition, or from a behavioral description. In using this strategy, the tester views the software under test as a “black box” whose behavior can only be determined by studying its inputs and the related outputs. For students, this is the approach you use when you think about writing tests using an assignment description.

In contrast, another family of testing approaches is called white-box testing, glass-box testing, or logic-driven testing. White-box testing approaches require the tester to examine the internal structure of the software under test. In using such a strategy, the tester derives test cases after reviewing the program’s internal logic and structure.

Unit testing is used to describe activities where one is testing a single software “unit” in isolation, independently of any of the other code being written. This is usually the technique you use when you are starting out, writing tests for one class at a time.

Integration Testing: Once individual program components or classes (units) have been tested, they must be integrated to create a partial or complete system. The integration process involves testing combinations of units working together for problems that come up with component interaction. The combination being tested may start out as small as two units interacting, and grow to be the full application. Usually, this is the time when pieces of a full system written by different developers are being tested together.

System testing is where an entire, fully integrated program is tested with respect to its typical working environment. This involves combining the code from all the developers into the full, complete software product.

11.1.1.4. Unit Testing in More Detail¶

Unit testing is used to describe activities where one is testing a single software “unit” in isolation, independently of any of the other code being written.

Exactly what unit means can vary from one programming language to another, from one programming paradigm to another, and from one organization to another, but is intended to mean a cleanly delimited, identifiable piece of software that can be executed independently. In Pascal, a “unit” is typically a procedure or a function. In object-oriented languages, a “unit” is often a single class, although sometimes it can be a single method. Unless otherwise specified, from here on out we will interpret “unit” to mean a single class in an object-oriented language, unit under test (UUT) to mean the class we are currently testing or writing tests for.

Unit testing is usually carried out by the programmer or programmers that wrote the unit under test (UUT), before the unit is combined with other pieces of software to form a larger application. The goal is to confirm to the greatest extent possible that the unit has no errors of its own before combining it with other software. That is because the smaller the piece of software you are testing, the easier it is to locate and remove defects that are revealed.

For students, it turns out that most of the testing you do on your own assignments is unit testing: testing individual classes in isolation to ensure they meet their individual design requirements.

11.1.1.5. Test Driven Developement (TDD)¶

Test-driven development (TDD) is a programming technique that involves constantly alternating between writing one or more small test cases, and then writing a small increment of code, so that you can gradually build up a working code base one piece at a time. There are three principal ideas behind TDD:

• Test first. That is, each time you are about to write some part of your solution, first write down the test cases that are necessary to confirm your solution works the way you want, and only then write the code. For this reason, TDD is also known as test-first coding.

• Write in tiny increments. Rather than writing large chunks of code at once, you should add new code in “baby steps”: one small method, or one small piece of a method at a time, writing a new test case or two for each small bit.

  For example, the isEven() discussed earlier is small enough that you could write test cases for it, then write the method body in one step (just one line of code is needed), and finally run your test cases and debug as necessary.

  However, if you were writing a method with more complex behavior, it can be far too complex to write in one step. Consider a method that takes three numbers representing the lengths of the three sides to a triangle, where the method is supposed to return whether the corresponding triangle is equilateral, isosceles, or scalene, while reporting that the numbers do not form a triangle if no triangle exists with those side lengths.

  If you are writing tests for this method, you will need to check many distinct types of situations. Does it handle zero-length sides? Does it handle negative numbers? Does it handle lengths that do not form any triangle? What about isosceles? Equilateral? Scalene? You can separate each of these “cases” or branches in your solution’s logic into a separate small increment. Write the test cases you want to have for zero length sides. Then just implement that part of the method and run your tests. Next, add test cases for negative numbers, add just the extra code for this case, and re-run all your tests. Proceed through the remaining cases one step at a time, writing a few tests and then writing the code that implements the corresponding behavior.

  Using case analysis–that is, breaking a problem down into two or more subproblems, and defining the conditions under which one or another of these subproblems applies–is a powerful problem-solving tool. It comes up all the time in computer science, and provides a neat way to break apart complicated methods into smaller steps that can be incrementally tested.

• “When the bar is green, the code is clean.” This maxim of TDD characterizes the third key idea: each time you add a small piece of code, you re-run all of the tests you have for the unit under development, and you do not move on to the next step until all of your tests pass 100%.

  In short, add a small number of tests first, then add the corresponding (small) piece of code, run all your tests, and debug any problems immediately. You’re never ready to move to the next coding step (or finished with your solution, or ready to contribute your code to an open source project, etc.) unless all your tests pass. Your test cases are your expression of what “correct behavior” is for your code. Thus, these test cases are your yardstick for measuring your success, and as you incrementally grow your test suite, you can see how close you are to completing all the required behavior.

  Most testing tools for automatically running your test cases will show you a progress bar as the tests run, and color it green as long as tests are successful, and red when any tests fail.

A successful test in traditional testing finds one or more defects. But in TDD, when a test fails you have made progress because you now know that you need to resolve the problem. TDD increases your confidence that your system actually meets the requirements defined for it and that your system actually works. It is said that you should “test with a purpose” and know why you are testing something and to what extent it needs to be tested. Also with TDD, when you achieve a well tested program then every single line of code is tested. In general, this is something that traditional testing does recommend, but does not guarantee.

A significant advantage provided by TDD is that it enables you (encourages you!) to take small steps when writing software. For instance, suppose you add a small piece of new code, compile, and test it. Sooner or later when you do this, one or more of your tests will fail because of one or more defects in your code. However, by proceeding in small steps, it is much easier to find and fix those defects. The problem is most likely in the tiny bit of code you just wrote, since all the other code passes all the other tests you’d previously written. If some previously working behavior breaks, again it is probably a result of interference caused by the newly added code. And bugs are so much easier to find if you have only written five new lines of code rather than five hundred, or five thousand.

Writing software tests for each method as you go is your best defense for confirming you understand what your code does, confirming your code behaves the way you intend, and finding problems as soon as possible so they won’t cause trouble later. The longer you put off testing, the harder it is to find problems, and the more bugs you’ll have to fix–if you let too many pile up, it gets increasingly challenging to get your code to work at all.