Can explain the need for strategies to combine test inputs
An SUT can take multiple inputs. You can select values for each input (using equivalence partitioning, boundary value analysis, or some other technique).
an SUT that takes multiple inputs and some values chosen as values for each input:
- Method to test:
calculateGrade(participation, projectGrade, isAbsent, examScore) - Values to test:
Input valid values to test invalid values to test participation 0, 1, 19, 20 21, 22 projectGrade A, B, C, D, F isAbsent true, false examScore 0, 1, 69, 70, 71, 72
Testing all possible combinations is effective but not efficient. If you test all possible combinations for the above example, you need to test 6x5x2x6=360 cases. Doing so has a higher chance of discovering bugs (i.e. effective) but the number of test cases can be too high (i.e. not efficient). Therefore, we need smarter ways to combine test inputs that are both effective and efficient.
Can explain some basic test input combination strategies
Given below are some basic strategies for generating a set of test cases by combining multiple test input combination strategies.
Let's assume the SUT has the following three inputs and you have selected the given values for testing:
SUT: foo(p1 char, p2 int, p3 boolean)
Values to test:
| Input | Values |
|---|---|
| p1 | a, b, c |
| p2 | 1, 2, 3 |
| p3 | T, F |
The all combinations strategy generates test cases for each unique combination of test inputs.
the strategy generates 3x3x2=18 test cases
| Test Case | p1 | p2 | p3 |
|---|---|---|---|
| 1 | a | 1 | T |
| 2 | a | 1 | F |
| 3 | a | 2 | T |
| ... | ... | ... | ... |
| 18 | c | 3 | F |
The at least once strategy includes each test input at least once.
this strategy generates 3 test cases.
| Test Case | p1 | p2 | p3 |
|---|---|---|---|
| 1 | a | 1 | T |
| 2 | b | 2 | F |
| 3 | c | 3 | VV/IV |
VV/IV = Any Valid Value / Any Invalid Value
The all pairs strategy creates test cases so that for any given pair of inputs, all combinations between them are tested. It is based on the observations that a bug is rarely the result of more than two interacting factors. The resulting number of test cases is lower than the all combinations strategy, but higher than the at least once approach.
this strategy generates 9 test cases:
see steps
Let's first consider inputs p1 and p2:
| Input | Values |
|---|---|
| p1 | a, b, c |
| p2 | 1, 2, 3 |
These values can generate (a,1)(a,2)(a,3)(b,1)(b,2),...3x3=9 combinations, and the test cases should cover all of them.
Next, let's consider p1 and p3.
| Input | Values |
|---|---|
| p1 | a, b, c |
| p3 | T, F |
These values can generate (a,T)(a,F)(b,T)(b,F),...3x2=6 combinations, and the test cases should cover all of them.
Similarly, inputs p2 and p3 generates another 6 combinations.
The 9 test cases given below covers all those 9+6+6 combinations.
| Test Case | p1 | p2 | p3 |
|---|---|---|---|
| 1 | a | 1 | T |
| 2 | a | 2 | T |
| 3 | a | 3 | F |
| 4 | b | 1 | F |
| 5 | b | 2 | T |
| 6 | b | 3 | F |
| 7 | c | 1 | T |
| 8 | c | 2 | F |
| 9 | c | 3 | T |
A variation of this strategy is to test all pairs of inputs but only for inputs that could influence each other.
Testing all pairs between p1 and p3 only while ensuring all p3 values are tested at least once
| Test Case | p1 | p2 | p3 |
|---|---|---|---|
| 1 | a | 1 | T |
| 2 | a | 2 | F |
| 3 | b | 3 | T |
| 4 | b | VV/IV | F |
| 5 | c | VV/IV | T |
| 6 | c | VV/IV | F |
The random strategy generates test cases using one of the other strategies and then pick a subset randomly (presumably because the original set of test cases is too big).
There are other strategies that can be used too.
Can apply heuristic ‘each valid input at least once in a positive test case’
Consider the following scenario.
SUT: printLabel(fruitName String, unitPrice int)
Selected values for fruitName (invalid values are underlined ):
| Values | Explanation |
|---|---|
| Apple | Label format is round |
| Banana | Label format is oval |
| Cherry | Label format is square |
| Dog | Not a valid fruit |
Selected values for unitPrice:
| Values | Explanation |
|---|---|
| 1 | Only one digit |
| 20 | Two digits |
| 0 | Invalid because 0 is not a valid price |
| -1 | Invalid because negative prices are not allowed |
Suppose these are the test cases being considered.
| Case | fruitName | unitPrice | Expected |
|---|---|---|---|
| 1 | Apple | 1 | Print label |
| 2 | Banana | 20 | Print label |
| 3 | Cherry | 0 | Error message “invalid price” |
| 4 | Dog | -1 | Error message “invalid fruit" |
It looks like the test cases were created using the at least once strategy. After running these tests can we confirm that square-format label printing is done correctly?
- Answer: No.
- Reason:
Cherry-- the only input that can produce a square-format label -- is in a negative test case which produces an error message instead of a label. If there is a bug in the code that prints labels in square-format, these tests cases will not trigger that bug.
In this case a useful heuristic to apply is each valid input must appear at least once in a positive test case. Cherry is a valid test input and we must ensure that it appears at least once in a positive test case. Here are the updated test cases after applying that heuristic.
| Case | fruitName | unitPrice | Expected |
|---|---|---|---|
| 1 | Apple | 1 | Print round label |
| 2 | Banana | 20 | Print oval label |
| 2.1 | Cherry | VV | Print square label |
| 3 | VV | 0 | Error message “invalid price” |
| 4 | Dog | -1 | Error message “invalid fruit" |
VV/IV = Any Invalid or Valid Value VV=Any Valid Value
Can apply heuristic ‘no more than one invalid input in a test case’
Consider the test cases designed in [Heuristic: each valid input at least once in a positive test case].
| Case | fruitName | unitPrice | Expected |
|---|---|---|---|
| 1 | Apple | 1 | Print round label |
| 2 | Banana | 20 | Print oval label |
| 2.1 | Cherry | VV | Print square label |
| 3 | VV | 0 | Error message “invalid price” |
| 4 | Dog | -1 | Error message “invalid fruit" |
VV/IV = Any Invalid or Valid Value VV=Any Valid Value
After running these test cases can you be sure that the error message “invalid price” is shown for negative prices?
- Answer: No.
- Reason:
-1-- the only input that is a negative price -– is in a test case that produces the error message “invalid fruit”.
In this case a useful heuristic to apply is no more than one invalid input in a test case. After applying that, we get the following test cases.
| Case | fruitName | unitPrice | Expected |
|---|---|---|---|
| 1 | Apple | 1 | Print round label |
| 2 | Banana | 20 | Print oval label |
| 2.1 | Cherry | VV | Print square label |
| 3 | VV | 0 | Error message “invalid price” |
| 4 | VV | -1 | Error message “invalid price" |
| 4.1 | Dog | VV | Error message “invalid fruit" |
VV/IV = Any Invalid or Valid Value VV=Any Valid Value
Exercises
Can define test cases precisely
Applying the heuristics covered so far, we can determine the precise number of test cases required to test any given SUT effectively.
False
Explanation: These heuristics are, well, heuristics only. They will help you to make better decisions about test case design. However, they are speculative in nature (especially, when testing in black-box fashion) and cannot give you precise number of test cases.
Can apply multiple test input combination techniques together
Consider the calculateGrade scenario given below:
- SUT :
calculateGrade(participation, projectGrade, isAbsent, examScore) - Values to test: invalid values are underlined
- participation: 0, 1, 19, 20, 21, 22
- projectGrade: A, B, C, D, F
- isAbsent: true, false
- examScore: 0, 1, 69, 70, 71, 72
To get the first cut of test cases, let’s apply the at least once strategy.
Test cases for calculateGrade V1
| Case No. | participation | projectGrade | isAbsent | examScore | Expected |
|---|---|---|---|---|---|
| 1 | 0 | A | true | 0 | ... |
| 2 | 1 | B | false | 1 | ... |
| 3 | 19 | C | VV/IV | 69 | ... |
| 4 | 20 | D | VV/IV | 70 | ... |
| 5 | 21 | F | VV/IV | 71 | Err Msg |
| 6 | 22 | VV/IV | VV/IV | 72 | Err Msg |
VV/IV = Any Valid or Invalid Value, Err Msg = Error Message
Next, let’s apply the each valid input at least once in a positive test case heuristic. Test case 5 has a valid value for projectGrade=F that doesn't appear in any other positive test case. Let's replace test case 5 with 5.1 and 5.2 to rectify that.
Test cases for calculateGrade V2
| Case No. | participation | projectGrade | isAbsent | examScore | Expected |
|---|---|---|---|---|---|
| 1 | 0 | A | true | 0 | ... |
| 2 | 1 | B | false | 1 | ... |
| 3 | 19 | C | VV | 69 | ... |
| 4 | 20 | D | VV | 70 | ... |
| 5.1 | VV | F | VV | VV | ... |
| 5.2 | 21 | VV/IV | VV/IV | 71 | Err Msg |
| 6 | 22 | VV/IV | VV/IV | 72 | Err Msg |
VV = Any Valid Value VV/IV = Any Valid or Invalid Value
Next, we apply the no more than one invalid input in a test case heuristic. Test cases 5.2 and 6 don't follow that heuristic. Let's rectify the situation as follows:
Test cases for calculateGrade V3
| Case No. | participation | projectGrade | isAbsent | examScore | Expected |
|---|---|---|---|---|---|
| 1 | 0 | A | true | 0 | ... |
| 2 | 1 | B | false | 1 | ... |
| 3 | 19 | C | VV | 69 | ... |
| 4 | 20 | D | VV | 70 | ... |
| 5.1 | VV | F | VV | VV | ... |
| 5.2 | 21 | VV | VV | VV | Err Msg |
| 5.3 | 22 | VV | VV | VV | Err Msg |
| 6.1 | VV | VV | VV | 71 | Err Msg |
| 6.2 | VV | VV | VV | 72 | Err Msg |
Next, let us assume that there is a dependency between the inputs examScore and isAbsent such that an absent student can only have examScore=0. To cater for the hidden invalid case arising from this, we can add a new test case where isAbsent=true and examScore!=0. In addition, test cases 3-6.2 should have isAbsent=false so that the input remains valid.
Test cases for calculateGrade V4
| Case No. | participation | projectGrade | isAbsent | examScore | Expected |
|---|---|---|---|---|---|
| 1 | 0 | A | true | 0 | ... |
| 2 | 1 | B | false | 1 | ... |
| 3 | 19 | C | false | 69 | ... |
| 4 | 20 | D | false | 70 | ... |
| 5.1 | VV | F | false | VV | ... |
| 5.2 | 21 | VV | false | VV | Err Msg |
| 5.3 | 22 | VV | false | VV | Err Msg |
| 6.1 | VV | VV | false | 71 | Err Msg |
| 6.2 | VV | VV | false | 72 | Err Msg |
| 7 | VV | VV | true | !=0 | Err Msg |
Exercises
statements about test input combinations
Which of these contradict the heuristics recommended when creating test cases with multiple inputs?
(a) inputs.
Explanation: If you test all invalid test inputs together, you will not know if each one of the invalid inputs are handled correctly by the SUT. This is because most SUTs return an error message upon encountering the first invalid input.
Combine test inputs for the consume method
Apply heuristics for combining multiple test inputs to improve the E&E of the following test cases, assuming all 6 values in the table need to be tested. underlines indicate invalid values. Point out where the heuristics are contradicted and how to improve the test cases.
SUT: consume(food, drink)
| Test case | food | drink |
|---|---|---|
| TC1 | bread | water |
| TC2 | rice | lava |
| TC3 | rock | acid |