Orthogonal Array Testing Prince Bhanwra [801031024] Ravinder Dahiya [801031025] Presented By :

Contents Software Testing Conventional Test Cases Conventional Testing Issues Software Faults Why OATS..?? Terminology of OATS Examples OATS Advantages OATS Limitations Challenges Mixed Mode OA Available Test Automation Tools References Thapar University, Patiala 2

Testing Software Testing Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Conventional / Exhaustive Testing A test approach in which the test suite comprises all combinations of input values and preconditions. Thapar University, Patiala 3

Conventional Test Cases Thapar University, Patiala 4 Example: If we have three variables (A,B,C), each can have 3 values say (Red, Green, and Blue). The possible combinations in conventional test cases would be 27 i.e. 33

Conventional Testing Issues Conventional Test Cases: Variables:3 Input: 4 Possible cases: 81=34 Variables: 3 Input: 5 Possible Cases: 243 = 35 … Variables: 5 Input: 5 Possible Cases: 3125 = 55 Thapar University, Patiala 5 Orthogonal Test Cases Variables:3 Input: 4 Possible cases: 9 Variables: 3 Input: 5 Possible Cases: 11 … Variables: 5 Input: 5 Possible Cases: 21

Software Faults Thapar University, Patiala 6 Region Faults. Isolated Faults. Orthogonal Array based testing is highly effective for the detection of region faults with a relatively small number of tests.

Why Orthogonal Array Testing (OATS)..?? Systematic, statistical way to test pair-wise interactions. Interactions and integration points are a major source of defects. Most defects arise from simple pair-wise interactions. “When the background is blue and the font is Arial and the layout has menus on the right and the images are large and it’s a Thursday then the tables don’t line up properly.” Exhaustive testing is impossible. Execute a well-defined, concise set of tests that are likely to uncover most (not all) bugs. Orthogonal approach guarantees the pair-wise coverage of all variables. Thapar University, Patiala 7

Terminology for working with OA’s OA’s are commonly represented as : LRuns(LevelsFactors) or OA(Runs, Factors, Levels, Strength) or OAλ(Runs(N); Factors(k), Levels(v), Strength(t)) is an N × k array on v symbols such that every N × t sub-array contains all tuples of size t from v symbols exactly λ times. Runs (N) – Number of rows in the array, which translates into the number of Test Cases that will be generated. Factors (k) – Number of columns in the array, which translates into the maximum number of variables that can be handled by the array. Levels (v) – Maximum number of values that can be taken on by any single factor. Strength (t) – The number of columns it takes to see all the possibilities equal number of times. ƒ λ -1 for software testing and is often omitted Thapar University, Patiala 8

Example 1 Thapar University, Patiala 9 A Web Page has three distinct sections (Top, Middle, Bottom) that can be individually shown or hidden from user No.of Factors=3 (Top,middle,Bottom) No.of Levels =2 (Hidden or shown) Array Type =L4(2 3) or OA(4,3,2,2) If we go for exhaustive testing we need :2 x 2 x 2 = 8 Test Cases OA(Runs, Factors, Levels, Strength)

Example 1 (contd..) Thapar University, Patiala 10 Fixed Level Array: L4(2 3) L423 – OA with 4 Runs 3 factors with 2 levels The Four Test Scenarios (4 Vs. 8) 1 - Display the home page and hide all sections. 2 - Display the home page and show all but the Top section. 3 - Display the home page and show all but the Middle section. 4 - Display the home page and show all but the Bottom section.

Example 2 Thapar University, Patiala 11 Example: If we have three variables (A,B,C), each can have 3 values say (Red, Green, and Blue). The possible combinations in OATS test cases would be 9.

OATS advantage to select a test set: Guarantees testing the pair-wise combinations of all the selected variables. Creates an efficient and concise test set with many fewer test cases than testing all combinations of all variables. Creates a test set that has an even distribution of all pair-wise combinations. Exercises some of the complex combinations of all the variables. Is simpler to generate and less error prone than test sets created by hand. Thapar University, Patiala 12

OATS basic fault model: Interactions and integrations are a major source of defects. Most of these defects are not a result of complex interactions such as “When the background is blue and the font is Arial and the layout has menus on the right and the images are large and it’s a Thursday then the tables don’t line up properly.” Most of these defects arise from simple pair-wise interactions such as “When the font is Arial and the menus are on the right the tables don’t line up properly.” Thapar University, Patiala 13

OATS basic fault model (contd..): With so many possible combinations of components or settings, it is easy to miss one. Randomly selecting values to create all of the pair-wise combinations is bound to create inefficient test sets and test sets with random, senseless distribution of values. It covers 100% (9 of 9) of the pair-wise combinations, 33% (9 of 27) of the three-way combinations and 11% (9 of 81) of the four-way combinations. Thapar University, Patiala 14

The Challenges The number of tests may increase rapidly as the degree of interactions increases ‰ Test generation algorithms must be more sensitive in terms of both time and space requirements ‰ The need for test automation becomes even more serious. Impractical to manually execute and inspect the results of a large number of test runs However, on the Web there are pre-calculated orthogonal array tables for certain numbers of variables and values. Thapar University, Patiala 15

Mixed Mode OA Thapar University, Patiala 16

Available Test Automation Tools Thapar University, Patiala 17

References http://csrc.nist.gov/groups/SNS/acts/documents/ipo-nist.pdf http://csrc.nist.gov/groups/SNS/acts/documents/aberdeen.ppt http://ocw.mit.edu/courses/aeronautics-and-astronautics/16-881-robust-system-design-summer-1998/lecture-notes/l3_matrix_experiments4.pdf http://ocw.mit.edu/courses/aeronautics-and-astronautics/16-881-robust-system-design-summer-1998/lecture-notes/l8_orth_arrays.pdf http://delivery.acm.org/10.1145/1290000/1283384/p1-abam.pdf?key1=1283384&key2=5753427921&coll=DL&dl=ACM&CFID=9321178&CFTOKEN=33991028 http://www.developsense.com/pairwiseTesting.html http://learnsoftwareprocesses.com/2007/08/21/orthogonal-array-testing-strategy-oats/comment-page-1/#comment-7394 http://www.vietnamesetestingboard.org/zbxe/?document_srl=219676 http://www.pairwise.org/tools.asp Thapar University, Patiala 18

Queries..? Thapar University, Patiala 19

Thank you !! Thapar University, Patiala 20