PICT 3.3 User’s Guide
Jacek Czerwonka, Test Lead, Microsoft Corporation
Overview
Using PICT to Combine Test Case Parameters
Running PICT
Model Files
Model Sections
Simple Model
Pairwise and Higher-Order Generation
Sub-Models
Model Options
Constraints
Conditional Constraints
Unconditional Constraints (Invariants)
Parameter Types
Aliasing
Negative Testing
Weighting
Seeding
Output Randomization
Minimizing the Number of Test Cases
Case Sensitivity
Output Format
Warning Messages
All or no values satisfy relation…
Restrictive constraints. Output will not contain following values…
Sample Models
Complete Model File for the Volume Partitioning Example
Model to Test Hardware Configurations
Constraints Syntax Examples
Overview
The Pairwise Independent Combinatorial Testing tool (PICT) can help you efficiently design test cases and test configurations for software systems. With PICT, you can generate tests that are more effective than manually generated tests and create them in a fraction of the time required by hands-on test case design. PICT generates a compact set of parameter value choices that represent the test cases you should use to get comprehensive combinatorial coverage of your parameters.
Using PICT to Combine Test Case Parameters
PICT runs as a command line tool. You prepare a model file detailing the parameters of the interface (or set of configurations, or data) you want to test. PICT generates a compact set of parameter value choices that represent the test cases you should use to get comprehensive combinatorial coverage of your parameters.
For instance, if you wish to create a test suite for partition and volume creation, the domain can be described by the following parameters: Type, Size, File system, Format method, Cluster size, and Compression. Each parameter has a limited number of possible values, each of which is determined by its nature (for example, Compression can only be On or Off) or as an equivalence partition (such as Size).
Type: Primary, Logical, Single, Span, Stripe, Mirror, RAID-5
Size: 10, 100, 500, 1000, 5000, 10000, 40000
Format method: quick, slow
File system: FAT, FAT32, NTFS
Cluster size: 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536
Compression: on, off
There are over 4,700 possible combinations of these values. It would be very difficult to test all of them in a reasonable amount of time. Research shows that testing all pairs of possible values provides very good coverage and the number of test cases will remain manageable. For example, {Primary, FAT} is one pair and {10, slow} is another; a single test case can cover many pairs.
For the set of parameters shown above, PICT will produce 60 test cases. The last section of this guide contains a complete model for this volume/partition example.
Running PICT
PICT is a command-line tool that accepts a plain-text model file as an input and outputs a set of test cases.
Usage: pict model [options]
Options:
/o:N - Order of combinations (default: 2)
/d:C - Separator for values (default: ,)
/a:C - Separator for aliases (default: |)
/n:C - Negative value prefix (default: ~)
/e:file - File with seeding rows
/r[:N] - Randomize generation, N - seed
/c - Case-sensitive model evaluation
/s - Show model statistics
The PICT installer adds PICT to the path, so you can run it from any folder on your system. To see the PICT output for the example in the previous section, create the model file with any plain text editor, such as Windows® Notepad®, and run PICT with this command:
C:\YourFolder> pict ModelFile.txt
To create a tab-delimited output file, use command line output redirection.
C:\YourFolder> pict ModelFile.txt > OutputFile.txt
You can use Microsoft® Excel® to sort or filter PICT’s output. You either import the output file into Excel, or use an .xls extension when you create the file and open it directly in Excel.
C:\YourFolder> pict ModelFile.txt > OutputFile.xls
C:\YourFolder> OutputFile.xls
Model Files
Model Sections
A model consists of at least one, and at most, three sections:
parameter definitions
[sub-model definitions]
[constraint definitions]
Model sections should always be specified in the order shown above and cannot overlap. The parameters definition section comes first, followed by the optional sub-model and constraints sections, if you use them. Sections do not require any special separators between them. Empty lines can appear anywhere. You can include comments by prefixing lines with the “#”character.
Simple Model
To produce a very basic model file, list parameter names–each on a separate line–with their possible values delimited by commas:
: , , , ...
Example:
/#
/# This is a sample model for testing volume create/delete functions
/#
Type: Primary, Logical, Single, Span, Stripe, Mirror, RAID-5
Size: 10, 100, 500, 1000, 5000, 10000, 40000
Format method: quick, slow
File system: FAT, FAT32, NTFS
Cluster size: 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536
Compression: on, off
A comma is the default separator but you can specify a different one using /d: option.
Pairwise and Higher-Order Generation
By default, PICT generates a pairwise, or order two, suite of test cases–all pairs covered. You can set the order to a value larger than two using the option /o:. For example, if you specify /o:3, the resultant test cases will cover all triplets of values, producing a larger number of tests than the pairwise option, but potentially giving the test suite more coverage. The maximum order for a simple model is equal to the number of parameters, which will result in an exhaustive, all possible combinations, test suite. Following the same principle, specifying /o:1 will produce a test suite that covers all values only once (combinations of 1).