Combining Takuan with the TestGenerator random test data generator

If the invariants produced by Takuan are incorrect or insufficient, it is usually because there are not enough test cases to fully exercise the logic of the WS-BPEL composition.

One way to quickly expand the number of available test cases is to generate new test cases with random input data. Takuan does not implement this feature, but it can be combined with other tools to achieve this purpose.

Roughly, there are four major steps involved in the process:
  1. Revise the .bpts file so it is based on /test case templates/ and not concrete test cases.
  2. Write a TestSpec specification of the desired test data, and generate it.
  3. Run Takuan as usual on the modified .bpts file and examine the obtained invariants.

The following sections will provide some details on the first two steps. For the last step, nothing special is required: whether the .bpts uses test case templates or not is transparent to Takuan.

Writing a template-based test suite

The .bpts file has to be modified to read test data from a .vm file, instead of having concrete test cases. The concrete format for these templates is described in BPELUnit's documentation, and the UCASE research group has several examples in its composition repository.

The process cannot be fully automated, but we suggest the following approach:
  1. Enumerate all possible paths through the composition.
  2. Divide actions in each possible path over the partner traces.
  3. Use BPELUnit's "assume" attribute to only run the activities required by the test data.
  4. Write a first version of the .vm data file by translating the original test cases.
  5. Try out the .bpts manually and make sure it passes.
The UCASE research group has a prototype tool (BPTSGenerator) that attempts to automate this process and works on reasonably simple cases (such as the TacService sample composition), but it is not guaranteed to work for all compositions. It may be a good idea to try it first before attempting to do it manually. Using it is very simple:
  1. Download the latest -uberjar.zip from its Nexus repository.
  2. For UNIX-based systems, add the bpts-generator to your PATH and run it as follows:
    path/to/bpts-generator file.bpel > file.bpts
    
  3. Windows-based systems can simply run the provided .jar file:
    java -jar path/to/bpts-generator.jar file.bpel > file.bpts
    

If successful, this will produce a first version of the required .bpts, data.spec and data.vm files. You may want to customize them for your needs, however.

Writing a TestSpec specification for random test data generation

After obtaining the template-based .bpts test suite and a working .vm test data file, the next step is describing the structure of the test data in the .vm file using the TestSpec DSL. This will allow the TestGenerator tool to produce automatically new random test data.

To try out the .spec file, TestGenerator can be used as follows:
  1. Download the latest -dist.zip file from its Nexus repository.
  2. Run the testgenerator script to produce data from the .spec file (use the --help flag to list additional options):
    path/to/testgenerator file.spec > data.vm
    
  3. Examine the data.vm and revise the .spec if desired.

It is important to note that TestGenerator includes additional utilities for generating random XML documents or working with .vm files. It can also produce .spec files from WSDL fragments using the bundled specgenerator script.

Using automated seeding to combine random test data with useful constants

Pure random test generation may be inefficient in detecting certain corner cases. For instance, if the WS-BPEL composition has a x > 10000 check somewhere, we are not guaranteed to obtain a test case where x will be exactly 10000. Doing this requires looking at the code itself and taking advantage of these useful constants. The UCASE research group has implemented an "automated seeding" tool which combines the random test data generation of TestGenerator with an analysis of the .bpel code itself, and it can provide much better results in compositions that have this kind of check.

Using it consists of these steps:
  1. Download the latest -uberjar.zip from its Nexus repository.
  2. Run the test-generator-autoseed to produce data from the .bpel and .spec files (use the --help flag to list additional options):
    path/to/test-generator-autoseed file.bpel data.spec --output data.vm
    
  3. The tool will write one test case for each variable+constant combination it has found, and will print a short report of the constants it has tried out in each of the variables.

The tool can sometimes take advantage of additional information in the .bpts file through the -O flag, but it is not guaranteed to work for every .bpts file.