Skip to content

Commit

Permalink
Release 4.0.0 (#268)
Browse files Browse the repository at this point in the history
  • Loading branch information
@kerrykimbrough
kerrykimbrough authored Jan 6, 2023
1 parent 37e53de commit 294bf1f
Show file tree
Hide file tree
Showing 23 changed files with 3,093 additions and 1,709 deletions.
62 changes: 40 additions & 22 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
# Tcases: A Model-Based Test Case Generator #

[![Maven](https://img.shields.io/badge/maven-3.8.6-green.svg)](https://search.maven.org/search?q=tcases-shell)
[![Javadoc](https://img.shields.io/badge/javadoc-3.8.6-green.svg)](https://javadoc.io/doc/org.cornutum.tcases/tcases-shell)
[![Maven](https://img.shields.io/badge/maven-4.0.0-green.svg)](https://search.maven.org/search?q=tcases-shell)
[![Javadoc](https://img.shields.io/badge/javadoc-4.0.0-green.svg)](https://javadoc.io/doc/org.cornutum.tcases/tcases-shell)

## What's New? ##
* The latest version ([Tcases 3.8.6](ReleaseNotes.md#386)) is now available at the Maven Central Repository.
* The latest version ([Tcases 4.0.0](ReleaseNotes.md#400)) is now available at the Maven Central Repository.
See [*How To Download Tcases*](HowToDownload.md) for download instructions.

* Tcases 3.8.6 provides improvements to Tcases for OpenAPI. See the [release notes](ReleaseNotes.md#386) for details.
* With Tcases 4.0.0, JSON becomes the preferred format for all Tcases documents. Also new in this release:
[schema-based value definitions](./Tcases-Guide.md#defining-value-schemas) and the
[`tcases-copy`](./Tcases-Guide.md#copying-a-tcases-project) command.

* Having trouble with Tcases? Check out [these tips](./Troubleshooting-FAQs.md).

Expand All @@ -16,40 +18,56 @@
## What Does It Do? ##

Tcases is a tool for designing tests. It doesn't matter what kind of system you are testing -- UI, command line,
[REST-ful API](tcases-openapi/README.md#tcases-for-openapi-from-rest-ful-to-test-ful), or backend.
Nor does it matter what level of the system you are testing -- unit, subsystem, or full system. You can use Tcases to design your tests in any of these situations. With Tcases, you define the input space for your system-under-test and the level of coverage that you want. Then Tcases generates a minimal set of test cases that meets your requirements.
[REST-ful API](tcases-openapi/README.md#tcases-for-openapi-from-rest-ful-to-test-ful), or backend. Nor does it matter
what level of the system you are testing -- unit, subsystem, or full system. You can use Tcases to design your tests in any of
these situations. With Tcases, you define the input space for your system-under-test and the level of coverage that you
want. Then Tcases generates a minimal set of test cases that meets your requirements.

Tcases is primarily a tool for black-box test design. For such tests, the concept of "coverage" is different from structural testing criteria such as line coverage, branch coverage, etc. Instead, Tcases is guided by coverage of the input space of your system.
Tcases is primarily a tool for black-box test design. For such tests, the concept of "coverage" is different from structural
testing criteria such as line coverage, branch coverage, etc. Instead, Tcases is guided by coverage of the input space of your
system.

Tcases gives you a way to define the input space for your system in a form that is concise but comprehensive. Then Tcases allows you to control the number of test cases in your sample subset by specifying the level of coverage you want. You can start with a basic level of coverage, and Tcases will generate a small set of test cases that touches every significant element of the input space. Then you can improve your tests by selectively adding coverage in specific high-risk areas. For example, you can specify pairwise coverage or higher-order combinations of selected input variables.
Tcases gives you a way to define the input space for your system in a form that is concise but comprehensive. Then Tcases allows
you to control the number of test cases in your sample subset by specifying the level of coverage you want. You can start with a
basic level of coverage, and Tcases will generate a small set of test cases that touches every significant element of the input
space. Then you can improve your tests by selectively adding coverage in specific high-risk areas. For example, you can specify
pairwise coverage or higher-order combinations of selected input variables.

## How Does It Work? ##

First, you create a system input definition, a document that defines your system as a set of functions. For each system
function, the system input definition defines the variables that characterize the function input space. If you are testing a Web
service API, you can even [generate a system input definition automatically](tcases-openapi/README.md#tcases-for-openapi-from-rest-ful-to-test-ful) from an OpenAPI definition.

Then, you can create a generator definition. That's another document that defines the coverage you want for each system function. The generator definition is optional. You can skip this step and still get a basic level of coverage.

Finally, you run Tcases. Tcases is a Java program that you can run from the command line or using the [Tcases Maven Plugin](http://www.cornutum.org/tcases/docs/tcases-maven-plugin/). The command line version of Tcases comes with built-in support for running using a shell script or an ant target. Using your input definition and your generator definition, Tcases generates a system test definition. The system test definition is a document that lists, for each system function, a set of test cases that provides the specified level of coverage. Each test case defines a specific value for every function input variable. Tcases generates not only valid input values that define successful test cases but also invalid values for the tests cases that are needed to verify expected error handling.

Of course, the system test definition is not something you can execute directly. (Unless it was [derived automatically from an
OpenAPI definition](tcases-openapi/README.md#how-do-you-run-generated-api-test-cases)!) But it follows a well-defined schema, which
means you can use a variety of transformation tools to convert it into a form that is suitable for testing your system. For
example, Tcases comes with a built-in transformer that converts a system test definition into a Java source code template for a
JUnit or TestNG test class.
service API, you can even [generate a system input definition automatically](tcases-openapi/README.md#tcases-for-openapi-from-rest-ful-to-test-ful)
from an OpenAPI definition.

Then, you can create a generator definition. That's another document that defines the coverage you want for each system
function. The generator definition is optional. You can skip this step and still get a basic level of coverage.

Finally, you run Tcases. Tcases is a Java program that you can run from the command line or using the
[Tcases Maven Plugin](http://www.cornutum.org/tcases/docs/tcases-maven-plugin/). The command line version of Tcases comes with built-in
support for running using a shell script or an ant target. Using your input definition and your generator definition, Tcases
generates a system test definition. The system test definition is a document that lists, for each system function, a set of test
cases that provides the specified level of coverage. Each test case defines a specific value for every function input
variable. Tcases generates not only valid input values that define successful test cases but also invalid values for the tests
cases that are needed to verify expected error handling.

Of course, the system test definition is not something you can execute directly. (Unless it was
[derived automatically from an OpenAPI definition](tcases-openapi/README.md#how-do-you-run-generated-api-test-cases)!)
But it follows a well-defined schema, which means you can use a variety of transformation tools to convert it into a form that
is suitable for testing your system. For example, Tcases comes with a built-in transformer that converts a system test
definition into a Java source code template for a JUnit or TestNG test class.

## Get Started! ##

* **The Lowdown**
* [Tcases: The Complete Guide](http://www.cornutum.org/tcases/docs/Tcases-Guide.htm)
* [Tcases: The JSON Guide](http://www.cornutum.org/tcases/docs/Tcases-Json.htm): A companion to _The Complete Guide_ adding info specific to JSON
* [Tcases: The Complete Guide](./Tcases-Guide.md#tcases-the-complete-guide)
* [Tcases for OpenAPI](tcases-openapi/README.md#tcases-for-openapi-from-rest-ful-to-test-ful): Testing a REST-ful API? Generate test cases directly from your OpenAPI v3 definition.
* [The Tcases Maven Plugin](http://www.cornutum.org/tcases/docs/tcases-maven-plugin/)

* **Helpful Guides**
* [How To Download Using Maven](HowToDownload.md)
* [Troubleshooting FAQ](./Troubleshooting-FAQs.md)
* [How To Setup a Tcases Web Service](./Tcases-Web-Service.md)
* [Troubleshooting FAQ](./Troubleshooting-FAQs.md#troubleshooting-faqs)
* [Release Notes](ReleaseNotes.md)

* **More Info**
Expand Down
15 changes: 15 additions & 0 deletions ReleaseNotes.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,20 @@
# Release Notes #

## 4.0.0 ##

With this major release, JSON becomes the preferred format for all Tcases documents. From now on, unless otherwise stated, new
features will be supported only in JSON documents, although existing XML documents will continue to be supported as before.
[_Tcases: The Complete Guide_](./Tcases-Guide.md#tcases-the-complete-guide) has been updated accordingly.

This release also introduces an important new feature: [schema-based value definitions](./Tcases-Guide.md#defining-value-schemas).
Using standard JSON Schema keywords, you have the option to create a more concrete definition of a value class, which Tcases can
use to automatically generate test cases with actual representative values. You can even use a schema to generate all of the
value definitions for a variable automatically.

This release also introduces a new shell command -- `tcases-copy` -- and its Maven counterpart -- `tcases:copy`. You can use
[`tcases-copy`](./Tcases-Guide.md#copying-a-tcases-project) to copy a Tcases project to a new location or to convert it from XML to JSON.


## 3.8.6 ##

This release provides the following improvements to Tcases for OpenAPI.
Expand Down
192 changes: 192 additions & 0 deletions Schema-Keywords.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,192 @@
# Schema Keywords #

## Introduction ##

You can use _schema keywords_ to define the [value classes](Tcases-Guide.md#defining-value-schemas) for a
[variable definition](Tcases-Guide.md#defining-input-variables) in your Tcases
[input model](Tcases-Guide.md#modeling-the-input-space).

Tcases uses a [subset](#compared-to-json-schema) of the standard [JSON Schema Validation vocabulary](https://json-schema.org/draft/2020-12/json-schema-validation.html),
as described in the following sections.

## Generic Keywords ##

* `type`

The value of this keyword is a string that identifies the type of values in this value class. The `type` must one of the following values.

* "array"
* "boolean"
* "integer"
* "number"
* "string"

The `type` keyword is optional. If undefined, the expected value type will be derived from other schema keywords.

* `const`

The value of this keyword is single value of any type -- array, boolean, integer, number, string -- or `null`.
`const` identifies a value class containing exactly one member. If defined, any other type-specific keyword is ignored.

* `enum`

The value of this keyword is an array of unique values of any type -- array, boolean, integer, number, string -- or
`null`. All non-null values in the `enum` array must have the same type. `enum` identifies a value class containing only the
specified members. If defined, any other type-specific keyword is ignored.

* `format`

The value of this keyword is a string that identifies a specific form for values in this class. The `format` keyword is
optional and it can have any value, although certain values have standard definitions. (See [_Supported Formats_](#supported-formats).)


## String Keywords ##

The following optional keywords apply to values of type "string". If present, they imply an expected type of "string" for all
members of this value class.

* `maxLength`

The value of this keyword is an non-negative integer that defines an inclusive upper bound for the length of a value
string. All members of this value class must have a length less than or equal to the given value.

* `minLength`

The value of this keyword is an non-negative integer that defines an inclusive lower bound for the length of a value
string. All members of this value class must have a length greater than or equal to the given value.

* `pattern`

The value of this keyword is a string containing a valid regular expression (as defined by the
[ECMAScript](https://www.ecma-international.org/ecma-262/#sec-patterns) standard). If defined, this keyword
specifies that all members of this value class must match the given regular expression.


## Number Keywords ##

The following optional keywords apply to values of type "number" or "integer". If present, they imply an expected type of "number" for all
members of this value class, unless `"type": "integer"` is also specified.

* `maximum`

The value of this keyword is a number that defines an inclusive upper bound for members of this value class, which must be
less than or equal to the given value.

* `minimum`

The value of this keyword is a number that defines an inclusive lower bound for members of this value class, which must be
greater than or equal to the given value.

* `exclusiveMaximum`

The value of this keyword is a number that defines an exclusive upper bound for members of this value class, which must be
strictly less than the given value.

* `exclusiveMinimum`

The value of this keyword is a number that defines an exclusive lower bound for members of this value class, which must be
strictly greater than the given value.

* `multipleOf`

The value of this keyword must be a positive number. If defined, division by this number must produce an integer result for
all members of this value class.

## Array Keywords ##

The following optional keywords apply to values of type "array". If present, they imply an expected type of "array" for all
members of this value class.

* `maxItems`

The value of this keyword is a non-negative integer that defines an inclusive upper bound for the size of arrays in this
value class.

* `minItems`

The value of this keyword is a non-negative integer that defines an inclusive lower bound for the size of arrays in this
value class.

* `uniqueItems`

The value of this keyword is a boolean. If `true`, elements must be unique for any array in this value class. Otherwise, if
false or undefined, member arrays can contain duplicate elements.

* `items`

The value of this keyword is a schema -- an object containing only schema keywords. If defined, each element of an array
in this value class must belong to the specified value class. Otherwise, array elements can be any value, including `null`.

## Supported Formats ##

Tcases provides special handling for the following values for the `format` keyword. Any other value is accepted but ignored. In any case, the
`format` value is propagated to generated test definitions in the form of a [variable binding annotation](Tcases-Guide.md#variable-binding-annotations).

* "int32"

This format applies to values of type "integer". If defined, a member of the value class must be a 32-bit integer.

* "int64"

This format applies to values of type "integer". If defined, a member of the value class must be a 64-bit integer.

* "date-time"

This format applies to values of type "string". If defined, a member of this class must represent a valid date-time value as
specified in [RFC3339](https://www.rfc-editor.org/info/rfc3339).

* "date"

This format applies to values of type "string". If defined, a member of this class must represent a valid date value as
specified in [RFC3339](https://www.rfc-editor.org/info/rfc3339).

* "email"

This format applies to values of type "string". If defined, a member of this class must represent a valid email address as
specified in [RFC5321](https://www.rfc-editor.org/info/rfc5321).

* "uuid"

This format applies to values of type "string". If defined, a member of this class must represent a valid UUID as
specified in [RFC4291](https://www.rfc-editor.org/info/rfc4291).



## Compared To JSON Schema ##

In general, Tcases schema keywords conform to the standard [JSON Schema Validation vocabulary](https://json-schema.org/draft/2020-12/json-schema-validation.html),
with the following exceptions.

* **`object` type not supported**

In a Tcases input model, a complex input composed of multiple parts is modeled as a [variable set](Tcases-Guide.md#defining-variable-sets).
Schemas apply only to the values of the individual variable definitions within a variable set. Consequently, there is no place for an
`object` schema.

* **Keywords for applying subschemas not supported**

Keywords that combine multiple subschemas -- such as `allOf`, `anyOf`, `not`, etc. -- or that apply subschemas conditionally
-- such as `if`, `contains`, etc -- are not supported.

* **No multi-type value classes**

Although the JSON Schema standard allows a schema to validate multiple types of value instances, this is not supported in a
Tcases schema. The members of a value class must all have the same type. When no `type` keyword is given, the value type is
defined implicitly by the type-specific keywords that are specified. It is an error to specify keywords that imply different
types. For example, a schema that defines both `minLength` (which implies the "string" type) and `maxItems` (which implies
the "array" type) is invalid.

It's possible for a schema to contain no type-specific keywords, but this is valid only in a default schema supplied by a
variable definition. Ultimately, the effective schema for any value definition must specify exactly one type.

* **`null` is a value, not a type**

In a Tcases input model, `null` is treated much like it is in Java and other programming languages -- `null` is a special value that can
be assigned to a value of any type. This is different than the way JSON Schema handles `null` as a distinct type. In Tcases, any input
variable can be "nullable", depending on how it is modeled.

* **Not all defined formats are recognized**

Tcases recognizes [some](#supported-formats) but not all of the
[`format` values defined by JSON Schema](https://json-schema.org/draft/2020-12/json-schema-validation.html#name-defined-formats).

Loading

0 comments on commit 294bf1f

Please sign in to comment.