A fully featured Gherkin parser and test runner. Works with Flutter and Dart 2.
This implementation of the Gherkin tries to follow as closely as possible other implementations of Gherkin and specifically Cucumber in it's various forms.
Available as a Dart package https://pub.dartlang.org/packages/gherkin Available as a Flutter specific package https://pub.dartlang.org/packages/flutter_gherkin which contains specific implementations to help instrument an application to test.
# Comment
Feature: Addition
@tag
Scenario: 1 + 0
Given I start with 1
When I add 0
Then I end up with 1
Scenario: 1 + 1
Given I start with 1
When I add 1
Then I end up with 2See https://docs.cucumber.io/gherkin/ for information on the Gherkin syntax and Behaviour Driven Development (BDD).
See example readme for a quick start guide to running the example features.
To get started with BDD in Dart the first step is to write a feature file and a test scenario within that.
First create a folder called test, within the folder create a folder called features, then create a file called calculator_can_add.feature.
Feature: Calculator
Tests the addition feature of the calculator
Scenario: Add two numbers
Given the numbers 1.5 and 2.1
When they are added
Then expected result is 3.6Now we have created a scenario we need to implement the steps within. Steps are just classes that extends from the base step definition class or any of its variations Given, Then, When, And, But.
Granted the example is a little contrived but is serves to illustrate the process.
To implement a step we have to create a simple step definition class.
import 'package:gherkin/gherkin.dart';
import '../worlds/custom_world.world.dart';
class GivenTheNumbers extends Given2WithWorld<num, num, CalculatorWorld> {
@override
Future<void> executeStep(num input1, num input2) async {
world.calculator.storeNumericInput(input1);
world.calculator.storeNumericInput(input2);
}
@override
RegExp get pattern => RegExp(r"the numbers {num} and {num}");
}As you can see the class inherits from Given2WithWorld and specifies the types of the two input parameters. The third type CalculatorWorld is a special world context object that allow access context to pass between steps in the same scenario execution instance. If you did not need this you could inherit from Given2 which does not type the world context object but still provides two input parameters.
The input parameters are retrieved via the pattern regex from well know parameter type {num} explained below. They are just special syntax to indicate you are expecting a string, an integer or a number etc at those points in the step text. Therefore, when the step to execute is Given the numbers 1.5 and 2.1 the parameters 1.5 and 2.1 will be passed into the step as the correct types. Note that in the pattern you can use any regex capture group to indicate any input parameter. For example the regex RegExp(r"the (numbers|integers|digits) {num} and {num} indicates 3 parameters and would match to either of the below step text.
Given the numbers 1.5 and 2.1 // passes 3 parameters "numbers", 1.5 & 2.1
Given the integers 1 and 2 // passes 3 parameters "integers", 1 & 2It is worth noting that this library does not rely on mirrors (reflection) for many reasons but most prominently for ease of maintenance and to fall inline with the principles of Flutter not allowing reflection. All in all this make for a much easier to understand and maintain code base as well as much easier debugging for the user. The downside is that we have to be slightly more explicit by providing instances of custom code such as step definition, hook, reporters and custom parameters.
Now that we have a testable app, a feature file and a custom step definition we need to create a class that will call this library and actually run the tests. Create a file called test.dart and put the below code in.
import 'dart:async';
import 'package:gherkin/gherkin.dart';
import 'package:glob/glob.dart';
import 'supporting_files/steps/given_the_numbers.step.dart';
import 'supporting_files/steps/then_expect_numeric_result.step.dart';
import 'supporting_files/steps/when_numbers_are_added.step.dart';
import 'supporting_files/worlds/custom_world.world.dart';
Future<void> main() {
final config = TestConfiguration()
..features = [Glob(r"features/**.feature")]
..reporters = [StdoutReporter(MessageLevel.error), ProgressReporter(), TestRunSummaryReporter()]
..createWorld = (TestConfiguration config) {
return Future.value(CalculatorWorld());
}
..stepDefinitions = [
GivenTheNumbers(),
WhenTheStoredNumbersAreAdded(),
ThenExpectNumericResult()
]
..exitAfterTestRun = true;
return GherkinRunner().execute(config);
}This code simple creates a configuration object and calls this library which will then promptly parse your feature files and run the tests. The configuration file is important and explained in further detail below. However, all that is happening is a Glob is provide which specifies the path to one or more feature files, it sets the reporters to the ProgressReporter report which prints the result of scenarios and steps to the standard output (console). The TestRunSummaryReporter prints a summary of the run once all tests have been executed.
Finally to actually run the tests run the below on the command line from within the example folder:
dart test.dartTo debug tests see Debugging.
Note: You might need to ensure dart is accessible by adding it to your path variable.
The configuration is an important piece of the puzzle in this library as it specifies not only what to run but classes to run against in the form of steps, hooks and reporters. Unlike other implementation this library does not rely on reflection so need to be explicitly told classes to use.
The parameters below can be specified in your configuration file:
Required
An iterable of Glob patterns that specify the location(s) of *.feature files to run. See https://pub.dartlang.org/packages/glob
Defaults to null.
An infix boolean expression which defines the features and scenarios to run based of their tags. See Tags.
Defaults to ExecutionOrder.random
The order by which scenarios will be run. Running an a random order may highlight any inter-test dependencies that should be fixed.
Defaults to en
This specifies the default langauge the feature files are written in. See https://cucumber.io/docs/gherkin/reference/#overview for supported languages.
Defaults to Iterable<StepDefinitionBase>
Place instances of any custom step definition classes Given, Then, When, And, But that match to any custom steps defined in your feature files.
import 'dart:async';
import 'package:gherkin/gherkin.dart';
import 'package:glob/glob.dart';
import 'supporting_files/steps/given_the_numbers.step.dart';
import 'supporting_files/steps/then_expect_numeric_result.step.dart';
import 'supporting_files/steps/when_numbers_are_added.step.dart';
import 'supporting_files/worlds/custom_world.world.dart';
Future<void> main() {
final config = TestConfiguration()
..features = [Glob(r"features/**.feature")]
..reporters = [StdoutReporter(MessageLevel.error), ProgressReporter(), TestRunSummaryReporter()]
..createWorld = (TestConfiguration config) {
return Future.value(CalculatorWorld());
}
..stepDefinitions = [
GivenTheNumbers(),
WhenTheStoredNumbersAreAdded(),
ThenExpectNumericResult()
]
..exitAfterTestRun = true;
return GherkinRunner().execute(config);Defaults to CustomParameter<dynamic>.
Place instances of any custom step parameters that you have defined. These will be matched up to steps when scenarios are run and their result passed to the executable step. See Custom Parameters.
import 'dart:async';
import 'package:gherkin/gherkin.dart';
import 'package:glob/glob.dart';
import 'supporting_files/parameters/power_of_two.parameter.dart';
import 'supporting_files/steps/given_the_numbers.step.dart';
import 'supporting_files/steps/given_the_powers_of_two.step.dart';
import 'supporting_files/steps/then_expect_numeric_result.step.dart';
import 'supporting_files/steps/when_numbers_are_added.step.dart';
import 'supporting_files/worlds/custom_world.world.dart';
Future<void> main() {
final config = TestConfiguration()
..features = [Glob(r"features/**.feature")]
..reporters = [StdoutReporter(MessageLevel.error), ProgressReporter(), TestRunSummaryReporter()]
..customStepParameterDefinitions = [PowerOfTwoParameter()]
..createWorld = (TestConfiguration config) {
return Future.value(CalculatorWorld());
}
..stepDefinitions = [
GivenTheNumbers(),
GivenThePowersOfTwo(),
WhenTheStoredNumbersAreAdded(),
ThenExpectNumericResult()
]
..exitAfterTestRun = true;
return GherkinRunner().execute(config);
}Hooks are custom bits of code that can be run at certain points with the test run such as before or after a scenario. Place instances of any custom Hook class instance in this collection. They will then be run at the defined points with the test run.
Attachment are pieces of data you can attach to a running scenario. This could be simple bits of textual data or even image like a screenshot. These attachments can then be used by reporters to provide more contextual information. For example when a step fails a screenshot could be taken and attached to the scenario which is then used by a reporter to display why the step failed.
Attachments would typically be attached via a Hook for example onAfterStep.
import 'package:gherkin/gherkin.dart';
class AttachScreenhotOnFailedStepHook extends Hook {
/// Run after a step has executed
@override
Future<void> onAfterStep(World world, String step, StepResult stepResult) async {
if (stepResult.result == StepExecutionResult.fail) {
final screenshotData = await _takeScreenshot();
world.attach(screenshotData, 'image/png', step);
}
}
Future<String> _takeScreenshot() async {
// logic to take screenshot
}
}
Reporters are classes that are able to report on the status of the test run. This could be a simple as merely logging scenario result to the console. There are a number of built-in reporter:
StdoutReporter: Logs all messages from the test run to the standard output (console).ProgressReporter: Logs the progress of the test run marking each step with a scenario as either passed, skipped or failed.TestRunSummaryReporter- prints the results and duration of the test run once the run has completed - colours the output.
You should provide at least one reporter in the configuration otherwise it'll be hard to know what is going on.
Note: Feel free to PR new reporters!
import 'dart:async';
import 'package:gherkin/gherkin.dart';
import 'package:glob/glob.dart';
import 'supporting_files/parameters/power_of_two.parameter.dart';
import 'supporting_files/steps/given_the_numbers.step.dart';
import 'supporting_files/steps/given_the_powers_of_two.step.dart';
import 'supporting_files/steps/then_expect_numeric_result.step.dart';
import 'supporting_files/steps/when_numbers_are_added.step.dart';
import 'supporting_files/worlds/custom_world.world.dart';
Future<void> main() {
final config = TestConfiguration()
..features = [Glob(r"features/**.feature")]
..reporters = [StdoutReporter(MessageLevel.error), ProgressReporter(), TestRunSummaryReporter()]
..stepDefinitions = [
GivenTheNumbers(),
WhenTheStoredNumbersAreAdded(),
ThenExpectNumericResult()
]
..exitAfterTestRun = true;
return GherkinRunner().execute(config);
}Defaults to null.
While it is not recommended so share state between steps within the same scenario we all in fact live in the real world and thus at time may need to share certain information such as login credentials etc for future steps to use. The world context object is created once per scenario and then destroyed at the end of each scenario. This configuration property allows you to specify a custom World class to create which can then be accessed in your step classes.
Future<void> main() {
final config = TestConfiguration()
..features = [Glob(r"features/**.feature")]
..reporters = [StdoutReporter(MessageLevel.error), ProgressReporter(), TestRunSummaryReporter()]
..createWorld = (TestConfiguration config) {
return Future.value(CalculatorWorld());
}
..stepDefinitions = [
GivenTheNumbers(),
WhenTheStoredNumbersAreAdded(),
ThenExpectNumericResult()
]
..exitAfterTestRun = true;
return GherkinRunner().execute(config);
}Defaults to true
True to exit the program after all tests have run. You may want to set this to false during debugging.
Step definitions are the coded representation of a textual step in a feature file. Each step starts with either Given, Then, When, And or But. It is worth noting that all steps are actually the same but semantically different. The keyword is not taken into account when matching a step. Therefore the two below steps are actually treated the same and will result in the same step definition being invoked.
Note: Step definitions (in this implementation) are allowed up to 5 input parameters. If you find yourself needing more than this you might want to consider making your step more isolated or using a Table parameter.
Given there are 6 kangaroos
Then there are 6 kangaroosHowever, the domain language you choose will influence what keyword works best in each context. For more information https://docs.cucumber.io/gherkin/reference/#steps.
Given steps are used to describe the initial state of a system. The execution of a Given step will usually put the system into well defined state.
To implement a Given step you can inherit from the Given class.
Given Bob has logged inWould be implemented like so:
import 'package:gherkin/gherkin.dart';
class GivenWellKnownUserIsLoggedIn extends Given1<String> {
@override
Future<void> executeStep(String wellKnownUsername) async {
// implement your code
}
@override
RegExp get pattern => RegExp(r"(Bob|Mary|Emma|Jon) has logged in");
}If you need to have more than one Given in a block it is often best to use the additional keywords And or But.
Given Bob has logged in
And opened the dashboardThen steps are used to describe an expected outcome, or result. They would typically have an assertion in which can pass or fail.
Then I expect 10 applesWould be implemented like so:
import 'package:gherkin/gherkin.dart';
class ThenExpectAppleCount extends Then1<int> {
@override
Future<void> executeStep(int count) async {
// example code
final actualCount = await _getActualCount();
expectMatch(actualCount, count);
}
@override
RegExp get pattern => RegExp(r"I expect {int} apple(s)");
}Caveat: The expect library currently only works within the library's own test function blocks; so using it with a Then step will cause an error. Therefore, the expectMatch or expectA or this.expect methods have been added which mimic the underlying functionality of except in that they assert that the give is true. The Matcher within Dart's test library still work and can be used as expected.
By default a step will timeout if it exceed the defaultTimeout parameter in the configuration file. In some cases you want have a step that is longer or shorter running and in the case you can optionally proved a custom timeout to that step. To do this pass in a Duration object in the step's call to super.
For example, the below sets the step's timeout to 10 seconds.
import 'package:gherkin/gherkin.dart';
class TapButtonNTimesStep extends When2WithWorld<String, int, World> {
TapButtonNTimesStep()
: super(StepDefinitionConfiguration()..timeout = Duration(seconds: 10));
@override
Future<void> executeStep(String input1, int input2) async {
// step logic
}
@override
RegExp get pattern => RegExp(r"I tap the {string} button {int} times");
}Multiline strings can follow a step and will be give to the step it proceeds as the final argument. To denote a multiline string the pre and postfix can either be third double or single quotes """ ... """ or ''' ... '''.
For example:
Given I provide the following "review" comment
"""
Some long review comment.
That can span multiple lines
Skip lines
Maybe even include some numbers
1
2
3
"""The matching step definition would then be:
import 'package:gherkin/gherkin.dart';
class GivenIProvideAComment extends Given2<String, String> {
@override
Future<void> executeStep(String commentType, String comment) async {
// TODO: implement executeStep
}
@override
RegExp get pattern => RegExp(r"I provide the following {string} comment");
}
import 'package:gherkin/gherkin.dart';
/// This step expects a multiline string proceeding it
///
/// For example:
///
/// `Given I add the users`
/// | Firstname | Surname | Age | Gender |
/// | Woody | Johnson | 28 | Male |
/// | Edith | Summers | 23 | Female |
/// | Megan | Hill | 83 | Female |
class GivenIAddTheUsers extends Given1<Table> {
@override
Future<void> executeStep(Table dataTable) async {
for (var row in dataTable.rows) {
// do something with row
row.columns.forEach((columnValue) => print(columnValue));
}
// or get the table as a map (column values keyed by the header)
final columns = dataTable.asMap();
final personOne = columns.elementAt(0);
final personOneName = personOne["Firstname"];
print('Name of first user: `$personOneName`');
}
@override
RegExp get pattern => RegExp(r"I add the users");
}In addition to being able to define a step's own parameters (by using regex capturing groups) there are some well known parameter types you can include that will automatically match and convert the parameter into the correct type before passing it to you step definition. (see https://docs.cucumber.io/cucumber/cucumber-expressions/#parameter-types).
In most scenarios theses parameters will be enough for you to write quite advanced step definitions.
| Parameter Name | Description | Aliases | Type | Example |
|---|---|---|---|---|
| {word} | Matches a single word surrounded by a quotes | {word}, {Word} | String | Given I eat a {word} would match Given I eat a "worm" |
| {string} | Matches one more words surrounded by a quotes | {string}, {String} | String | Given I eat a {string} would match Given I eat a "can of worms" |
| {int} | Matches an integer | {int}, {Int} | int | Given I see {int} worm(s) would match Given I see 6 worms |
| {num} | Matches an number | {num}, {Num}, {float}, {Float} | num | Given I see {num} worm(s) would match Given I see 0.75 worms |
Note that you can combine the well known parameters in any step. For example Given I {word} {int} worm(s) would match Given I "see" 6 worms and also match Given I "eat" 1 worm
As the aim of a feature is to convey human readable tests it is often desirable to optionally have some word pluaralised so you can use the special pluralisation syntax to do simple pluralisation of some words in your step definition. For example:
The step string Given I see {int} worm(s) has the pluralisation syntax on the word "worm" and thus would be matched to both Given I see 1 worm and Given I see 4 worms.
While the well know step parameter will be sufficient in most cases there are time when you would want to defined a custom parameter that might be used across more than or step definition or convert into a custom type.
The below custom parameter defines a regex that matches the words "red", "green" or "blue". The matches word is passed into the function which is then able to convert the string into a Color object. The name of the custom parameter is used to identity the parameter within the step text. In the below example the word "colour" is used. This is combined with the pre / post prefixes (which default to "{" and "}") to match to the custom parameter.
import 'package:gherkin/gherkin.dart';
enum Colour { red, green, blue }
class ColourParameter extends CustomParameter<Colour> {
ColourParameter()
: super("colour", RegExp(r"(red|green|blue)", caseSensitive: true), (c) {
switch (c.toLowerCase()) {
case "red":
return Colour.red;
case "green":
return Colour.green;
case "blue":
return Colour.blue;
}
});
}The step definition would then use this custom parameter like so:
import 'package:gherkin/gherkin.dart';
import 'colour_parameter.dart';
class GivenIPickAColour extends Given1<Colour> {
@override
Future<void> executeStep(Colour input1) async {
print("The picked colour was: '$input1'");
}
@override
RegExp get pattern => RegExp(r"I pick the colour {colour}");
}This customer parameter would be used like this: Given I pick the colour red. When the step is invoked the word "red" would matched and passed to the custom parameter to convert it into a Colour enum which is then finally passed to the step definition code as a Colour object.
Tags are a great way of organizing your features and marking them with filterable information. Tags can be uses to filter the scenarios that are run. For instance you might have a set of smoke tests to run on every check-in as the full test suite is only ran once a day. You could also use an @ignore or @todo tag to ignore certain scenarios that might not be ready to run yet.
You can filter the scenarios by providing a tag expression to your configuration file. Tag expression are simple infix expressions such as:
@smoke
@smoke and @perf
@billing or @onboarding
@smoke and not @ignore
You can even us brackets to ensure the order of precedence
@smoke and not (@ignore or @todo)
You can use the usual boolean statement "and", "or", "not"
Also see https://docs.cucumber.io/cucumber/api/#tags
In order to allow features to be written in a number of languages, you can now write the keywords in languages other than English. To improve readability and flow, some languages may have more than one translation for any given keyword. See https://cucumber.io/docs/gherkin/reference/#overview for a list of supported langauges.
You can set the default language of feature files in your project via the configuration setting see featureDefaultLanguage
For example these two features are the same the keywords are just written in different languages. Note the # language: de on the second feature. English is the default language.
Feature: Calculator
Tests the addition of two numbers
Scenario Outline: Add two numbers
Given the numbers <number_one> and <number_two>
When they are added
Then the expected result is <result>
Examples:
| number_one | number_two | result |
| 12 | 5 | 17 |
| 20 | 5 | 25 |
| 20937 | 1 | 20938 |
| 20.937 | -1.937 | 19 |
# language: de
Funktionalität: Calculator
Tests the addition of two numbers
Szenariogrundriss: Add two numbers
Gegeben sei the numbers <number_one> and <number_two>
Wenn they are added
Dann the expected result is <result>
Beispiele:
| number_one | number_two | result |
| 12 | 5 | 17 |
| 20 | 5 | 25 |
| 20937 | 1 | 20938 |
| 20.937 | -1.937 | 19 |
Please note the language data is take and attributed to the cucumber project https://github.com/cucumber/cucumber/blob/master/gherkin/gherkin-languages.json
A hook is a point in the execution that custom code can be run. Hooks can be run at the below points in the test run.
- Before any tests run
- After all the tests have run
- Before each scenario
- After each scenario
To create a hook is easy. Just inherit from Hook and override the method(s) that signifies the point in the process you want to run code at. Note that not all methods need to be override, just the points at which you want to run custom code.
import 'package:gherkin/gherkin.dart';
class HookExample extends Hook {
/// The priority to assign to this hook.
/// Higher priority gets run first so a priority of 10 is run before a priority of 2
int get priority => 1;
@override
/// Run before any scenario in a test run have executed
Future<void> onBeforeRun(TestConfiguration config) async {
print("before run hook");
}
@override
/// Run after all scenarios in a test run have completed
Future<void> onAfterRun(TestConfiguration config) async {
print("after run hook");
}
@override
/// Run before a scenario and it steps are executed
Future<void> onBeforeScenario(
TestConfiguration config, String scenario) async {
print("running hook before scenario '$scenario'");
}
@override
/// Run after the scenario world is created but run before a scenario and its steps are executed
/// Might not be invoked if there is not a world object
Future<void> onAfterScenarioWorldCreated(World world, String scenario) {
print("running hook after world scenario created'$scenario'");
}
@override
/// Run after a scenario has executed
Future<void> onAfterScenario(
TestConfiguration config, String scenario) async {
print("running hook after scenario '$scenario'");
}
}Finally ensure the hook is added to the hook collection in your configuration file.
import 'dart:async';
import 'package:gherkin/gherkin.dart';
import 'package:glob/glob.dart';
import 'supporting_files/hooks/hook_example.dart';
import 'supporting_files/parameters/power_of_two.parameter.dart';
import 'supporting_files/steps/given_the_numbers.step.dart';
import 'supporting_files/steps/given_the_powers_of_two.step.dart';
import 'supporting_files/steps/then_expect_numeric_result.step.dart';
import 'supporting_files/steps/when_numbers_are_added.step.dart';
import 'supporting_files/worlds/custom_world.world.dart';
Future<void> main() {
final config = TestConfiguration()
..features = [Glob(r"features/**.feature")]
..reporters = [StdoutReporter(MessageLevel.error), ProgressReporter(), TestRunSummaryReporter()]
..hooks = [HookExample()]
..customStepParameterDefinitions = [PowerOfTwoParameter()]
..createWorld = (TestConfiguration config) {
return Future.value(CalculatorWorld());
}
..stepDefinitions = [
GivenTheNumbers(),
GivenThePowersOfTwo(),
WhenTheStoredNumbersAreAdded(),
ThenExpectNumericResult()
]
..exitAfterTestRun = true;
return GherkinRunner().execute(config);
}A reporter is a class that is able to report on the progress of the test run. In it simplest form it could just print messages to the console or be used to tell a build server such as TeamCity of the progress of the test run. The library has a number of built in reporters.
StdoutReporter- prints all messages from the test run to the console.ProgressReporter- prints the result of each scenario and step to the console - colours the output.TestRunSummaryReporter- prints the results and duration of the test run once the run has completed - colours the output.JsonReporter- creates a JSON file with the results of the test run which can then be used by 'https://www.npmjs.com/package/cucumber-html-reporter.' to create a HTML report. You can pass in the file path of the json file to be created.
You can create your own custom reporter by inheriting from the base Reporter class and overriding the one or many of the methods to direct the output message. The Reporter defines the following methods that can be overridden. All methods must return a Future<void> and can be async.
onTestRunStartedonTestRunFinishedonFeatureStartedonFeatureFinishedonScenarioStartedonScenarioFinishedonStepStartedonStepFinishedonExceptionmessagedispose
Once you have created your custom reporter don't forget to add it to the reporters configuration file property.
Note: PR's of new reporters are always welcome.