Minitest::Reporters::JsonReporter

This is an extension gem for the minitest-reporters gem. It adds JSON output as an output format.

Abstract

You can use this gem to interface Minitest output into automated tools like CI, CD or IDEs or code editors. An example interface might be to the Atom editor: https://atom.io

Use of JSON as a format for test runs opens up possibilities for different types of analysis. You can organize the output based on elapsed time or number of assertions of tests, for example. See the 'jq sort' example below.

I originally wrote this gem to interface to the Viper audible code editor for the blind community. See: https://github.com/edhowland/viper Using this gem with Viper also requires the 'viper_ruby' package. See https://github.com/edhowland/viper_ruby

Version 1.0.0

Installation

Add this line to your application's Gemfile:

gem 'minitest-reporters-json_reporter'

And then execute:

$ bundle

Or install it yourself as:

$ gem install minitest-reporters-json_reporter

Usage

Add this line to your 'spec_helper.rb' or 'test_helper.rb':

require 'minitest/reporters/json_reporter'

In that same file, add something like this near the bottom of the file:

Minitest::Reporters.use! [ Minitest::Reporters::JsonReporter.new ]

Then run your tests as normal. You will get a JSON formatted string in stdout.

Object Summary

• status of the overall test run. Object.
  • code - Success, Passed with skipped tests or Failed. String.
  • color green, yellow or red. String.
• metadata The metadata of this particular test run. Object.
  • generated_by Class name of the Minitest Reporter. Will usually be Minitest::Reporters::JsonReporter. String.
  • version - Version of the minitest-reporters-json_reporter gem. String.
  • ruby_version - RUBY_VERSION. String.
  • ruby_patchlevel - RUBY_PATCHLEVEL. Integer.
  • ruby_platform - RUBY_PLATFORM. String.
  • time - Time of test run in UTC and ISO-8601 format. String.
  • options - Options object as constructed by passed options, superclasses and command line args. Object.
    • io - The class name of the IO object. Will be the string 'STDOUT' if equal to $stdout. String.
    • keys of the options hash, if any. Various key/value pairs.
    • args Array of arguments passed to the program. Array of String.
    • total_count - Total number of tests ran. Integer.
• statisticsThe accumulated counts for this test run. Object.
  • total Number of total runs. Integer.
  • assertions Count of assertions performed. Integer.
  • failures Number of failed tests. Integer.
  • errors Number of errors encountered. Integer.
  • skips Number of skipped tests. Integer.
  • passes Number of passed tests. Integer.
• timings Object containing the computed timing information for the entire test run. Object.
  • total_seconds total number of seconds for the entire test run. Float.
  • runs_per_second Averaged number of runs per second. Float.
  • assertions_per_second Averaged number of assertions per second. Float.
• fails - Array of failed or errored tests. Array of Object.
  • type Type of the failure. Either 'failure' or 'error'. String.
  • classClass name of the test. String.
  • name Name of the test. String.
  • assertions Number of the assertions for this test. Integer.
  • time Time in seconds for this test. Float.
  • message Message reported by this failure or error. String.
  • location File name and line number. file:line. String.
  • backtrace (exists only if type is 'error') Array of backtrace paths and line numbers. Array of String.
• skips Array of skipped tests. Array of Object.
  • type The string 'skipped'. String.
  • class Class name of the test. String.
  • name The name of the test. String.
  • assertions The count of the assertions for this test. Integer.
  • time The time of this test in seconds. Float.
  • message The message string passed to the 'skip()' method. String.
  • location File name and line number. file:line. String.
• passes Array of passed test runs. Array of Object.
  • type The string 'passed'. String.
  • class Class name of the test. String.
  • name The name of the test. String.
  • assertions The count of the assertions for this test. Integer.
  • time The time of this test in seconds. Float.

The last 2 objects: skips[] and passes[] are absent unless the --verbose command line flag was passed or if the options[:verbose] value is true.

Sample output

# Use jq to pretty print the JSON output
$  cd test/functional/
$ ruby report_spec.rb | jq .
{
  "status": {    "code": "Failed",    "color": "red"  },
  "metadata": {
    "generated_by": "Minitest::Reporters::JsonReporter",
    "version": "1.0.0",
    "ruby_version": "2.2.2",
    "ruby_patchlevel": 95,
    "ruby_platform": "x86_64-linux",
    "time": "2016-05-03T20:57:42Z",
    "options": {
      "io": "STDOUT",
      "verbose": true,
      "seed": 3023,
      "args": "--verbose --seed 3023",
      "total_count": 5
    }
  },
  "statistics": {    "total": 5,    "assertions": 4,    "failures": 2,
    "errors": 1,    "skips": 1,    "passes": 1  },
  "timings": {
    "total_seconds": 0.0014724910142831504,
    "runs_per_second": 3395.606459733908,
    "assertions_per_second": 2716.4851677871266
  },
  "fails": [
    {
      "type": "failed",      "class": "second failure",      "name": "test_0001_anonymous",
      "assertions": 1,      "time": 9.327399311587214e-05,
      "message": "Expected: 9\n  Actual: 3",
      "location": "report_spec.rb:20"
    },
    {
      "type": "failed",
      "class": "failure",      "name": "test_0001_anonymous",      "assertions": 1,
      "time": 3.400500281713903e-05,
      "message": "Expected: 2\n  Actual: 1",
      "location": "report_spec.rb:14"
    },
    {
      "type": "error",      "class": "Error",
      "name": "test_0001_anonymous",      "assertions": 0,
      "time": 2.1556013962253928e-05,
      "message": "RuntimeError: should fail\n    report_spec.rb:6:in `block (2 levels) in <main>'",
      "location": "report_spec.rb:6",
      "backtrace": [
        "report_spec.rb:6:in `block (2 levels) in <main>'"
      ]
    }
  ],
  "skips": [
    {
      "type": "skipped",      "class": "skipped test",      "name": "test_0001_anonymous",
      "assertions": 0,
      "time": 3.7049001548439264e-05,
      "message": "what a layabout",
      "location": "report_spec.rb:34"
    }
  ],
  "passes": [
    {
      "type": "passed",      "class": "working assertion",
      "name": "test_0001_should have 2 working assertions",      "assertions": 2,
      "time": 2.5684013962745667e-05
    }
  ]
}

Example analysis feedback

JSON can be parsed and manipulated to provide many types of useful information. Below are some example usages. We use the 'jq' program to parse and select and arrange the output. The version of 'jq' is 1.5. It can be downloaded/install instructions here: Download JQ See: JQ Developer Manual

Sort by time, slowest first

This sort would show you the slowest tests first, getting faster further down the array. Similar to Minitest::Reporters::MeanTimeReporter which produces a report summary showing the slowest running tests.

$ ruby timings_spec.rb  --verbose | jq '.passes | sort_by(.time) | reverse[] | .name, .time'
"test_0001_should be slow"
5.001584862009622
"test_0003_should be slightly faster"
1.0003409570199437
"test_0002_should be fast"
2.9060000088065863e-05

Group By Class example

Minitest usually runs your tests in a random sequence. This is great for test isolation and to check for state bleed-thru, but can be annoying if trying to determine where similar tests are failing. You can use the jq 'sort_by' or 'group_by' filters to get them back in some semblace of order.

Here we group the .fails[] array by their class name. (The file: 'group_by_spec.rb' contains 4 tests inside 2 classes.)

$ ruby group_by_spec.rb |jq '.fails | group_by(.class) | flatten[] | .class, .name'
"TestNumericalGroup"
"test_4_times_6_equals_24"
"TestNumericalGroup"
"test_positive_integers_are_greater_than_0"
"TestStringGroup"
"test_string_is_hello_world"
"TestStringGroup"
"test_value_length_equals_2"

Customizing the JSON format

You can adjust the contents of the returned JSON by sub-classing the Minitest::Reporters::JsonReporter class. Override the 'to_h' method and delete or modifythe hash it returns.

Here is a simple example that eliminates the 'metadata' and 'timmings' components:

# example spec_helper.rb:

class Minitest::Reporters::SlimJsonReporter < Minitest::Reporters::JsonReporter
    def to_h
    h = super
    h.delete(:metadata)
    h.delete(:timings)
    h
  end
end

Minitest::Reporters.use!( Minitest::Reporters::SlimJsonReporter.new )

Contributing

1. Fork it ( https://github.com/edhowland/minitest-reporters-json_reporter/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

Note: Extra credit if git flow feature branching was used.

Resources

• Code: https://github.com/edhowland/minitest-reporters-json_reporter
• Bugs: https://github.com/edhowland/minitest-reporters-json_reporter/issues
• Docs: http://www.rubydoc.info/gems/minitest-reporters-json_reporter/1.0.0
• Gem: https://rubygems.org/gems/minitest-reporters-json_reporter

Feedback

If you find this gem helpful, please slip me a note via e-mail: ed.howland@gmail.com

I would like to know if anyone has used this approach to interface Minitest to any automated framework or editor.

Thanks, Ed.