Skip to content

Latest commit

 

History

History
280 lines (192 loc) · 6.98 KB

benchmarks.md

File metadata and controls

280 lines (192 loc) · 6.98 KB
Raw

Benchmarks

JSCK has fairly comprehensive benchmarks which show it to be one of the fastest JSON Schema validators available for Node.js.

To run the benchmarks immediately after you git clone the repo:

git clone git@github.com:pandastrike/jsck.git && \
git submodule update --init && \
npm install && \
coffee benchmarks/

Benchmarking harness and content lives in benchmarks, with separate directories for each draft. For each draft there are a number of different benchmarks (contained in subdirectories) that can be run in whole or in part. This is the basic directory structure:

benchmarks/
  draft3/
  draft4/
    complex/
    medium/
    trivial/
    index.coffee
    validators.coffee

Adding Additional Validators

The validators are defined in validators.coffee. Here's an example from the Draft 4 benchmarks:

    setup: (schema) ->
      ajv = require("ajv")()
      ajv.compile(schema)
    validate: ({validator, schema, document}) ->
      validator(document)
    error: (result) ->
      if result == true
        false
      else
        validator.errors

The setup function returns a validator object usable by the validate function. The validate function performs the actual validation work that is measured in benchmarking. The error function is used only in the preflight stage of benchmarking to determine whether the result of the validate function contains any errors. This is useful in making sure that all the validators are actually operating correctly and thus being compared fairly.

Adding New Benchmarks

Each individual benchmark consists of a directory containing a schema, a valid document, and an index.coffee file which runs the benchmark.

validators = require "../validators"
{benchmark} = require("../../runner")(validators)

benchmark {
  name: "Event - Valid document"
  repeats: 1024
  schema: require "./schema"
  document: require "./valid_doc"
}

The "repeats" parameter determines how many times the document will be validated during a single measurement sample.

Running Benchmarks

You can run very specific benchmarks, like the medium-complexity benchmarks for draft 3 only, like so:

coffee benchmarks/draft3/medium

You can run all benchmarks for a specific JSON Schema draft:

coffee benchmarks/draft4

Or, to run all benchmarks:

coffee benchmarks/

You should then see something like this:

## Benchmarks for Draft 3

Schema: 'Event'.  A simple schema, exercising very few attributes
Sample size: 64
Validations per sample: 512

  JSCK
  Warming up: ................................
  Iterations: ................................................................

  amanda
  Warming up: ................................
  Iterations: ................................................................

  JSV
  Warming up: ................................
  Iterations: ................................................................

  json-gate
  Warming up: ................................
  Iterations: ................................................................


  JSCK: validations/millisecond
  median: 193.025    max: 202.692    min: 175.945 

  amanda: validations/millisecond
  median: 4.258    max: 4.608    min: 3.778 

  JSV: validations/millisecond
  median: 2.685    max: 2.748    min: 2.43 

  json-gate: validations/millisecond
  median: 82.514    max: 86.036    min: 48.234 

Relative speeds:
JSCK : 1.000
json-gate : 2.339
amanda : 45.337
JSV : 71.896


Schema: 'Configuration'.  A moderately complex schema with some nesting and value constraints
Sample size: 64
Validations per sample: 128

  JSCK
  Warming up: ................................
  Iterations: ................................................................

  amanda
  Warming up: ................................
  Iterations: ................................................................

  JSV
  Warming up: ................................
  Iterations: ................................................................

  json-gate
  Warming up: ................................
  Iterations: ................................................................


  JSCK: validations/millisecond
  median: 108.936    max: 109.966    min: 85.964 

  amanda: validations/millisecond
  median: 3.788    max: 5.651    min: 2.371 

  JSV: validations/millisecond
  median: 1.344    max: 1.359    min: 1.255 

  json-gate: validations/millisecond
  median: 44.176    max: 45.845    min: 41.803 

Relative speeds:
JSCK : 1.000
json-gate : 2.466
amanda : 28.760
JSV : 81.025

## Benchmarks for Draft 4

Schema: 'Event - Valid document'.  A simple schema, exercising very few attributes
Sample size: 64
Validations per sample: 1024

  JSCK
  Warming up: ................................
  Iterations: ................................................................

  tv4
  Warming up: ................................
  Iterations: ................................................................

  z-schema
  Warming up: ................................
  Iterations: ................................................................


  JSCK: validations/millisecond
  median: 186.81    max: 191.617    min: 138.716 

  tv4: validations/millisecond
  median: 54.377    max: 55.907    min: 46.942 

  z-schema: validations/millisecond
  median: 93.657    max: 93.979    min: 89.44 

Relative speeds:
JSCK : 1.000
z-schema : 1.995
tv4 : 3.435


Schema: 'Configuration'.  A moderately complex schema with some nesting and value constraints
Sample size: 64
Validations per sample: 256

  JSCK
  Warming up: ................................
  Iterations: ................................................................

  tv4
  Warming up: ................................
  Iterations: ................................................................

  z-schema
  Warming up: ................................
  Iterations: ................................................................


  JSCK: validations/millisecond
  median: 118.573    max: 119.07    min: 105.22 

  tv4: validations/millisecond
  median: 19.608    max: 20.389    min: 16.505 

  z-schema: validations/millisecond
  median: 40.606    max: 40.934    min: 34.934 

Relative speeds:
JSCK : 1.000
z-schema : 2.920
tv4 : 6.047


Schema: 'Transaction'.  
Sample size: 64
Validations per sample: 64

  JSCK
  Warming up: ................................
  Iterations: ................................................................

  tv4
  Warming up: ................................
  Iterations: ................................................................

  z-schema
  Warming up: ................................
  Iterations: ................................................................


  JSCK: validations/millisecond
  median: 12.814    max: 14.873    min: 6.984 

  tv4: validations/millisecond
  median: 1.946    max: 1.965    min: 1.832 

  z-schema: validations/millisecond
  median: 3.768    max: 3.836    min: 3.297 

Relative speeds:
JSCK : 1.000
z-schema : 3.401
tv4 : 6.583