From 109793ceb51caa10c24970e3ac1992cff281cccb Mon Sep 17 00:00:00 2001 From: Madalin Ilie Date: Sat, 9 Nov 2024 22:23:49 +0200 Subject: [PATCH] Update documentation with latest fuzzers and arguments --- docs/commands-and-arguments/arguments.md | 8 ++++++-- docs/getting-started/filtering-reports.md | 5 +++++ docs/troubleshooting/index.md | 10 ++++++---- 3 files changed, 17 insertions(+), 6 deletions(-) diff --git a/docs/commands-and-arguments/arguments.md b/docs/commands-and-arguments/arguments.md index d9bc0cfb..c2f485f7 100644 --- a/docs/commands-and-arguments/arguments.md +++ b/docs/commands-and-arguments/arguments.md @@ -58,6 +58,11 @@ You can get the full list of arguments by running `cats -h`. Below is a short de - `--ignoreResponseWords` COUNT_LIST a comma separated list of words count in the response that will be considered as SUCCESS, even if the Fuzzer will typically report it as WARN or ERROR - `--ignoreResponseLines` LINES_COUNT a comma separated list of lines count in the response that will be considered as SUCCESS, even if the Fuzzer will typically report it as WARN or ERROR - `--ignoreResponseRegex` a REGEX that will match against the response that will be considered as SUCCESS, even if the Fuzzer will typically report it as WARN or ERROR +- `--filterResponseCodes` HTTP_CODES_LIST a comma separated list of HTTP response codes that will be filtered and not included in the final report. You can use response code families as `2xx`, `4xx`, etc. +- `--filterResponseSize` SIZE_LIST a comma separated list of response sizes that will be filtered and not included in the final report +- `--filterResponseWords` COUNT_LIST a comma separated list of words count in the response that will be filtered and not included in the final report +- `--filterResponseLines` LINES_COUNT a comma separated list of lines count in the response that will be filtered and not included in the final report +- `--filterResponseRegex` a REGEX that will match against the response that will be filtered and not included in the final report - `--tests` TESTS_LIST a comma separated list of executed tests in JSON format from the cats-report folder. If you supply the list without the .json extension CATS will search the test in the cats-report folder - `--ignoreResponseCodeUndocumentedCheck` If supplied (not value needed) it won't check if the response code received from the service matches the value expected by the fuzzer and will return the test result as SUCCESS instead of WARN - `--ignoreResponseBodyCheck` If supplied (no value needed) it won't check if the response body received from the service matches the schema supplied inside the contract and will return the test result as SUCCESS instead of WARN @@ -91,10 +96,9 @@ You can get the full list of arguments by running `cats -h`. Below is a short de - `--fuzzersConfig=FILE` A properties file with Fuzzer configuration that changes default behaviour. Configuration keys are prefixed with the fully qualified Fuzzer name - `--mutators=FOLDER` A folder containing custom mutators. Only applicable when using the `cats random` sub-command - `--allowInvalidEnumValues` When set to true the `InvalidValuesInEnumsFieldsFuzzer` will expect a 2XX response code instead of 4XX -- `--[no-]filterXxxFromRequestPayloads` In extremely rare cases when CATS fails to generate anyOf/oneOf combinations some requests may still contain ONE_OF/ANY_OF markers. They are filtered out by default. Setting this to false will send them as requests which will probably fail. It's mostly for debug purposes -- `--generateXxxCombinationsForResponses` Generate anyOf/oneOf combinations also for response schemas. By default it creates one response payload with all possibilities - `--selfReferenceDepth=` Max depth for objects having cyclic dependencies - `--limitXxxOfCombinations=` Max number of anyOf/oneOf combinations +- `--limitFuzzedFields=` Max number of fields that will be fuzzed - `--[no-]useDefaults` If set to true, it will use the default values when generating examples - `--nameReplace` If set to true, it will simply do a replacement between the targetFields names provided and the fuzz values - `--stopAfterErrors=` Number of errors after which the continuous fuzzing will stop running. Errors are defined as conditions matching the given match arguments. Only available in `cats random` sub-command. diff --git a/docs/getting-started/filtering-reports.md b/docs/getting-started/filtering-reports.md index d6aeb4ac..2c52cb36 100644 --- a/docs/getting-started/filtering-reports.md +++ b/docs/getting-started/filtering-reports.md @@ -7,6 +7,7 @@ description: How to ignore specific HTTP response CATS reports can be huge. Opening a report with 20k tests it's not the best user experience. Outside the typical [Slicing Strategies](slicing-strategies), you can also choose to ignore reporting for specific HTTP responses. + ## Ignoring Specific HTTP Responses By default, CATS will report `warns` and `errors` according to the specific behaviour of each Fuzzer. There are cases though when you might want to focus only on critical bugs. You can use the `--ignoreResponseXXX` arguments to supply a list of response codes, response sizes, word counts, @@ -36,3 +37,7 @@ You can ignore specific response codes using the `--ignoreResponseCodes` argumen ## Ignore Response Lines, Words, Size, Regex You can ignore specific response lines, words, size or regex using the `--ignoreResponseLines`, `--ignoreResponseWords`, `--ignoreResponseSize`, `--ignoreResponseRegex` arguments. This will make CATS report these cases as `success` instead of `warn`. + +:::info All --ignoreXXX arguments have a corresponding --filterXXX argument +All `--ignoreXXX` arguments above have a corresponding `--filterXXX` argument which is equivalent to the `--ignoreXXX` argument AND will also consider `--skipReportingForIgnore` as being true. +::: diff --git a/docs/troubleshooting/index.md b/docs/troubleshooting/index.md index 9f74f48b..df0f198d 100644 --- a/docs/troubleshooting/index.md +++ b/docs/troubleshooting/index.md @@ -6,7 +6,7 @@ These are some of the common reasons why CATS might appear to be stuck: - **Network issues**: CATS might be waiting for a response from the server. Check the network connection and the server status. - **Server issues**: The server might be slow or unresponsive. Check the server logs and the server status. -- **Many fields inside the request(s)**: If the request has many fields, CATS might take longer to generate the tests, as many fuzzers are running for each field. +- **Many fields inside the request(s)**: If the request has many fields, CATS might take longer to generate the tests, as many fuzzers are running for each field. You can limit the number of fields to be fuzzed using `--limitFuzzedFields` argument. - **Usage of anyOf/oneOf in request schemas**: If the request schema contains `anyOf` or `oneOf`, CATS will generate tests for all possible combinations. This can lead to a large number of tests being generated. You can limit the number of combinations by using the `--limitXxxCombinations` argument. For example, `--limitOneOfCombinations=10` will limit the number of combinations for all `oneOf/anyOf` to 10. - **Self-reference in the request schema**: If the request schema contains self-references, when CATS generates the request sample it might enter an infinite loop. You can limit the depth of the schema by using the `--selfReferenceDepth` argument. For example, `--selfReferenceDepth=3` will limit the self-reference depth of the schema to 3. @@ -23,7 +23,7 @@ You can set examples for each property or provide specific formats in the schema CATS generates tests based on the request schema and the fuzzers that are enabled. If the request schema is complex or has many fields, CATS might generate a large number of tests. You can check the following pages for different strategies: [Slicing Strategies](https://endava.github.io/cats/docs/getting-started/slicing-strategies) and [Filtering Options](https://endava.github.io/cats/docs/getting-started/filtering-reports). -## CATS stops saying it cannot generate values for a field +## CATS stops, saying it cannot generate values for a field CATS generates values based on the request schema using examples or request field names and/or formats. If the request schema is not well-defined, CATS might not be able to generate values for some fields. There might be cases when the `pattern` for a specific property is too complicated and all internal generators are not able to create a value that it's matching that pattern. @@ -35,7 +35,7 @@ This is usually when CATS generates a large number of tests and the report is to ## CATS generates same number of tests in 2 consecutive runs with same configuration -Unlike typical fuzzers that generate random values, CATS generates tests based on the request schema and the fuzzers that are enabled. If the request schema and the configuration are the same, CATS will generate the same tests. +Unlike typical fuzzers that rely only on randomness, CATS generates tests based on the request schema and the fuzzers that are enabled. If the request schema and the configuration are the same, CATS will generate the same tests. CATS uses a deterministic approach to generate tests based on a set of fuzzers which run in a specific order and specific logic. Randomness is introduced only in the values generated for the fields. If you want randomness also in the tests generated, you must use the `cats random` sub-command. Check [Continuous Fuzzing](https://endava.github.io/cats/docs/getting-started/running-cats#continuous-fuzzing-mode) for more details. @@ -49,7 +49,7 @@ Assuming the OpenAPI file contains at least one valid path with at least one ope ## CATS generates tests that are not valid If you think some of the test cases generated are not valid in your scenario you can: -- **Filter out the invalid test cases**: You can use the `--ignoreXXX` arguments to filter out the test cases that are not valid +- **Filter out the invalid test cases**: You can use the `--ignoreXXX` or `--filterXXX` arguments to filter out the test cases that are not valid - **Filter out specific fuzzers**: You can use the `--skipFuzzer` argument to filter out specific fuzzers that are generating invalid test cases - **Filter out specific paths**: You can use the `--skpiPath` argument to filter out specific paths that are generating invalid test cases - **Filter out specific http methods**: You can use the `--skipHttpMethod` argument to filter out specific http methods that are generating invalid test cases @@ -58,6 +58,8 @@ If you think some of the test cases generated are not valid in your scenario you - **Filter out specific OpenAPI field types**: You can use the `--skipFieldType` argument to filter out specific OpenAPI field types that are generating invalid test cases - **Filter out specific request headers**: You can use the `--skipHeader` argument to filter out specific request headers that are generating invalid test cases +If you think there might be a bug in CATS, please open an issue on the [GitHub repository](https://github.com/Endava/cats/issues/new/choose). + ## CATS generates requests that are not valid In very rare cases CATS might generate requests that are not valid.