Wp Header Logo 45

CATS – REST API Fuzzer And Negative Testing Tool For OpenAPI Endpoints



REST API fuzzer and negative testing tool. Run thousands of self-healing API tests within minutes with no coding effort!

By using a simple and minimal syntax, with a flat learning curve, CATS (Contract Auto-generated Tests for Swagger) enables you to generate thousands of API tests within minutes with no coding effort. All tests are generated, run and reported automatically based on a pre-defined set of 89 Fuzzers. The Fuzzers cover a wide range of input data from fully random large Unicode values to well crafted, context dependant values based on the request data types and constraints. Even more, you can leverage the fact that CATS generates request payloads dynamically and write simple end-to-end functional tests.

This is a list of articles with step-by-step guides on how to use CATS:

CATS is bundled both as an executable JAR or a native binary. The native binaries do not need Java installed.

To get persistent autocomplete, add the above line in ~/.zshrc or ./bashrc, but make sure you put the fully qualified path for the cats_autocomplete script.

You can also check the cats_autocomplete source for alternative setup.

There is no native binary for Windows, but you can use the uberjar version. This requires Java 11+ to be installed.

You can run it as java -jar cats.jar.

You can build CATS from sources on you local box. You need Java 11+. Maven is already bundled.

Before running the first build, please make sure you do a ./mvnw clean. CATS uses a fork ok OKHttpClient which will install locally under the 4.9.1-CATS version, so don’t worry about overriding the official versions.

You can use the following Maven command to build the project:

./mvnw package -Dquarkus.package.type=uber-jar

cp target/

You will end up with a cats.jar in the target folder. You can run it wih java -jar cats.jar ....

You can also build native images using a GraalVM Java version.

./mvnw package -Pnative

You may see some ERROR log messages while running the Unit Tests. Those are expected behaviour for testing the negative scenarios of the Fuzzers.

In blackbox mode CATS will only report ERRORs if the received HTTP response code is a 5XX. Any other mismatch between what the Fuzzer expects vs what the service returns (for example service returns 400 and service returns 200) will be ignored.

The blackbox mode is similar to a smoke test. It will quickly tell you if the application has major bugs that must be addressed immediately.

The real power of CATS relies on running it in a non-blackbox mode also called context mode. Each Fuzzer has an expected HTTP response code based on the scenario under test and will also check if the response is matching the schema defined in the OpenAPI spec specific to that response code. This will allow you to tweak either your OpenAPI spec or service behaviour in order to create good quality APIs and documentation and also to avoid possible serious bugs.

You may notice a significant number of tests marked as skipped. CATS will try to apply all Fuzzers to all fields, but this is not always possible. For example the BooleanFieldsFuzzer cannot be applied to String fields. This is why that test attempt will be marked as skipped. It was an intentional decision to also report the skipped tests in order to show that CATS actually tries all the Fuzzers on all the fields/paths/endpoints.

CATS generates tests based on configured Fuzzers. Each Fuzzer has a specific scenario and a specific expected result. The CATS engine will run the scenario, get the result from the service and match it with the Fuzzer expected result. Depending on the matching outcome, CATS will report as follows:

CATS will iterate through all endpoints, all HTTP methods and all the associated requests bodies and parameters (including multiple combinations when dealing with oneOf/anyOf elements) and fuzz their values considering their defined data type and constraints. The actual fuzzing depends on the specific Fuzzer executed. Please see the list of fuzzers and their behaviour. There are also differences on how the fuzzing works depending on the HTTP method:

HTML_JS is the default report produced by CATS. The execution report in placed a folder called cats-report/TIMESTAMP or cats-report depending on the --timestampReports argument. The folder will be created inside the current folder (if it doesn’t exist) and for each run a new subfolder will be created with the TIMESTAMP value when the run started. This allows you to have a history of the runs. The report itself is in the index.html file, where you can:

Along with the summary from index.html each individual test will have a specific TestXXX.html page with more details, as well as a json version of the test which can be latter replayed using > cats replay TestXXX.json.

Understanding the Result Reason values:

And this is what you get when you click on a specific test: 

This format is similar with HTML_JS, but you cannot do any filtering or sorting.

The JUNIT report is written as junit.xml in the cats-report folder. Individual tests, both as .html and .json will also be created.

CATS has a significant number of Fuzzers. Currently, 89 and growing. Some of the Fuzzers are executing multiple tests for every given field within the request. For example the ControlCharsOnlyInFieldsFuzzer has 63 control chars values that will be tried for each request field. If a request has 15 fields for example, this will result in 1020 tests. Considering that there are additional Fuzzers with the same magnitude of tests being generated, you can easily get to 20k tests being executed on a typical run. This will result in huge reports and long run times (i.e. minutes, rather than seconds).

Below are some recommended strategies on how you can separate the tests in chunks which can be executed as stages in a deployment pipeline, one after the other.

You can use the --paths=PATH argument to run CATS sequentially for each path.

You can use the --checkXXX arguments to run CATS only with specific Fuzzers like: --checkHttp, -checkFields, etc.

You can use various arguments like --fuzzers=Fuzzer1,Fuzzer2 or -skipFuzzers=Fuzzer1,Fuzzer2 to either include or exclude specific Fuzzers. For example, you can run all Fuzzers except for the ControlChars and Whitespaces ones like this: --skipFuzzers=ControlChars,Whitesspaces. This will skip all Fuzzers containing these strings in their name. After, you can create an additional run only with these Fuzzers: --fuzzers=ControlChars,Whitespaces.

These are just some recommendations on how you can split the types of tests cases. Depending on how complex your API is, you might go with a combination of the above or with even more granular splits.

Please note that due to the fact that ControlChars, Emojis and Whitespaces generate huge number of tests even for small OpenAPI contracts, they are disabled by default. You can enable them using the --includeControlChars, --includeWhitespaces and/or --includeEmojis arguments. The recommendation is to run them in separate runs so that you get manageable reports and optimal running times.

By default, CATS will report WARNs and ERRORs according to the specific behaviour of each Fuzzer. There are cases though when you might want to focus only on critical bugs. You can use the --ignoreResponseXXX arguments to supply a list of response codes, response sizes, word counts, line counts or response body regexes that should be ignored as issues (overriding the Fuzzer behaviour) and report those cases as success instead or WARN or ERROR. For example, if you want CATS to report ERRORs only when there is an Exception or the service returns a 500, you can use this: --ignoreResultCodes="2xx,4xx".

You can also choose to ignore checks done by the Fuzzers. By default, each Fuzzer has an expected response code, based on the scenario under test and will report and WARN the service returns the expected response code, but the response code is not documented inside the contract. You can make CATS ignore the undocumented response code checks (i.e. checking expected response code inside the contract) using the --ignoreResponseCodeUndocumentedCheck argument. CATS with now report these cases as SUCCESS instead of WARN.

Additionally, you can also choose to ignore the response body checks. By default, on top of checking the expected response code, each Fuzzer will check if the response body matches what is defined in the contract and will report an WARN if not matching. You can make CATS ignore the response body checks using the --ingoreResponseBodyCheck argument. CATS with now report these cases as SUCCESS instead of WARN.

When CATS runs, for each test, it will export both an HTML file that will be linked in the final report and individual JSON files. The JSON files can be used to replay that test. When replaying a test (or a list of tests), CATS won’t produce any report. The output will be solely available in the console. This is useful when you want to see the exact behaviour of the specific test or attach it in a bug report for example.

The syntax for replaying tests is the following:

Some notes on the above example:

To list all available commands, run:

All available subcommands are listed below:

> cats help or cats -h will list all available options

> cats list --fuzzers will list all the existing fuzzers, grouped on categories

> cats list --fieldsFuzzingStrategy will list all the available fields fuzzing strategies

> cats list --paths --contract=CONTRACT will list all the paths available within the contract

> cats replay "test1,test2" will replay the given tests test1 and test2

> cats fuzz will fuzz based on a given request template, rather than an OpenAPI contract

> cats run will run functional and targeted security tests written in the CATS YAML format

> cats lint will run OpenAPI contract linters, also called ContractInfoFuzzers

This will run CATS against http://localhost:8080 using my.yml as an API spec and will only run the HTTP headers Fuzzers.

To get a list of fuzzers run cats list --fuzzers. A list of all available fuzzers will be returned, along with a short description for each.

There are multiple categories of Fuzzers available:

Additional checks which are not actually using any fuzzing, but leverage the CATS internal model of running the tests as Fuzzers:

CATS has currently 42 registered Field Fuzzers:

You can run only these Fuzzers by supplying the --checkFields argument.

CATS has currently 28 registered Header Fuzzers:

You can run only these Fuzzers by supplying the --checkHeaders argument.

CATS has currently 6 registered HTTP Fuzzers:

You can run only these Fuzzers by supplying the --checkHttp argument.

Usually a good OpenAPI contract must follow several good practices in order to make it easy digestible by the service clients and act as much as possible as self-sufficient documentation:

CATS has currently 9 registered ContractInfo Fuzzers:

You can run only these Fuzzers using > cats lint --contract=CONTRACT.

It’s important to note that reference data won’t get replaced when using the FunctionalFuzzer. So if there are reference data fields, you must also supply those in the FunctionalFuzzer.

The FunctionalFuzzer will only trigger if a valid functionalFuzzer.yml file is supplied. The file has the following syntax:

And a typical run will look like:

This is a description of the elements within the functionalFuzzer.yml file:

When you have request payloads which can take multiple object types, you can use the oneOfSelection keyword to specify which of the possible object types is required by the FunctionalFuzzer. If you don’t provide this element, all combinations will be considered. If you supply a value, this must be exactly the one used in the discriminator.

As CATs mostly relies on generated data with small help from some reference data, testing complex business scenarios with the pre-defined Fuzzers is not possible. Suppose we have an endpoint that creates data (doing a POST), and we want to check its existence (via GET). We need a way to get some identifier from the POST call and send it to the GET call. This is now possible using the FunctionalFuzzer. The functionalFuzzerFile can have an output entry where you can state a variable name, and its fully qualified name from the response in order to set its value. You can then refer the variable using ${variable_name} from another test in order to use its value.

Here is an example:

Suppose the test_1 execution outputs:

When executing test_1 the value of the pet id will be stored in the petId variable (value 2). When executing test_2 the id parameter will be replaced with the petId variable (value 2) from the previous case.

Please note: variables are visible across all custom tests; please be careful with the naming as they will get overridden.

The FunctionalFuzzer can verify more than just the expectedResponseCode. This is achieved using the verify element. This is an extended version of the above functionalFuzzer.yml file.

Considering the above file:

The following json response will pass test_1:

But this one won’t (pet#name is missing):

You can also refer to request fields in the verify section by using the ${request#..} qualifier. Using the above example, by having the following verify section:

It will verify if the response contains a pet#name element and that its value equals My Pet as sent in the request.

Some notes:

The following keywords are reserved in FunctionalFuzzer tests: output, expectedResponseCode, httpMethod, description, oneOfSelection, verify, additionalProperties, topElement and mapValues.

And a typical run:

You can also supply output, httpMethod, oneOfSelection and/or verify (with the same behaviour as within the FunctionalFuzzer) if they are relevant to your case.

This is what the SecurityFuzzer will do after parsing the above securityFuzzerFile:

If you want the above logic to apply to all paths, you can use all as the path name:

Instead of specifying the field names, you can broader to scope to target certain fields types. For example, if we want to test for XSS in all string fields, you can have the following securityFuzzerFile:

The following keywords are reserved in SecurityFuzzer tests: output, expectedResponseCode, httpMethod, description, verify, oneOfSelection, targetFields, targetFieldTypes, stringsFile, additionalProperties, topElement and mapValues.

The TemplateFuzzer can be used to fuzz non-OpenAPI endpoints. If the target API does not have an OpenAPI spec available, you can use a request template to run a limited set of fuzzers. The syntax for running the TemplateFuzzer is as follows (very similar to curl:

The command will:

It was a deliberate choice to limit the fields for which the Fuzzer will run by supplying them using the -t argument. For nested objects, supply fully qualified names: field.subfield.

Headers can also be fuzzed using the same mechanism as the fields.

This Fuzzer will send the following type of data:

For a full list of options run > cats fuzz -h.

You can also supply your own dictionary of data using the -w file argument.

HTTP methods with bodies will only be fuzzed at the request payload and headers level.

HTTP methods without bodies will be fuzzed at path and query parameters and headers level. In this case you don’t need to supply a -d argument.

This is an example for a GET request:

There are often cases where some fields need to contain relevant business values in order for a request to succeed. You can provide such values using a reference data file specified by the --refData argument. The reference data file is a YAML-format file that contains specific fixed values for different paths in the request document. The file structure is as follows:

For each path you can supply custom values for properties and sub-properties which will have priority over values supplied by any other Fuzzer. Consider this request payload:

and the following reference data file file:

This will result in any fuzzed request to the /path/0.1/auth endpoint being updated to contain the supplied fixed values:

You can use environment (system) variables in a ref data file using: $$VARIABLE_NAME. (notice double $$)

As additional properties are maps i.e. they don’t actually have a structure, CATS cannot currently generate valid values. If the elements within such a data structure are essential for a request, you can supply them via the refData file using the following syntax:

The additionalProperties element must contain the actual key-value pairs to be sent within the requests and also a top element if needed. topElement is not mandatory. The above example will output the following json (considering also the above examples):

The following keywords are reserved in a reference data file: additionalProperties, topElement and mapValues.

You can also have the ability to send the same reference data for ALL paths (just like you do with the headers). You can achieve this by using all as a key in the refData file:

This will try to replace address#zip in all requests (if the field is present).

There are (rare) cases when some fields may not make sense together. Something like: if you send firstName and lastName, you are not allowed to also send name. As OpenAPI does not have the capability to send request fields which are dependent on each other, you can use the refData file to instruct CATS to remove fields before sending a request to the service. You can achieve this by using the cats_remove_field as a value for the fields you want to remove. For the above case the refData field will look as follows:

You can leverage the fact that the FunctionalFuzzer can run functional flows in order to create dynamic --refData files which won’t need manual setting the reference data values. The --refData file must be created with variables ${variable} instead of fixed values and those variables must be output variables in the functionalFuzzer.yml file. In order for the FunctionalFuzzer to properly replace the variables names with their values you must supply the --refData file as an argument when the FunctionalFuzzer runs.

The functionalFuzzer.yml file:

The refData.yml file:

After running CATS using the command and the 2 files above, you will get a refData_replace.yml file where the id will get the value returned into the petId variable.

The refData_replaced.yml:

You can now use the refData_replaced.yml as a --refData file for running CATS with the rest of the Fuzzers.

This can be used to send custom fixed headers with each payload. It is useful when you have authentication tokens you want to use to authenticate the API calls. You can use path specific headers or common headers that will be added to each call using an all element. Specific paths will take precedence over the all element. Sample headers file:

This will add the Accept header to all calls and the jwt header to the specified paths. You can use environment (system) variables in a headers file using: $$VARIABLE_NAME. (notice double $$)

DELETE is the only HTTP verb that is intended to remove resources and executing the same DELETE request twice will result in the second one to fail as the resource is no longer available. It will be pretty heavy to supply a large list of identifiers within the --refData file and this is why the recommendation was to skip the DELETE method when running CATS.

For example, suppose that a POST to /pets responds with:

When doing a DELETE request, CATS will discover that {petId} and pet_id are used as identifiers for the Pet resource, and will do the DELETE at /pets/2.

If these conventions are followed (which also align to good REST naming practices), it is expected that DELETE and POSTrequests will be on-par for most of the entities.

Some APIs might use content negotiation versioning which implies formats like application/v11+json in the Accept header.

You can handle this in CATS as follows:

by having clear separation between versions, you can pass the --contentType argument with the version you want to test: cats ... --contentType="application/v6+json".

If the OpenAPI contract is not version aware (you already exported it specific to a version) and the content looks as:

and you still need to pass the application/v5+json Accept header, you can use the --headers file to add it:

There isn’t a consensus on how you should handle situations when you trail or prefix valid values with spaces. One strategy will be to have the service trimming spaces before doing the validation, while some other services will just validate them as they are. You can control how CATS should expect such cases to be handled by the service using the --edgeSpacesStrategy argument. You can set this to trimAndValidate or validateAndTrim depending on how you expect the service to behave:

This is a global setting i.e. configured when CATS starts and all Fuzzer expects a consistent behaviour from all the service endpoints.

There are cases when certain parts of the request URL are parameterized. For example a case like: /{version}/pets. {version} is supposed to have the same value for all requests. This is why you can supply actual values to replace such parameters using the --urlParams argument. You can supply a ; separated list of name:value pairs to replace the name parameters with their corresponding value. For example supplying --urlParams=version:v1.0 will replace the version parameter from the above example with the value v1.0.

CATS also supports schemas with oneOf, allOf and anyOf composition. CATS wil consider all possible combinations when creating the fuzzed payloads.

The following configuration files: securityFuzzerFile, functionalFuzzerFile, refData support setting dynamic values for the inner fields. For now the support only exists for java.time.* and org.apache.commons.lang3.*, but more types of elements will come in the near future.

Let’s suppose you have a date/date-time field, and you want to set it to 10 days from now. You can do this by setting this as a value T(java.time.OffsetDateTime).now().plusDays(10). This will return an ISO compliant time in UTC format.

A functionalFuzzer using this can look like:

You can also check the responses using a similar syntax and also accounting for the actual values returned in the response. This is a syntax than can test if a returned date is after the current date: T(java.time.LocalDate).now().isBefore(T(java.time.LocalDate).parse(expiry.toString())). It will check if the expiry field returned in the json response, parsed as date, is after the current date.

If you need to run CATS behind a proxy, you can supply the following arguments: --proxyHost and --proxyPort. A typical run with proxy settings on localhost:8080 will look as follows:

By default, CATS trusts all server certificates and doesn’t perform hostname verification.

For two-way SSL you can specify a JKS file (Java Keystore) that holds the client’s private key using the following arguments:

When using the native binaries (not the uberjar) there might be issues when using dynamic values in the CATS files. This is due to the fact that GraalVM only bundles whatever can discover at compile time. The following classes are currently supported:

At this moment, CATS only works with OpenAPI specs and has limited functionality using template payloads through the cats fuzz ... subcommand.

The Fuzzers has the following support for media types and HTTP methods:

If a response contains a free Map specified using the additionalParameters tag CATS will issue a WARN level log message as it won’t be able to validate that the response matches the schema.

source


Leave a Comment

Leave a Reply

Your email address will not be published.

FISSURE – Frequency Independent SDR-based Signal Understanding and Reverse Engineering

Machine Learning: How To Become A Machine Learning Engineer?

Critical RCE Bug Could Let Hackers Remotely Take Over DrayTek Vigor Routers

Hackers Are Stealing Bandwidth For Proxy Services Via Microsoft SQL Servers