Skip to main content

Frequently Asked Questions

What types of errors can Schemathesis identify?​

  • Internal Server Errors: Detecting scenarios where your API may fail.
  • Response conformance: Ensuring responses align with your API's defined schema.
  • Functional Discrepancies: Detecting cases where the API behaves differently from what is outlined in the schema.
  • Security vulnerabilities: Uncovering potential security loopholes in your API.
  • Performance regressions: Highlighting areas where the API's performance may deteriorate, and pinpointing the input parameters that lead to these issues.

How long does it typically take for Schemathesis to test an app?​

The testing duration is influenced by various factors such as the complexity of the API, network connection speed, and the configuration of Schemathesis. Generally, testing ranges from a few seconds to several minutes, although in some exceptional cases, it might extend to an hour or more.

What API specifications are supported?​

Schemathesis supports the following API specifications:

  • OpenAPI 2.0OpenAPI 2.0
  • OpenAPI 3.0OpenAPI 3.0
  • GraphQLGraphQL

We have recently introduced experimental support for OpenAPI 3.1. While it's currently in an experimental stage, we are actively working on enhancing its support based on user feedback and requirements. Feel free to try it out and provide us with your feedback to help us improve!

What should I do if my application doesn’t have an API schema?​

Initially, you can utilize schema generators like flasgger for Python, GrapeSwagger for Ruby, or Swashbuckle for ASP.Net to create an API schema. Subsequently, refining the schema definitions can be achieved by running Schemathesis against the generated API schema.

What environment should I use to run Schemathesis tests?​

We recommend running tests in a staging environment that closely mirrors your production setup to uncover issues effectively while keeping your production system safe. Schemathesis can also run locally via the CLI for added convenience.

danger

Never run Schemathesis tests on your production environment!

How does Schemathesis generate test data?​

Schemathesis employs property-based testing to craft random but meaningful data. The methodology involves:

  • Valid Samples: Generated in alignment with your schema to test expected behavior.
  • Slightly Invalid Inputs: Introduced to assess the API's robustness against incorrect data.
  • Explicit Examples: Extracted from your schema to validate predefined cases.

Swarm Testing and Schema Fuzzing methodologies are employed for diverse and evolving test scenarios.

How does Schemathesis assist in reproducing identified errors?​

Schemathesis minimizes the test case to find the smallest reproducible input data and generates a cURL command, like the example below, for error reproduction. Full response data is also provided.

curl -X POST \
  -d '{"number": ":"}' \
  -H 'Content-Type: application/json' \
https://example.schemathesis.io/internal-server-errors/improper-input-type-handling