Skip to main content

Frequently Asked Questions

What types of errors can Schemathesis identify?​

  • Internal Server Errors: Detecting scenarios where your API may fail.
  • Response conformance: Ensuring responses align with your API's defined schema.
  • Functional Discrepancies: Detecting cases where the API behaves differently from what is outlined in the schema.
  • Security vulnerabilities: Uncovering potential security loopholes in your API.
  • Performance regressions: Highlighting areas where the API's performance may deteriorate, and pinpointing the input parameters that lead to these issues.

How long does it typically take for Schemathesis to test an app?​

The testing duration is influenced by various factors such as the complexity of the API, network connection speed, and the configuration of Schemathesis. Generally, testing ranges from a few seconds to several minutes, although in some exceptional cases, it might extend to an hour or more.

What API specifications are supported?​

Schemathesis supports the following API specifications:

  • OpenAPI 2.0OpenAPI 2.0
  • OpenAPI 3.0OpenAPI 3.0
  • GraphQLGraphQL

We have recently introduced experimental support for OpenAPI 3.1. While it's currently in an experimental stage, we are actively working on enhancing its support based on user feedback and requirements. Feel free to try it out and provide us with your feedback to help us improve!

What should I do if my application doesn’t have an API schema?​

Initially, you can utilize schema generators like flasgger for Python, GrapeSwagger for Ruby, or Swashbuckle for ASP.Net to create an API schema. Subsequently, refining the schema definitions can be achieved by running Schemathesis against the generated API schema.

Can I install Schemathesis locally?​

Yes, you can install our open-source CLI locally to run Schemathesis on your machine. Follow the installation guide provided in our open-source documentation to get started.

What environment should I use to run Schemathesis tests?​

We recommend running tests in a staging environment that closely mirrors your production setup to uncover issues effectively while keeping your production system safe. Schemathesis can also run locally via the CLI for added convenience.

danger

Never run Schemathesis tests on your production environment!

Which IP addresses and User Agents does Schemathesis use to send test data?​

Schemathesis sends test data from 18.157.113.38 using the Schemathesis.io/0.1 User Agent. Adjust your firewall accordingly. For issues, consult our documentation or contact support.

What is the maximum specification size that Schemathesis can handle in its tests?​

The maximum specification size is 15 MB. This can be adjusted for corporate users — contact support for details.

How does Schemathesis generate test data?​

Schemathesis employs property-based testing to craft random but meaningful data. The methodology involves:

  • Valid Samples: Generated in alignment with your schema to test expected behavior.
  • Slightly Invalid Inputs: Introduced to assess the API's robustness against incorrect data.
  • Explicit Examples: Extracted from your schema to validate predefined cases.

Swarm Testing and Schema Fuzzing methodologies are employed for diverse and evolving test scenarios.

How does Schemathesis assist in reproducing identified errors?​

Schemathesis minimizes the test case to find the smallest reproducible input data and generates a cURL command, like the example below, for error reproduction. Full response data is also provided.

curl -X POST \
  -d '{"number": ":"}' \
  -H 'Content-Type: application/json' \
https://example.schemathesis.io/internal-server-errors/improper-input-type-handling

How does Schemathesis handle sensitive data?​

Schemathesis strictly adheres to data protection regulations, using primarily non-sensitive test data to improve service quality.

For enhanced control and data privacy, customers have the option to opt for an on-premises installation, allowing for complete management and control over their data.

It is also possible to filter out sensitive data using our open-source CLI. For concerns, reach out to us.

What is the data retention policy of Schemathesis?​

The period for which data is retained depends on your plan:

  • Free plan: 14 days
  • Professional / Starter (Team): 60 days
  • Growth: 180 days
  • Enterprise: 1 year

If you require customization or extension of the data retention period, please reach out to our support team for assistance.