GitHub Tutorial

This tutorial will show you how to use GitHub Actions to automate API testing with Schemathesis. By the end of this tutorial, you'll learn how to use and configure Schemathesis in your GitHub Actions in order to effectively test your API.

The tutorial uses Docker Compose to run Demo API implemented with Flask and SQLAlchemy. You'll need a basic understanding of Docker, Python, familiarity with GitHub, and an existing GitHub account.

All the starter code is available in our demo repository.

Set up the demo repository

The repository contains a Flask application with intentional errors for Schemathesis testing. Here is a quick excerpt from the source code to give you a better understanding of the Demo API:

from flask import Flask, jsonify, request

app = Flask(__name__)


@app.route("/improper-unicode-encoding", methods=["POST"])
def improper_unicode_encoding():
    data = request.json
    if "text" not in data:
        return jsonify({"success": False, "error": "Missing text"}), 400

    try:
        # Simulating improper Unicode handling
        data["text"].encode("ascii")
        return jsonify({"success": True})
    except UnicodeEncodeError:
        return jsonify({"success": False, "error": "Unicode error"}), 500

1. Fork the Repository: Create a public fork of our demo repository on your personal GitHub account. A public fork is necessary for PR comment reports.

2. Clone the Repository: Clone your fork to your local machine for direct access to the app.

3. Run the App Locally: Execute docker compose up in your terminal to start the app. Once running, access the API UI at http://127.0.0.1:5123/ui/ to see all the available endpoints.

Running Schemathesis GitHub Action

The Schemathesis GitHub Action automates API testing in your CI pipeline, providing immediate feedback on your API's reliability with each commit.

Adding the GitHub Action

1. Create a New Branch. In your local fork, create a new branch for the GitHub Action integration:

git checkout -b add-schemathesis-action

2. Add the GitHub Action Workflow. Create a new file inside .github/workflows/ named schemathesis.yml. This file will define the Schemathesis action.
3. Define the Action. In schemathesis.yml, add the following basic configuration:

name: Schemathesis Test

on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4

    - name: Start containers
      run: docker compose up -d --build

    - uses: schemathesis/action@v1
      with:
        schema: 'http://127.0.0.1:5123/openapi.json'

    - name: Stop containers
      if: always()
      run: docker-compose down

This setup triggers the action on pull request events and runs Schemathesis tests against the API.

4. Commit and Push. Commit the new file to your branch and push it to GitHub using git commit and git push commands.
5. Open a Pull Request (PR): Go to your GitHub repository and open a new PR for the branch you just pushed. This will trigger the GitHub Action you defined.

info

When opening pull requests, be sure to select your own repository as the base branch.

After you open a PR, you will see how tests are failing in the defined action. Here is an excerpt that contains the found issue and a code sample to reproduce it:

________ POST /internal-server-errors/improper-unicode-encoding ________
1. Test Case ID: F7IxDy

- Server error

[500] Internal Server Error:

    `{"error":"Unicode error","success":false}`

Reproduce with:

    curl -X POST -H 'Content-Type: application/json' -d '{"text": "\u0080"}' http://127.0.0.1:5123/internal-server-errors/improper-unicode-encoding

Customizing Test Runs

You can customize how Schemathesis runs tests via the GitHub Action configuration options. Here is an example of how adjust the schemathesis.yml file to set the Authorization header and set the maximum response time to 200ms:

    # ... Omitted for brevity

    - name: Set access token
      run: echo "ACCESS_TOKEN=super-secret" >> $GITHUB_ENV

    - uses: schemathesis/action@v1
      with:
        schema: 'http://127.0.0.1:5123/openapi.json'
        args: '-H "Authorization: Bearer ${{ env.ACCESS_TOKEN }}" --max-response-time=200'

    # ... Omitted for brevity

For the full list of configuration options, refer to the GitHub Action documentation.

Conclusion

In this tutorial, you've learned how to automate API testing using Schemathesis with GitHub Actions and customize your testing setup.