GitHub Tutorial

This tutorial will show you how to use GitHub Actions to automate API testing with Schemathesis. By the end of this tutorial, you'll learn how to use and configure Schemathesis in your GitHub Actions in order to effectively test your API.

The tutorial uses Docker Compose to run Demo API implemented with Flask and SQLAlchemy. You'll need a basic understanding of Docker, Python, familiarity with GitHub, and an existing GitHub account.

All the starter code is available in our demo repository.

Set up the demo repository

The repository contains a Flask application with intentional errors for Schemathesis testing. Here is a quick excerpt from the source code to give you a better understanding of the Demo API:

from flask import Flask, jsonify, request

app = Flask(__name__)


@app.route("/improper-unicode-encoding", methods=["POST"])
def improper_unicode_encoding():
    data = request.json
    if "text" not in data:
        return jsonify({"success": False, "error": "Missing text"}), 400

    try:
        # Simulating improper Unicode handling
        data["text"].encode("ascii")
        return jsonify({"success": True})
    except UnicodeEncodeError:
        return jsonify({"success": False, "error": "Unicode error"}), 500

1. Fork the Repository: Create a public fork of our demo repository on your personal GitHub account. A public fork is necessary for PR comment reports.

2. Clone the Repository: Clone your fork to your local machine for direct access to the app.

3. Run the App Locally: Execute docker compose up in your terminal to start the app. Once running, access the API UI at http://127.0.0.1:5123/ui/ to see all the available endpoints.

Running Schemathesis GitHub Action

The Schemathesis GitHub Action automates API testing in your CI pipeline, providing immediate feedback on your API's reliability with each commit.

Adding the GitHub Action

1. Create a New Branch. In your local fork, create a new branch for the GitHub Action integration:

git checkout -b add-schemathesis-action

2. Add the GitHub Action Workflow. Create a new file inside .github/workflows/ named schemathesis.yml. This file will define the Schemathesis action.
3. Define the Action. In schemathesis.yml, add the following basic configuration:

name: Schemathesis Test

on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4

    - name: Start containers
      run: docker compose up -d --build

    - uses: schemathesis/action@v1
      with:
        schema: 'http://127.0.0.1:5123/openapi.json'

    - name: Stop containers
      if: always()
      run: docker-compose down

This setup triggers the action on pull request events and runs Schemathesis tests against the API.

4. Commit and Push. Commit the new file to your branch and push it to GitHub using git commit and git push commands.
5. Open a Pull Request (PR): Go to your GitHub repository and open a new PR for the branch you just pushed. This will trigger the GitHub Action you defined.

info

When opening pull requests, be sure to select your own repository as the base branch.

After you open a PR, you will see how tests are failing in the defined action. Here is an excerpt that contains the found issue and a code sample to reproduce it:

________ POST /internal-server-errors/improper-unicode-encoding ________
1. Test Case ID: F7IxDy

- Server error

[500] Internal Server Error:

    `{"error":"Unicode error","success":false}`

Reproduce with:

    curl -X POST -H 'Content-Type: application/json' -d '{"text": "\u0080"}' http://127.0.0.1:5123/internal-server-errors/improper-unicode-encoding

Customizing Test Runs

You can customize how Schemathesis runs tests via the GitHub Action configuration options. Here is an example of how adjust the schemathesis.yml file to set the Authorization header and set the maximum response time to 200ms:

    # ... Omitted for brevity

    - name: Set access token
      run: echo "ACCESS_TOKEN=super-secret" >> $GITHUB_ENV

    - uses: schemathesis/action@v1
      with:
        schema: 'http://127.0.0.1:5123/openapi.json'
        args: '-H "Authorization: Bearer ${{ env.ACCESS_TOKEN }}" --max-response-time=200'

    # ... Omitted for brevity

For the full list of configuration options, refer to the GitHub Action documentation.

Integrating PR Comments

PR comments enhance your workflow by providing immediate summaries of test failures directly in your pull requests. This integration allows for quicker reviews and identification of potential issues with your API, improving the efficiency of your development process.

tip

SOON: We are going to run extra schema conformance check on all reports uploaded from public GitHub repositories. Stay tuned.

1. Sign Up for Schemathesis.io. Visit Schemathesis.io and create an account to get started. Signing up will enable access to additional features, including PR comments.
2. Install GitHub App. Install Schemathesis GitHub App and give it permissions to your repository. Installing the app allows the bot to make comments in your PRs.
3. Generate a Token. Once signed up, generate a personal access token. This token will authenticate your GitHub Actions with Schemathesis.io.
4. Add Token to Repository Settings. Go to your repository on GitHub, and navigate to Settings > Secrets and variables > Actions. Here, add the generated token as a new secret. Name the secret something recognizable, like SCHEMATHESIS_TOKEN.
5. Update Action Configuration. Edit your schemathesis.yml file and add the token configuration option:

    # ... Omitted for brevity

    - uses: schemathesis/action@v1
      with:
        schema: 'http://127.0.0.1:5123/openapi.json'
        token: ${{ secrets.SCHEMATHESIS_TOKEN }}

    # ... Omitted for brevity

This step ensures that the token is available in your GitHub Action workflow for Schemathesis to use.

6. Observe the PR Comment Report: After integrating PR comments, when your GitHub Actions run, Schemathesis will post a comment on the PR with a summary of any test failures. This comment provides a quick and accessible report, simplifying the review process.

By following these steps, you will have integrated Schemathesis PR comments into your GitHub repository, enabling a more efficient and informed review process for your API changes.

Conclusion

In this tutorial, you've learned how to automate API testing using Schemathesis with GitHub Actions, integrate PR comments for immediate feedback on pull requests, and customize your testing setup.