Parallel Test Execution

Parallel test execution is a widely adopted strategy designed to speed up the execution of large test suites that otherwise would require considerable time to execute.

When running tests in parallel, organizations will spin up multiple CI machines, distribute the tests across these instances and run the tests using the testing framework of choice.

For example, consider the following code running pytest in parallel on 8 machines in GitHub Actions using the pytest-split plugin:

name: Redefine

on:
  pull_request:
    branches: [ "main" ]

jobs:
  worker:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        # Running 8 workers
        index: [1,1,1,1,1,1,1,1]
    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: 3.11

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run pytest
        run: pytest --splits ${{ strategy.job-total }} --group ${{ strategy.job-index }} -n auto tests/

Running tests in parallel means organizations can cut their testing time, but it also means running additional machines which increases the CI costs significantly.

Redefine

Redefine enhances parallel test execution efficiency, yielding not only time savings for developers but also substantial reductions in CI-related costs. There are multiple supported parallel testing modes:

• Redefine Parallel - Optimal test distribution among parallel workers, delivered by Redefine.

• Remote Workers - an Orchestrator + Remote workers architecture, where the Orchestrator runs the prediction first, and the user distributes the tests to the testing machines.