Skip to content

Knapsack Pro parallel testing

Run 1 hour test suite in 2 minutes with optimal parallelization on your CI

Source: https://knapsackpro.com

Learn by example

Check out our example Knapsack Pro repository.

User Guide

For more information on how to install it and use it, please see the Knapsack Pro integration tutorial.

Supported Knapsack test runners

Knapsack Pro supports a few test runners out of the box:

As of now, Ruby has the most test runners supported by Knapsack:

  • RSpec
  • Cucumber
  • Minitest
  • test-unit
  • Spinach

We will limit this tutorial to Cucumber, as the setup for each test runner is generally pretty much the same.

Less Javascript runners are supported:

  • Cypress.io
  • Jest

We won't handle these in this tutorial, but the general setup is also the same here. For more information, you can read up on the offical Github repositories for Knapsack in Cypress.io and Jest:

We will now provide you with a Calliope pipeline example for Cucumber. We'll limit these to Gitlab and Github Actions for now.

# We setup the services required to perform the tests.
services:
  - selenium/standalone-chrome:3.141.59

# We setup the stages in order to properly use the nodes.
stages:
  - test
  - upload

# We set some variables required for both Knapsack and Calliope.
variables:
  KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER: YourKnapsackApiKey
  CALLIOPE_API_KEY: YourCalliopeApiKey
  PROFILE_ID: YourCalliopeProfileID

# A simple before script to prepare the project for testing.
before_script:
  - gem install bundler
  - bundle install
  - export SELENIUM_ENV=remote

# Here we run the TA suite on 2 nodes to keep things simple, you can increase the amount of nodes if your environment can handle it.
# The more nodes, the quicker your TA suite will be.
cucumber:
  parallel: 2
  stage: test
  script:
    - bundle exec knapsack_pro cucumber "--format json --out artifacts/cucumber_${CI_NODE_INDEX}.json --format pretty" || true
  artifacts:
  paths:
    - artifacts/cucumber_${CI_NODE_INDEX}.json
  allow_failure: true

# To avoid having to change the upload script when deciding to use more nodes, we zip all the result files together to 1 zipfile,
# which we then finally upload to Calliope.
upload:
  dependencies: [cucumber]
  stage: upload
  script:
  - apt-get update
  - apt-get install zip unzip -y
  - cd artifacts; zip -r ../combined-results.zip . -i *.json; cd ..
  - curl -X POST
    -H "Content-Type:multipart/form-data"
    -H "x-api-key:${CALLIOPE_API_KEY}"
    -F "file[]=@combined-results.zip"
    "https://app.calliope.pro/api/v2/profile/${PROFILE_ID}/import/cucumber?tag[]=YOUR_OS&tag[]=YOUR_PLATFORM&tag[]=YOUR_BUILD"

# Here we give a name to the whole pipeline. You can call this whatever you want.
name: CI

# Here we decide when we want to run the pipeline.
# In this example we run it whenever a commit is pushed or a pull request is made to the master branch.
on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    # The default environment of a GitHub job is Ubuntu, which we will leave for what it is.
    # We also define a Ruby version, in this example we use Ruby 2.7.
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        ruby-version: ['2.7']
        ci_node_total: [2]
        # Indexes for parallel jobs (starting from zero).
        # E.g. use [0, 1] for 2 parallel jobs, [0, 1, 2] for 3 parallel jobs, etc.
        ci_node_index: [0, 1]

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    # To automatically get bug fixes and new Ruby versions for ruby/setup-ruby,
    # change this to (see https://github.com/ruby/setup-ruby#versioning):
    # uses: ruby/setup-ruby@v1
    - name: Setup Ruby
      uses: ruby/setup-ruby@473e4d8fe5dd94ee328fdfca9f8c9c7afc9dae5e
      with:
        # We use the Ruby version that we just set. 
        # This step will also automatically install all the gems with Bundler, so we don't have to explicitly run it.
        ruby-version: ${{ matrix.ruby-version }}
        bundler-cache: true 

    # Now we run the Cucumber tests and export their results to a JSON file.
    - name: Run tests
      env: 
        KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
        KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
        KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER }}
      run: bundle exec knapsack_pro cucumber "--profile test --format json --tag @fta --out cucumber_${{ matrix.ci_node_index }}.json --format pretty"

    # We save all the test reports as artifacts.
    - name: Save artifacts for upload
      if: always()
      uses: actions/upload-artifact@v2
      with:
        path: cucumber_${{ matrix.ci_node_index }}.json

  # After the test stage, we arrive at the upload stage.
  upload:
    if: always()
    runs-on: ubuntu-latest
    needs: [test]

    steps:
    # We have to manually download the artifacts first.
    - name: Retrieve test result artifacts
      uses: actions/download-artifact@v2
      with:
        path: results

    # We display the artifacts to the screen to make sure they were exported and imported correctly.
    - name: Display downloaded artifacts
      run: ls -R

    # To avoid having to change the upload script when deciding to use more nodes, we zip all the result files together to upload as 1 file.
    - name: Zip artifacts together
      uses: papeloto/action-zip@v1
      with:
        files: results/
        dest: results.zip

    # Finally we upload the zipped files to Calliope.
    - name: Upload results to Calliope
      run: curl -X POST
           -H "x-api-key:${{ secrets.API_KEY }}"
           -H "multipart/form-data"
           -F "file[]=@./results.zip"
           "https://app.calliope.pro/api/v2/profile/${{ secrets.PROFILE_ID }}/import/cucumber?tag[]=myos&tag[]=myplatform&tag[]=mybuild"

Knapsack supports more than just these two CI/CD tools. You can read up on them in the aforementioned installation tutorial.

Please note that the only CI/CD tools other than these two that we fully support are Jenkins and Travis CI. This does not mean Calliope cannot be used with these other tools, but you will have to implement those yourself.