Skip to content

Calliope GitLab Integration

Our GitLab platform provides you with an easy way to manage your test scripts and execution. To make use of this feature, you first have to register on our environment. After registration, you have to set up your test suite as a GitLab project. We suggest starting out with forking our Cucumber boilerplate test suite by pressing the fork button right under the repository name. Once you've done that you can then download the repository using the git clone command in combination with the URL that you can find on your newly forked repository page. The resulting command should look similar to this, but using your GitLab account name instead of.

$ git clone<GITLAB_ACCOUNT_NAME>/ecommerce-cucumber-web-test-automation-suite.git

If you do not have Git installed, you can read how to install Git for your respective operating system over here. We also recommend taking a look at this Git tutorial if you are unfamiliar with Git and wish to learn more about it before using it.

When we open the project that we just cloned there are two files that we're interested in for setting up our test suite so it can export the results to Calliope.


The '' file is where we tell GitLab what command to run when a job gets triggered. When you're using our boilerplate you should see something like this:

GitLab with Selenium image running on Cucumber

image: "spritecloud/ruby-selenium-runner:2.6"

- selenium/standalone-chrome:3.8.1

  SELENIUM_URL: "http://selenium__standalone-chrome:4444/wd/hub/"

- bundle install #Install the required gems
- export SELENIUM_ENV=remote

    - bundle exec cucumber -f json -o ./results/latest_results.json && true
    - >
      curl -X POST
      -H "x-api-key:$API_KEY"
      -H "Content-Type:multipart/form-data"
      -F "file[]=@results/latest_results.json"
  when: manual

The following variables are defined:

  • SELENIUM_URL: Is only required for this specific test suite when it's run from GitLab and using the selenium/standalone-chrome service
  • API_KEY: Required to import tests to Can be found on your account page
  • PROFILE_ID: Tests will be imported in one of your profiles on You can find the ID in the curl command (see below).

The 'script:' part is the command that is going to run when the job gets triggered.

  • bundle exec cucumber -f json -o ./results/latest_results.json && true runs all the Cucumber tests and stores the results in './results/latest_results.json'. && true makes sure the next commands are also executed.
  • curl -X POST ... will import the tests into the platform.

Make sure the file @results/latest_results.json points to the results file you generated.

Customized API import curl command

You can find and copy your specific API import curl command on the API-Upload tab in the 'Upload a result' window of any profile page.

Calliope api import

After you've updated the .gitlab-ci.yml file and committed the changes to your repository, your test suite is ready to import results to Calliope, and you can configure Calliope to use the GitLab jobs you've created.

You have a different example?

We're very interested in updating our docs with more examples, so if you have one, feel free to reach out to us!

Configuring GitLab in

We're ready to configure the profile to use GitLab! Make sure you have set your account GitLab settings on your account page.

Now you can select your GitLab test suite, when we go to our profile that we set our test suite to export to and click 'edit profile' we can select a 'repository' and a 'job' from a list.

Calliope gitlab settings

Select the repository you created on our GitLab environment and select the job you created in the '' file ('curl-upload-example' if you used our boilerplate). Finally, we can save the profile by clicking 'save'.

When we return to our dashboard we can see that something has changed, the disabled play button is now active and we can click it to start a test run. Also our profile's page has a two new tabs, 'log' and 'schedules'. In 'log' we can see the logs of our current or latest test run and in 'schedules' we can let our test suite run according to a schedule that we set.

Advanced gitlab-ci.yml setup

Below there will be a more advanced setup for gitlab, which includes:

  • Setting os, build & platform dynamically through an txt file.
    • You will need to create these txt files in your project, and make your own script to set the files to the correct values.
  • Create a yaml template of the Calliope push script, that's included into the jobs.
  • Generates an GitLab Artifact of results.json file.
    • This way, if the import fails for whatever reason, you will still have the result file.
image: spritecloud/ruby-selenium-runner:2.6

  - selenium/standalone-chrome:3.141.59

  - bundle install
  - export SELENIUM_ENV=remote

  RESULT_OUTPUT: results/latest.json

.base_job: &base_job
    when: always
      - results/latest.json # <-- Same as RESULT_OUTPUT
  when: manual
    - TA_OS=`cat results/ta_os.txt` && TA_PLATFORM=`cat results/ta_platform.txt` && TA_BUILD=`cat results/ta_build.txt` 
    - echo "$TA_OS / $TA_PLATFORM / $TA_BUILD"
    - >
      curl -X POST
      -H "x-api-key:${API_KEY}"
      -H "Content-Type:multipart/form-data"
      -F "file[]=@$RESULT_OUTPUT"
      -F "os=$TA_OS"
      -F "build=$TA_BUILD"
      -F "platform=$TA_PLATFORM"
      "$PROFILE_ID/import/cucumber" -k

  <<: *base_job
    PROFILE_ID: 12345
    - bundle exec cucumber -p test -p desktop -f json -o $RESULT_OUTPUT -f pretty -c

  <<: *base_job
    PROFILE_ID: 12346
    - bundle exec cucumber -p test -p mobile -f json -o $RESULT_OUTPUT -f pretty -c

  <<: *base_job
    PROFILE_ID: 12347
    - bundle exec cucumber -p staging -p desktop -f json -o $RESULT_OUTPUT -f pretty -c

  <<: *base_job
    PROFILE_ID: 12348
    - bundle exec cucumber -p staging -p mobile -f json -o $RESULT_OUTPUT -f pretty -c