Skip to content

Calliope GitLab Integration

Our GitLab platform provides you with an easy way to manage your test scripts and execution. To make use of this feature, you first have to register on our gitlab.spritecloud.com environment. After registration, you have to set up your test suite as a GitLab project. We suggest starting out with forking our Cucumber boilerplate test suite by pressing the fork button right under the repository name. Once you've done that you can then download the repository using git clone https://gitlab.spritecloud.com/<GITLAB_ACCOUNT_NAME>/ecommerce-cucumber-web-test-automation-suite.git in combination with the URL that you can find on your newly forked repository page. The resulting command should look similar to this, but using your GitLab account name instead of .

If you do not have Git installed, you can read how to install Git for your respective operating system over this Git tutorial if you are unfamiliar with Git and wish to learn more about it before using it.

When we open the project that we just cloned there are two files within this project that we're interested in for setting up our test suite, so it can export the results to Calliope.

.gitlab-ci.yml

The '.gitlab.ci.yml' file is where we tell GitLab what command to run when a job gets triggered. When you're using our boilerplate you should see something like this:

GitLab with Selenium image running on Cucumber

image: "spritecloud/ruby-selenium-runner:2.6"

services:
- selenium/standalone-chrome:3.8.1

variables:
  SELENIUM_URL: "http://selenium__standalone-chrome:4444/wd/hub/"
  API_KEY: <YOUR_API_KEY>
  PROFILE_ID: <YOUR_PROFILE_ID>

before_script:
- bundle install #Install the required gems
- export SELENIUM_ENV=remote

curl-upload-example:
  script:
    - bundle exec cucumber -f json -o ./results/latest_results.json && true
    - >
      curl -X POST
      -H "x-api-key:$API_KEY"
      -H "Content-Type:multipart/form-data"
      -F "file[]=@results/latest_results.json"
      "https://test.calliope.pro/api/v2/profile/$PROFILE_ID/import/cucumber?tag[]=myos&tag[]=myplatform&tag[]=mybuild"
  when: manual

The following variables are defined:

  • SELENIUM_URL: Is only required for this specific test suite when it's run from GitLab and using the selenium/standalone-chrome service
  • API_KEY: Required to import tests to Calliope.pro. Can be found on your Calliope.pro account page
  • PROFILE_ID: Tests will be imported in one of your profiles on Calliope.pro. You can find the ID in the curl command (see below).

The 'script:' part is the command that is going to run when the job gets triggered.

  • bundle exec cucumber -f json -o ./results/latest_results.json && true runs all the Cucumber tests and stores the results in './results/latest_results.json'. && true makes sure the next commands are also executed.
  • curl -X POST ... will import the tests into the Calliope.pro platform.

Formatter endpoints

It is recommended to add the formatter to the endpoint to obtain the best interpreted results. More information about the recommended formatters here.

Make sure the file @results/latest_results.json points to the results file you generated.

Customized API import curl command

You can find and copy your specific API import curl command on your Calliope.pro upload view:

  • Click the Upload-button at the top of the screen.
  • Select any file to open the upload.
  • Click on the API Upload-tab.
  • Here you will find a code-block containing your personal curl-command.

After you've updated the .gitlab-ci.yml file and committed the changes to your repository, your test suite is ready to import results to Calliope, and you can configure Calliope to use the GitLab jobs you've created.

You have a different example?

We're very interested in updating our docs with more examples, so if you have one, feel free to reach out to us!

Configuring GitLab in Calliope.pro

We're ready to configure the Calliope.pro profile to use GitLab! Make sure you have set your GitLab settings on your company page.

Now you can select your GitLab test suite, when we go to our profile that we set our test suite to export to and click 'edit profile', go to 'GitLab integration' at the left navigation bar where we can select a 'repository', 'branch' and a 'job' from a list.

Calliope gitlab settings

Select the repository you created on our GitLab environment and select the job you created in the '.gitlab.ci.yml' file ('curl-upload-example' if you used our boilerplate). Finally, we can save the profile by clicking 'save'.

When we return to our dashboard we can see that something has changed, there is now a play button, and we can click it to start a test run. Also, our profile's page has a two new tabs, 'log' and 'schedules'. In 'log' we can see the logs of our current or latest test run and in 'schedules' we can let our test suite run according to a schedule that we set.

Advanced gitlab-ci.yml setup

Below there will be a more advanced setup for gitlab, which includes:

  • Setting os, build & platform dynamically through a txt file.
    • You will need to create these txt files in your project, and make your own script to set the files to the correct values.
  • Create a yaml template of the Calliope push script, that's included into the jobs.
  • Generates an GitLab Artifact of results.json file.
    • This way, if the import fails for whatever reason, you will still have the result file.
image: spritecloud/ruby-selenium-runner:2.6

services:
  - selenium/standalone-chrome:3.141.59

before_script:
  - bundle install
  - export SELENIUM_ENV=remote

variables:
  API_KEY: {YOUR_CALLIOPE_API_KEY}
  RESULT_OUTPUT: results/latest.json

.base_job: &base_job
  artifacts:
    when: always
    paths:
      - results/latest.json # <-- Same as RESULT_OUTPUT
  when: manual
  after_script:
    - TA_OS=`cat results/ta_os.txt` && TA_PLATFORM=`cat results/ta_platform.txt` && TA_BUILD=`cat results/ta_build.txt` 
    - echo "$TA_OS / $TA_PLATFORM / $TA_BUILD"
    - >
      curl -X POST
      -H "x-api-key:${API_KEY}"
      -H "Content-Type:multipart/form-data"
      -F "file[]=@$RESULT_OUTPUT"
      -F "tag[]=os:$TA_OS"
      -F "tag[]=build:$TA_BUILD"
      -F "tag[]=platform:$TA_PLATFORM"
      "https://app.calliope.pro/api/v2/profile/$PROFILE_ID/import/cucumber" -k

test-environment-desktop:
  <<: *base_job
  variables:
    PROFILE_ID: 12345
  script:
    - bundle exec cucumber -p test -p desktop -f json -o $RESULT_OUTPUT -f pretty -c

test-environment-mobile:
  <<: *base_job
  variables:
    PROFILE_ID: 12346
  script:
    - bundle exec cucumber -p test -p mobile -f json -o $RESULT_OUTPUT -f pretty -c

staging-environment-desktop:
  <<: *base_job
  variables:
    PROFILE_ID: 12347
  script:
    - bundle exec cucumber -p staging -p desktop -f json -o $RESULT_OUTPUT -f pretty -c

staging-environment-mobile:
  <<: *base_job
  variables:
    PROFILE_ID: 12348
  script:
    - bundle exec cucumber -p staging -p mobile -f json -o $RESULT_OUTPUT -f pretty -c

Using a wildcard to select files for importing

In some cases, some of your test scenarios will produce file result with a randomly generated name.

Random name files

This could be a problem when you want to integrate your Test Automation Suite, and these multiple randomly files are generated in your gitlab.ci.yml. Below are some steps that could help you upload your files

variables:
    API_KEY: <YOUR_API_KEY>
    PROFILE_ID: <YOUR_PROFILE_ID>

image: 
    name: postman/newman_alpine33
    entrypoint: [""]

stages:
    - test

postman_tests:
    stage: test
    script:
      - apk add zip unzip
      - newman run <YOUR_COLLECTION>.json -r cli,junit
      - ls
    artifacts:
      when: always
      expire_in: 1 week
      paths:
        - new-filename.zip
    after_script:
      - cd newman; zip -r ../new-filename.zip *.xml; cd ..
      - ls
      - apk add curl
      - >
        curl -X POST
        -H "x-api-key: ${API_KEY}"
        -F file[]=@newman_result.zip
        "https://app.calliope.pro/api/v2/profile/"${PROFILE_ID}"/import" -k

Note that the command newman run Collection.postman_collection.json -r cli,junit will produce randomly named file like the picture above

  1. Use apk add zip unzip to install zip in your gitlab CI.
  2. To upload multiple randomly named files you need to zip it first and rename the files using zip -r ../new-filename.zip *.xml.
  3. Add curl to Upload your files to calliope.
  4. In the curl post, your file should be file[]=@new-filename.zip.
  5. Check your calliope profile as it should be successfully imported.