Few days ago OpenAI released the GPT-4 vision capabilities on their API, so I thought on using AI and GPT4 vision models to analyze images of a security camera connected to a well-known home automation system: Home Assistant.

The goal was to trigger a notification to my mobile, if the system detects a potential security issue at home using AI :-)

Possible? Perhaps! Let's see!

Quick overview

Let me start by introducing some concepts.

About Home Assistant

Home Assistant is a well-known open-source home automation project that enables easy implementation of domotics / home automation systems. It is a very flexible tool that can be extended in order to provide further functionalities. In HASS, we have:

• devices
• entities, usually provided from deviced
• automations, made of "triggers", "conditions" and "actions"
• services
• ... and much more!

About the OpenAI Conversation integration

Home Assistant provides an integration with OpenAI that can:

• enable an assistant using GPT-3 language model, so that users can use the Assist feature to interact and, for example, get information about their devices and sensors using a natural language interface (e.g., english)
• generate images based on a prompt

This integration works but it doesn't provide image analysis though.

Initial challenges

I had a look at the OpenAI Conversation integration provided in Home Assistant, dive into the code and found out that it was using an old version of the OpenAI python library; I tried to bump it to the latest version but the API changed a lot so adapting the existing code was not going to be an easy task to me.

I also don't know much the Home Assistant codebase and Python is not something I use on a daily basis.

However, these challenges wouldn't stop me.

How I made it

Quick oveview

I started by creating a copy of the existing code and tried to "hack it" locally, to a local directory /custom_components/openai_conversation_custom . I had the official integration already installed which somehow conflicted with this code. I then,

• removed the official integration from Home Assistant
• changed the manifest.json

{
 "domain": "openai_conversation_custom",
 "name": "OpenAI Conversation Custom",
 "codeowners": ["@balloob", "@bitcoder"],
 "config_flow": true,
 "dependencies": ["conversation"],
 "documentation": "https://www.home-assistant.io/integrations/openai_conversation",
 "integration_type": "service",
 "iot_class": "cloud_polling",
 "requirements": ["openai==0.27.2", "httpx"],
 "version": "1.0.0-custom-20231124"
}

• did changes on the code to add additional features (more info ahead)
• installed the custom integration

The code I added to support new features

The code that I'm about to share is not totally clean and it's like putting a parallel OpenAI client to support my needs, that include the image processing. Please consider it as a proof-of-concept and adapt it to your needs.

I created my OpenAI client inspired by Piotr Skalski and added it under the myopenai_client directory. The code uses the `httpx`` library for making asynchronous HTTP requests. Let me share some brief snippets of it.

import os
import numpy as np
import httpx

from .utils import compose_payload, compose_text_payload


class OpenAIConnector:

    def __init__(self, api_key: str):
        if api_key is None:
            raise ValueError("API_KEY is not set")
        self.api_key = api_key

    async def simple_prompt(self, prompt: str, image_path: str = None) -> str:
        headers = {
            "Content-Type": "application/json",
            "Authorization": f"Bearer {self.api_key}"
        }
        if image_path is None:
            payload = compose_text_payload(prompt=prompt)
        else:
            payload = compose_payload(image_path=image_path, prompt=prompt)

        async with httpx.AsyncClient() as client:
            response = (await client.post("https://api.openai.com/v1/chat/completions",
                                         headers=headers, json=payload, timeout=20)).json()

        return response['choices'][0]['message']['content']

def compose_payload(image_path: str, prompt: str) -> dict:
    """
    Composes a payload dictionary with a base64 encoded image and a text prompt for the GPT-4 Vision model.

    Args:
        image_path (str): The image path to encode and include in the payload.
        prompt (str): The prompt text to accompany the image in the payload.

    Returns:
        dict: A dictionary structured as a payload for the GPT-4 Vision model, including the model name,
              an array of messages each containing a role and content with text and the base64 encoded image,
              and the maximum number of tokens to generate.
    """

    # encode a base64 image from the image path
    with open(image_path, "rb") as image_file:
         encoded_string = base64.b64encode(image_file.read())
         base64_image = encoded_string.decode('utf-8')

    return {
        "model": "gpt-4-vision-preview",
        "messages": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": prompt
                    },
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/jpeg;base64,{base64_image}"
                        }
                    }
                ]
            }
        ],
        "max_tokens": 600
    }

def compose_text_payload(prompt: str) -> dict:
    """
    Composes a payload dictionary with a base64 encoded image and a text prompt the standard text model.

    Args:
        prompt (str): The prompt text to accompany the image in the payload.

    Returns:
        dict: A dictionary structured as a payload for the GPT-4 Vision model, including the model name,
              an array of messages each containing a role and content with text and the base64 encoded image,
              and the maximum number of tokens to generate.
    """
    return {
        "model": "gpt-4-vision-preview",
        "messages": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": prompt
                    }
                ]
            }
        ],
        "max_tokens": 600
    }

The idea is to provide the possibility of asking something to OpenAI GPT-4 vision model, passing a question (i.e., the prompt) together with the image.

This was encapsulated in a new service. Therefore, on the __init__.py this new service that I called process_image:

async def async_setup(hass: HomeAssistant, config: ConfigType) -> bool:
    """Set up OpenAI Conversation."""

    async def process_image(call: ServiceCall) -> ServiceResponse:
        """Process an image..."""
        api_key = hass.data[DOMAIN][call.data["config_entry"]]
        client = OpenAIConnector(api_key=api_key)
        prompt = call.data["prompt"]
        image_path = call.data["file_path"]
        response = await client.simple_prompt(prompt=prompt, image_path=image_path)

        return {
            "response": response
        }

    hass.services.async_register(
        DOMAIN,
        SERVICE_PROCESS_IMAGE,
        process_image,
        schema=vol.Schema(
            {
                vol.Required("config_entry"): selector.ConfigEntrySelector(
                    {
                        "integration": DOMAIN,
                    }
                ),
                vol.Required("prompt"): cv.string,
                vol.Required("file_path"): cv.string,
            }
        ),
        supports_response=SupportsResponse.ONLY,
    )
 
    return True
...

On the services.yaml I added the description of the configuration for this new service, so that it can be used in an automation action later on.

process_image:
  fields:
    config_entry:
      required: true
      selector:
        config_entry:
          integration: openai_conversation_custom
    prompt:
      required: true
      selector:
        text:
          multiline: true
    file_path:
      required: true
      selector:
        text:
          multiline: false

I ended up having the previous functionality provided by the original OpenAI Conversation integration using its own API client together with a parallel client to support this new service. Not very clean, I know, but it works for my PoC.

Implementation

After having the custom integration installed and configured with the OpenAI API key, I created a few things in my Home Automation.

These were the main automation rules I created.

• Automation rule: take a screenshot whenever the camera detects a movement, and then analyze it using AI

In this rule I'll use the new service that I created earlier. I have to pass a prompt. I'm using the following ones. Note that making a good prompt is essential; you can read a bit more on Prompt Engineering and some good practices.

As a home security expert looking at the photo of a camera, do you see any security related issues that are signs of vandalism, robbery, broken items, open windows, fire, smoke, or other conditions that can put the house and its inhabitants at risk? Please answer with NORMAL or WARNING or CRITICAL in the first line, by increasing risk level respectively, and then please describe the main risks you have identified.

• Automation rule: notify my mobile when the image analysis changes to a certain state (e.g., "CRITICAL" or "WARNING)

In terms of dashboard and entities, I added a few things.

To see the photo that the camera took, I used the local_file camera platform. This allowed to me overcome some issues I faced with the picture entity that wasn't able to refresh the image upon new snapshots being taken. I also created a template based sensor entity to store the status of the image analysis. That way I can also show it on the dashboard.

Seeing it working

After having my Home Assistant configuration worked out, I simulated someone entering the room. This was my dashboard showed.

This is the mobile notification I got.

Cool, isnt' it?

Things to have in mind

There are a few things worth mentioning:

• please test the prompt you use to analyze the images; the quality of results will be highly dependent on that
• don't trust 100% on the image analysis; you can get false positives and false negatives
• test, a lot :)
• and remember, that this is just a PoC!

In sum

This small project was challenging but also fun. It shows how AI, in this case using OpenAI vision and the power of GPT-4, can be use to augment some of the the things we do, such as being used to analyze images from security cameras.

It's not a foolproof solution, but it can be used together with other data to provide some automation in your existing home automation projects! If you want to try this yourself, and since you got to this point on the tutorial, please check out the GitHub repository I put online.

Have fun!

Principle: A belief, a basic truth from where other truths can be built

Whenever testing, we also have a set of principles that guide and support our testing.

Let's see the principles I think might be important.

Exploratory Testing Principles v1.0

1. Testing is about searching for meaningful quality related information

Testing provides information about quality which in turn has multiple dimensions and multiple stakeholders that value those dimensions (quality criteria) differently. The perceived value on each quality dimension also evolves with time...

2. Testing is highly context-dependent and context-driven

The risks that we want to assess and how we perform that assessment depend on the context. Likewise, results of testing are contextualized and need to be interpreted.

3. Testing embraces a whole set of activities, to challenge, investigate, explore, check, report and provide actionable information

Although the essence of testing is around finding problems through investigation, exploration, and checking, testing also involves planning, reporting, and other activities.

4. Testing is mainly a human centered process that can be augmented by tools to an extent

Testing is mainly driven by knowledge, curiosity, and context; these are some main tools used by a tester. Software and hardware tools can also be used to assist testers and augment certain testing activities and may provide information that otherwise would be hard or impossible to obtain.

5. There is no single testing solution

Similarly to coding where there is no single implementation for a given feature, there is also no single testing solution to uncover risks and the risks that matter. However, there are techniques, heuristics, and approaches that maximize results towards specific testing goals.

6. Testing uncovers more risks whenever performed with someone else

Testing can be a “lone ranger” activity but we know that diversity plays a major role around improving knowledge and understanding. Each person has a different background, is aware of different heuristics and risks, and has a different skill set. Testing with someone else, including developers, is a proven-way of exposing risks that otherwise could be missed. Besides, if done with developers for example, it’s a way of improving testing knowledge so that quality can be built-in in the future.

7. Testing requires extensive skills, technical, social, among others

These skills need to evolve, as context evolves, including the software, how it is crafted, how it is delivered and used. The art of crafting value through software is a team effort, thus social and communication skills are also important to foster an overall team improvement environment.

8. Testing is not limited to certain roles, stages, types, levels, or artifacts

Everyone in the team can contribute to testing to have better and timed information about quality, so that risks can be addressed properly and promptly. Testing can and should be applied to all aspects that ultimately lead to a feature in production, in other words, all aspects from ideation up to production.

9. Testing is never complete

As there are numerous stakeholders (internal and external) interested in our products, that in turn have different expectations and use the product differently, the number of test scenarios that we could depict would be infinite. Similarly to coding where we can always refactor code snippets, testing is never complete as we can always learn new information about the software, its internals and its externals. The more time and resources we have, the more we can test. However, there is always a decision of when to stop testing that can be around the confidence the team has so far or restricted by some time constraint, for example.

10. There are no bug free products

Bug free products are a myth. We can only say that we have not found bugs during and for the testing that we performed. Bugs are all aspects that negatively impact quality. Even though we cannot assure a product is bug free, it’s always possible to assess risks and mitigate them through testing, among other options that the team can take.

11. Testing is maximized with clear, meaningful, and fast feedback-loops

Information discovered while testing needs to be shared with the team, so the team can act on it. This information needs to be understandable, manageable, and meaningful. In order for the team to improve, this information needs to be properly delivered and on-time. Enabling fast feedback loops is essential to iterate more often and thus enable agility.

In sum

This is a first iteration of the principles I think may relevant for testers, including exploratory testers; in reality, being exploratory is part of the intrinsic nature of testers. Looking forward for your feedback in order to improve these principles or can even create your own variant of your testing principles.

I've met Grafana in the past but never implemented an end-to-end scenario with it. Well, it was time; hope this helps you out and inspires you to try some of these tools. In this rather long detailed article, we'll also cover InfluxDB and how it can be useful to store time series data.

Background

Xray

Xray is a well-known test management solution that works highly integrated with Jira (it needs it, actually). Xray cloud (i.e., for Jira cloud) provides a REST API and a GraphQL API. The latter is more powerfull and that's the one we can use to extract the test runs.

The Xray server/datacenter product provides a REST API only, for the time being. In this case, there's an endpoint that we can use to export the test runs from.

InfluxDB

InfluxDB is a time series database. It can be used to store and query our time-related data: in this case our test runs. It also provides integration with Grafana among other.

Grafana

Grafana is a platform used to analyze and visualize data, supporting multiple sources, dashboard customization, plugins, and more.

How to

I started by trying to use Grafana directly to import the test runs from the source application (i.e., Xray), using either the GraphQL API, for Xray cloud, or the REST API, for Xray server/DC.

It's possible to do so using some Grafana plugins (GraphQL, JSON API, Infinity) but we will face many limitations, including:

• unable to deal with pagination of the underlying APIs;
• overload of API request, as data needs to be fetched each time it is needed
• limited query capabilities

These are just a few; I won't cover the configuration of these or explore this further to not make this article too exhaustive.

So, as this initial approach was unsuccessful, I tried to depict what would be the proper architecture for this. As a consequence, I came with something like this: Xray > InfluxDB > Grafana.

Extracting test runs from Xray

The first step is to extract data from Xray. Here, we actually have two different product flavors: one for server/datacenter and another cloud based. The underlying APIs are different, so I had to make two different implementations; in your case, you'll probably just need one of them.

Extracting test runs from Xray cloud

To achieve this, I've implemented some code in JavaScript. The code ahead uses a JQL expression to define the source of the data; in this case, I aim to export all test runs from project "CALC" in Jira, based on the related Test Execution issues. We need to obtain the related issue ids (Jira internal ids) of these, as the getTestRuns GraphQL function requires us to pass them (or the Test issue ids, if we prefer to obtain test runs based on that).

The code also has some basic logic to deal with pagination on the GraphQL requests; more error handling should be provided btw as, for example, the API can return temporary errors or rate limiting errors.

There's also logic to export test runs based on a modification date, and use that to avoid exporting them on next export operations.

var axios = require('axios');
const { GraphQLClient, gql } = require('graphql-request')
const fs = require('fs');

var xray_cloud_base_url = "https://xray.cloud.getxray.app/api/v2";
var xray_cloud_graphql_url = xray_cloud_base_url + "/graphql";
var client_id = process.env.CLIENT_ID || "215FFD69FE4644728C72182E00000000";
var client_secret = process.env.CLIENT_SECRET || "1c00f8f22f56a8684d7c18cd6147ce2787d95e4da9f3bfb0af8f02ec00000000";

var authenticate_url = xray_cloud_base_url + "/authenticate";

// Jira JQL query to define the list of Test Execution issues to export test runs from
jql = "project=CALC and issuetype = 'Test Execution'"


async function getTestExecutionIds (jql, start, limit) {
  return axios.post(authenticate_url, { "client_id": client_id, "client_secret": client_secret }, {}).then( (response) => {
    
    var auth_token = response.data;
    const graphQLClient = new GraphQLClient(xray_cloud_graphql_url, {
        headers: {
          authorization: `Bearer ${auth_token}`,
        },
      })

      // console.log(auth_token);

      const testexec_ids_query = gql` 
      query
      {
          getTestExecutions(jql: "${jql}", limit: ${limit}, start: ${start}) {
            results{
              issueId
            }
          }
      }
`

    return graphQLClient.request(testexec_ids_query).then(function(data) {
      testexec_ids = data['getTestExecutions']['results'].map(function(t){
        return t['issueId'];
      });

      // console.log(testexec_ids);
      return testexec_ids;
    }).catch(function(error) {
      console.log('Error performing query to obtain Test Execution ids: ' + error);
    });
  }).catch( (error) => {
      console.log('Error on Authentication: ' + error);
  });
}


async function getTestRuns (testExecIssueIds, start, limit, modifiedSince) {
  return axios.post(authenticate_url, { "client_id": client_id, "client_secret": client_secret }, {}).then( (response) => {

      var auth_token = response.data;
      const graphQLClient = new GraphQLClient(xray_cloud_graphql_url, {
          headers: {
            authorization: `Bearer ${auth_token}`,
          },
        })

        testexec_ids = testExecIssueIds.map(function(t){
          return '"' + t + '"';
        }).join(',');

        // console.log(testexec_ids);

        const query = gql` 
        {
          getTestRuns(testExecIssueIds: [${testexec_ids}], limit: ${limit}, start: ${start}, modifiedSince: "${modifiedSince}"  ) {
            total
            start
        
            results{
              id
              status{
                name
                description
              }
              comment
              evidence{
                filename
                downloadLink
              }
              defects
              executedById
              startedOn
              finishedOn
              assigneeId
        
              testType{
                name
              }
        
              steps {
                  id
                  action
                  data
                  result
                  customFields {
                    name
                    value
                  }
                  comment
                  evidence{
                    filename
                    downloadLink
                  }
                  attachments {
                      id
                      filename
                  }
                  defects
                  actualResult
                  status {
                    name
                  }
              }
        
              scenarioType
              gherkin
              examples {
                  id
                  status {
                      name
                      description
                  }
                  duration
              }
        
              unstructured
              
              customFields {
                  id
                  name
                  values
              }
        
              preconditions(limit:10) {
                results{
                    preconditionRef {
                        issueId
                        jira(fields: ["key"])
                    }
                    definition
                }
              }
              test {
                  issueId
                  jira(fields: ["key"])
              }
              testExecution {
                  issueId
                  jira(fields: ["key", "fixVersions"])
                  testEnvironments
                  testPlans(start: 0, limit: 10) {
										results{
											 jira(fields: ["key"])
										}
									}
              }    
            }
          }
        }  
  `

      return graphQLClient.request(query).then(function(data) {
            return data['getTestRuns']['results'];
      }).catch(function(error) {
        console.log('Error performing query to obtain testruns: ' + error);
      });
  }).catch( (error) => {
      console.log('Error on Authentication: ' + error);
  });
}

/**** main *****/

(async () => {

  let configFile = 'export_testruns.json'
  if (!fs.existsSync(configFile)) {
    fs.writeFileSync(configFile, "{}")
  }
  let config = JSON.parse(fs.readFileSync(configFile));
  let modifiedSince = config['modifiedSince'] || "2021-01-01T00:00:00Z"

  // obtain Test Execution issue ids
  let start = 0
  let limit = 100
  let testexecs = []
  let tes = []
  do {
    tes = await getTestExecutionIds(jql, start, limit)
    start += limit
    testexecs.push(...tes)
  } while (tes.length > 0)

  // obtain the Test Runs for the given Test Execution issue ids, modified since a given data
  let testruns = []
  start = 0
  let trs = []
  do {
    trs = await getTestRuns(testexecs, start, limit, modifiedSince)
    start += limit
    testruns.push(...trs)
  } while (trs.length > 0)
  
  console.log(JSON.stringify(testruns, undefined, 2))
  fs.writeFileSync('testruns.json', JSON.stringify(testruns));

  config['modifiedSince'] = new Date().toISOString().split('.')[0]+"Z"
  fs.writeFileSync(configFile, JSON.stringify(config));
})();

The previous script will generate a testruns.json file, having an array of JSON objects, each one corresponding to a test run.

Extracting test runs from Xray server/datacenter

To achieve this, once again I made some code in JavaScript that uses the Xray server/DC REST API endpoint for exporting test runs.

var axios = require('axios');
const { create } = require('domain');
const fs = require('fs');

var jira_base_url = "https://local_jiraserver.local";
//var personal_access_token = "OTE0ODc2NDE2NTgxOnrhigwOreFoyNIA9lXTZaOcgbNY";
var jira_username = 'someuser'
var jira_password = 'somepass'


async function createJiraFilter(jql) {
  var basicAuth = 'Basic ' + btoa(jira_username + ':' + jira_password);

  let url = jira_base_url + "/rest/api/2/filter"
  const filter = JSON.stringify(
      {
        "jql": jql,
        "name": "temporary filter",
        "description": "temporary filter to indirectly export test runs from",
        "favourite": false
      }
    );

  return axios.post(url, filter, {
    headers: { 'Authorization': basicAuth, 'Content-Type': 'application/json' }
    // headers: { 'Authorization': "Bearer " + personal_access_token }
  }).then(function(response) {
      console.log('success: created temporary filter ' + response.data["id"] + ' in Jira');
      return response.data;
  }).catch(function(error) {
      console.log('Error creating filter: ' + error);
  });
}

async function deleteJiraFilter(filterId) {
  var basicAuth = 'Basic ' + btoa(jira_username + ':' + jira_password);

  let url = jira_base_url + "/rest/api/2/filter/" + filterId
  return axios.delete(url, {
    headers: { 'Authorization': basicAuth }
    // headers: { 'Authorization': "Bearer " + personal_access_token }
  }).then(function(response) {
    console.log('success: created temporary filter ' + filterId + ' in Jira');
      return response.data;
  }).catch(function(error) {
      console.log('Error deleting filter: ' + error);
  });
}


async function getTestRuns(filterId, page, limit) {
  var basicAuth = 'Basic ' + btoa(jira_username + ':' + jira_password);

  var endpoint_url = jira_base_url + "/rest/raven/2.0/testruns";
  const params = new URLSearchParams({
    savedFilterId: filterId,
    includeTestFields: "issuelinks",
    page: page,
    limit: limit
  }).toString();
  const url = endpoint_url + "?" + params;

  return axios.get(url, {
      headers: { 'Authorization': basicAuth }
      // headers: { 'Authorization': "Bearer " + personal_access_token }
  }).then(function(response) {
      // console.log('success');
      // console.log(response.data);
      return response.data;
  }).catch(function(error) {
      console.log('Error exporting test runs: ' + error);
  });
}


/**** main *****/


(async () => {


  /*
    We can either use an existing Jira filter, by its id, or create a temporary one so we can define
     the JQL expression in the code. 
    If we choose the latter, i.e., to create a temporary filter, then we should cleanup at the end of the process.
  */

  // Jira JQL expression to indirectly obtain the test runs from
  jql = "project = BOOK and issuetype = 'Test Execution'"

  // create a temporary filter based on a JQL expression
  let filter =  await createJiraFilter(jql);
  let filterId = filter["id"]

  // obtain the Test Runs for the given Jira filter id
  let testruns = []
  let page = 1
  let limit = 100
  let trs = []
  do {
    trs = await getTestRuns(filterId, page, limit)
    page += 1
    testruns.push(...trs)
  } while (trs.length > 0)
  

  // delete temporary filter, if we created it 
  await deleteJiraFilter(filterId);

  console.log(JSON.stringify(testruns, undefined, 2))
  fs.writeFileSync('testruns_dc.json', JSON.stringify(testruns));

})();

This will generate a JSON file testruns_dc.json containing an array of Test Runs.

Importing test runs to InfluxDB

After we have the test runs in a local file, now we need to import it to our time series database: InfluxDB.

In InfluxDB we have a bunch of concepts; a careful reading should be done before proceeding with an actual implementation.

• bucket: All InfluxDB data is stored in a bucket. A bucket combines the concept of a database and a retention period
• measurement:A measurement acts as a container for tags, fields, and timestamps. Use a measurement name that describes your data.
• timestamp: All data stored in InfluxDB has a _time column that stores timestamps
• point: A point includes the series key, a field (key and value), and a timestamp (and optionally tags).
• series: A series key is a collection of points that share a measurement, tag set, and field key.
• field: A field includes a field key stored in the _field column and a field value stored in the _value column.
• tags: Tags include tag keys and tag values that are stored as strings and metadata; it's like an index

Therefore, we start by creating a bucket (like a database); we can do this from InfluxDB UI. I've named it "xray".

Then we need to think about naming the measurement; but what is exactly a measurement? That's an excellent question. We can think of a measurement like a big table of data. It can be used as an abstraction of a specific measurement (e.g., temperature); it's basically a name that describes our data. I used our test runs as an abstraction of the target of our measurement, and thus I named it "testrun".

Next, we need to think about how to map our test run details. We have fields and tags... so what we should them for?

The following code snippet shows the approach that I've followed, where fields are used to store dynamic information and tags are used for more static kinds of data. We have to be careful with series cardinality or else we'll hit some limits and cause performance issues; see best practices for schema design.

const point = new Point('testrun')
    .timestamp(ts)
    .tag('deployment_type', 'cloud')
    .tag('iter', iter)
    .tag('test_type', test_type)
    .tag('status', status)
    .stringField('test_key', test_key)
    .stringField('testexec_key', testexec_key)
    .stringField('testplan_key', testplan_key)
    .stringField('test_environments', test_environments)
    .stringField('version', version)
    .stringField('revision', revision)
    .stringField('testrun_id', testrun_id)
    .floatField('duration', duration)
    .intField('total_defects', total_defects)
    .intField('total_evidence', total_evidence)

I've used a tag for deployment type (e.g., "cloud" or "server"), another for an iteration identifier to ease my tests, another for the test type, and finally one for the test run status/result.

I've used one of the InfluxDB client examples and adapted it to my needs. The API is straightforward.

There's a small thing to have in mind: the previous JSON files I obtained for Xray server/DC and cloud don't follow exactly the same format. So, I'll need to normalize that whenever I upload these to InfluxDB.

#!/usr/bin/env node

import {InfluxDB, Point, HttpError} from '@influxdata/influxdb-client'
import {url, token, org, bucket} from './env.mjs'
import {hostname} from 'node:os'
import * as fs from 'fs';


function normalizeStatus(status){
  var statuses = {};
  statuses['PASSED'] = 'PASS';
  statuses['FAILED'] = 'FAIL';
  statuses['TO DO'] = 'TODO';

  if (statuses[status]) {
    return statuses[status];
  } else {
    return status;
  }
}

console.log('*** WRITE POINTS ***')
// create a write API, expecting point timestamps in nanoseconds (can be also 's', 'ms', 'us')
const writeApi = new InfluxDB({url, token}).getWriteApi(org, bucket, 'ns')
// setup default tags for all writes through this API
// writeApi.useDefaultTags({location: hostname()})

// a tag to help us diagnose the results
let iter = "i6";

// Revision custom field
let revision_cf = "customfield_10028";

// testEnvironments: dc, cloud via test execution
// fixversion: dc (no!), cloud via testexecution
// revision: dc (no!), cloud via testexecution andd custom field id (e.g., adding "customfield_xxxx" to the "jira" element)
// testplan key: dc (no!), cloud via test execution
// import Xray server/DC testruns
let testruns = JSON.parse( fs.readFileSync('./testruns_dc.json'));
for (let i = 0; i < testruns.length; i++) {
  let ts =  new Date()
  if ((testruns[i]["start"] !== undefined) && (testruns[i]["finish"] !== null)) {
    ts = new Date(testruns[i]["start"])
  }

  let duration = -1
  if ((testruns[i]["finish"] !== undefined) && (testruns[i]["finish"] !== null)) {
    duration = (Date.parse(testruns[i]["finish"]) - ts) / 1000.0;
  }

  let testrun_id = testruns[i]["id"];
  let test_key = testruns[i]["testKey"];
  let testexec_key = testruns[i]["testExecKey"];
  let test_type = testruns[i]["type"];
  let status = normalizeStatus(testruns[i]["status"]);
  let total_defects = testruns[i]["defects"].length;
  let total_evidence = testruns[i]["evidences"].length;

  const point = new Point('testrun')
    .timestamp(ts)
    .tag('deployment_type', 'server')
    .tag('iter', iter)
    .stringField('test_key', test_key)
    .stringField('testexec_key', testexec_key)
    .tag('test_type', test_type)
    .tag('status', status)
    .stringField('testrun_id', testrun_id)
    .floatField('duration', duration)
    .intField('total_defects', total_defects)
    .intField('total_evidence', total_evidence)

  writeApi.writePoint(point)  
  console.log(` ${point.toLineProtocol(writeApi)}`)
}


// import Xray cloud testruns
testruns = JSON.parse( fs.readFileSync('./testruns.json'));
for (let i = 0; i < testruns.length; i++) {
  let ts =  new Date()
  if ((testruns[i]["startedOn"] !== undefined) && (testruns[i]["startedOn"] !== null)) {
    ts = new Date(testruns[i]["startedOn"])
  }

  let duration = -1
  if ((testruns[i]["finishedOn"] !== undefined) && (testruns[i]["finishedOn"] !== null)) {
    duration = (Date.parse(testruns[i]["finishedOn"]) - ts) / 1000.0;
  }

  let testrun_id = testruns[i]["id"];
  let test_key = testruns[i]["test"]["jira"]["key"];
  let testexec_key = testruns[i]["testExecution"]["jira"]["key"];
  let testplan_key = testruns[i]["testExecution"]["testPlans"]["results"].map( function(tp) {
    return tp["jira"]["key"];
  }).join(',');
  let test_environments = testruns[i]["testExecution"]["testEnvironments"].join(',');
  let version = testruns[i]["testExecution"]["jira"]["fixVersions"].map( function(version) {
    return version["name"];
  }).join(',');
  let revision = testruns[i]["testExecution"]["jira"][revision_cf];

  let test_type = testruns[i]["testType"]["name"];
  let status = normalizeStatus(testruns[i]["status"]["name"]);
  let total_defects = testruns[i]["defects"].length;
  let total_evidence = testruns[i]["evidence"].length;

  const point = new Point('testrun')
    .timestamp(ts)
    .tag('deployment_type', 'cloud')
    .tag('iter', iter)
    .stringField('test_key', test_key)
    .stringField('testexec_key', testexec_key)
    .stringField('testplan_key', testplan_key)
    .stringField('test_environments', test_environments)
    .stringField('version', version)
    .stringField('revision', revision)
    .tag('test_type', test_type)
    .tag('status', status)
    .stringField('testrun_id', testrun_id)
    .floatField('duration', duration)
    .intField('total_defects', total_defects)
    .intField('total_evidence', total_evidence)
  writeApi.writePoint(point)  
  console.log(` ${point.toLineProtocol(writeApi)}`)
}


// WriteApi always buffer data into batches to optimize data transfer to InfluxDB server.
// writeApi.flush() can be called to flush the buffered data. The data is always written
// asynchronously, Moreover, a failed write (caused by a temporary networking or server failure)
// is retried automatically. Read `writeAdvanced.js` for better explanation and details.
//
// close() flushes the remaining buffered data and then cancels pending retries.
try {
  await writeApi.close()
  console.log('FINISHED!')
} catch (e) {
  console.error(e)
  if (e instanceof HttpError && e.statusCode === 401) {
    console.log('Try to setup a new InfluxDB database.')
  }
  console.log('\nFinished with ERROR')
}

After uploading the data, we can query it using Flux.

A Flux script/query starts by defining the source bucket using from(bucket: "somebucket"), then an absolute/relative time range using range(), followed by one or more filters using filter(fn: (r) => ... ).

After data is retrieved it can be transformed (e.g., grouped, manipulated). I would recommend having a look at Time to Awesome online book.

A simple Flux query example follows.

from(bucket: "somebucket")
|> range(start: -5d, stop: -1d)
|> filter(fn: (r) => r._field == "field1")
|> group(columns: ["tag1", "tag2"])

We'll see ahead some concrete Flux examples.

Configuring and using Grafana to analyze and visualize the test runs

In Grafana, we need to configure a data source.

Then we can create a dashboard and add multiple panels to visualize the information we need.

We may find it useful to define some variables that apply across all panels and related queries that we'll use on our dashboard. These can be used later on using the ${variable_name} syntax.

Total Test Runs, for the selected timeframe

In this example, we need to start by joining the multiple tables from the input stream. For that we use the group() statement; we may specify the column it will use, which by default will be the _value column. We can then use the count() function.

from(bucket: "xray")
  |> range(start: v.timeRangeStart, stop:v.timeRangeStop)
  |> filter(fn: (r) =>
    r._measurement == "testrun" and
    r._field == "duration" 
    and r.iter == "${iter}"
  )
 |> group()
 |> count(column: "_value")
 |> group()

Total Defects, for the selected timeframe

Here we use the sum() aggregation function.

from(bucket: "xray")
  |> range(start: v.timeRangeStart, stop:v.timeRangeStop)
  |> filter(fn: (r) =>
    r._measurement == "testrun" and
    r._field == "total_defects" 
    and r.iter == "${iter}"
  )
|> group()
|> sum(column: "_value")
|> group()

Number of Test Runs, grouped in 2 months blocks, for the selected timeframe

In this example, we use the aggregateWindow() function which is a nice way to perform a calculus for a certain time window. The following snippet shows doing a count for every 2 months window. Note that we have to do a group() before to join the multiple input data tables.

from(bucket: "xray")
  |> range(start: v.timeRangeStart, stop:v.timeRangeStop)
  |> filter(fn: (r) =>
    r._measurement == "testrun" and
    r._field == "duration" 
    and r.iter == "${iter}"
  )
 |> group()
 |> aggregateWindow(every: 2mo, fn: count, createEmpty: false)

Number of defects reported on Test Runs, grouped in 3 months blocks, for the selected timeframe

In this case, we use the aggregateWindow() together with the sum aggregation function.

from(bucket: "xray")
  |> range(start: v.timeRangeStart, stop:v.timeRangeStop)
  |> filter(fn: (r) =>
    r._measurement == "testrun" and
    r._field == "total_defects" 
    and r.iter == "${iter}"
  )
|> group()
|> aggregateWindow(every: 3mo, fn: sum, createEmpty: false)

Table with top 3 Test Runs by duration, for the selected timeframe

Here I use the top() function to obtain 3 rows of data (i.e., 3 test runs) based on the duration field. There's a nice trick here: the schema.fieldsAsCols() function will map all the fields to columns, named by the respective field key.

I also have created a new column "url", using the Flux map() function, that creates the URL based on two other columns that contain the Test and the Test Execution issue keys. I've considered a typical Xray cloud link for seeing the test run details.

Then I added an override to format the "url" field as a link, using a "data link" with the value ${__value.text}.

import "influxdata/influxdb/schema"

from(bucket: "xray")
  |> range(start: v.timeRangeStart, stop:v.timeRangeStop)
  |> filter(fn: (r) =>
    r._measurement == "testrun" and
    (r._field == "duration"  or r._field == "testrun_id" or r._field == "test_key" or r._field == "testexec_key")
    and r.iter == "$iter" 
  )
|> schema.fieldsAsCols()
|> group()
|> keep(columns: ["_time", "testrun_id", "test_key", "testexec_key", "test_type", "duration"])
|> top(n: 3, columns: ["duration"])
|> map(fn: (r) => ({r with url:  "https://sergiofreire.atlassian.net/plugins/servlet/ac/com.xpandit.plugins.xray/execution-page?ac.testExecIssueKey=${r.testexec_key}&ac.testIssueKey=${r.test_key}"  
}))

Table with the 3 most recent Test Runs having defects, for the selected timeframe

In this case, mostly similar to the previous one, I've used two filter() statements, where the last one filters test runs that just have reported defects.

import "influxdata/influxdb/schema"

from(bucket: "xray")
  |> range(start: v.timeRangeStart, stop:v.timeRangeStop)
  |> filter(fn: (r) =>
    r._measurement == "testrun" and
    (r._field == "duration"  or r._field == "testrun_id" or r._field == "test_key" or r._field == "testexec_key" or r._field == "total_defects")
    and r.iter == "$iter" 
  )
|> schema.fieldsAsCols()
|> group()
|> keep(columns: ["_time", "testrun_id", "test_key", "testexec_key", "test_type", "total_defects"])
|> filter(fn: (r) =>
  r.total_defects > 0 
)
|> top(n: 3, columns: ["_time"])
|> map(fn: (r) => ({r with url:  "https://sergiofreire.atlassian.net/plugins/servlet/ac/com.xpandit.plugins.xray/execution-page?ac.testExecIssueKey=${r.testexec_key}&ac.testIssueKey=${r.test_key}"  
}))

Test Runs counters by status

from(bucket: "xray")
  |> range(start: v.timeRangeStart, stop:v.timeRangeStop)
  |> filter(fn: (r) =>
    r._measurement == "testrun" and
    r._field == "duration" 
    and r.iter == "${iter}"
  )
 |> group(columns: ["status"])
 |> count(column: "_value")
 |> group()

Test Run duration (mean) grouped by test type

from(bucket: "xray")
  |> range(start: v.timeRangeStart, stop:v.timeRangeStop)
  |> filter(fn: (r) =>
    r._measurement == "testrun" and
    r._field == "duration" 
    and r.iter == "${iter}"
  )
 |> group(columns: ["test_type"])
 |> aggregateWindow(every: 2mo, fn: mean, createEmpty: false)

Final considerations

Overall, this was a very interesting exercise.

Extracting data from my source tool (i.e. Xray) was the easiest part, honestly.

I struggled more with InfluxDB kind of concepts; I'm glad I had a call with their team, which proactively reached out and helped clarify some basics. The biggest challenge for me was moving from a SQL mindset to a time series database, where data is returned as a stream of tables, for example.

Grafana is a very powerful tool, and its integration with InfluxDB seems a good fit. Sometimes I had some doubts with renaming columns or grouping data, if I should do it directly at the InfluxDB level (i.e., on the query), or by using the transformations that Grafana provides.

In the end I was able to implement a bunch of charts and listings with information that could be useful from a testing perspective.

I really think that these are powerful tools that can probably address many reporting kinds of needs teams may have. Some of the team I worked with, used Grafana for different purposes (e.g., testing, infrastructure/monitoring metrics).

Maybe these can also be useful to you, who knows?

Useful references

• Xray cloud GraphQL API
• Xray server/DC REST API endpoint for exporting test runs
• Grafana
• InfluxDB cloud documentation
• Flux scripting language
• Time to Awesome, a book on InfluxDB

This post I about sharing my learnings and how I got it working.

Background

Xray

Xray is a well-known test management solution that works highly integrated with Jira (it needs it, actually). Xray cloud (i.e., for Jira cloud) provides a REST API and a GraphQL API. The latter is more powerfull and that's the one we can use to extract the test runs.

Splunk

Splunk Cloud Platform is a data oriented platform used to search, visualize, and process data from multiple sources, providing analytics and observability kind of capabilities.

How to

The first step is to extract data from Xray cloud. To achieve this, I've implemented some code in JavaScript. The code ahead uses a JQL expression to define the source of the data; in this case, I aim to export all test runs from project "CALC" in Jira, based on the related Test Execution issues. We need to obtain the related issue ids (Jira internal ids) of these, as the getTestRuns GraphQL function requires us to pass them (or the Test issue ids, if we prefer to obtain test runs based on that).

The code also has some basic logic to deal with pagination on the GraphQL requests; more error handling should be provided btw as, for example, the API can return temporary errors or rate limiting errors.

There's also logic to export test runs based on a modification date, and use that to avoid exporting them on next export operations.

var axios = require('axios');
const { GraphQLClient, gql } = require('graphql-request')
const fs = require('fs');

var xray_cloud_base_url = "https://xray.cloud.getxray.app/api/v2";
var xray_cloud_graphql_url = xray_cloud_base_url + "/graphql";
var client_id = process.env.CLIENT_ID || "215FFD69FE4644728C72182E00000000";
var client_secret = process.env.CLIENT_SECRET || "1c00f8f22f56a8684d7c18cd6147ce2787d95e4da9f3bfb0af8f02ec00000000";

var authenticate_url = xray_cloud_base_url + "/authenticate";

// Jira JQL query to define the list of Test Execution issues to export test runs from
jql = "project=CALC and issuetype = 'Test Execution'"


async function getTestExecutionIds (jql, start, limit) {
  return axios.post(authenticate_url, { "client_id": client_id, "client_secret": client_secret }, {}).then( (response) => {
    
    var auth_token = response.data;
    const graphQLClient = new GraphQLClient(xray_cloud_graphql_url, {
        headers: {
          authorization: `Bearer ${auth_token}`,
        },
      })

      // console.log(auth_token);

      const testexec_ids_query = gql` 
      query
      {
          getTestExecutions(jql: "${jql}", limit: ${limit}, start: ${start}) {
            results{
              issueId
            }
          }
      }
`

    return graphQLClient.request(testexec_ids_query).then(function(data) {
      testexec_ids = data['getTestExecutions']['results'].map(function(t){
        return t['issueId'];
      });

      // console.log(testexec_ids);
      return testexec_ids;
    }).catch(function(error) {
      console.log('Error performing query to obtain Test Execution ids: ' + error);
    });
  }).catch( (error) => {
      console.log('Error on Authentication: ' + error);
  });
}


async function getTestRuns (testExecIssueIds, start, limit, modifiedSince) {
  return axios.post(authenticate_url, { "client_id": client_id, "client_secret": client_secret }, {}).then( (response) => {

      var auth_token = response.data;
      const graphQLClient = new GraphQLClient(xray_cloud_graphql_url, {
          headers: {
            authorization: `Bearer ${auth_token}`,
          },
        })

        testexec_ids = testExecIssueIds.map(function(t){
          return '"' + t + '"';
        }).join(',');

        // console.log(testexec_ids);

        const query = gql` 
        {
          getTestRuns(testExecIssueIds: [${testexec_ids}], limit: ${limit}, start: ${start}, modifiedSince: "${modifiedSince}"  ) {
            total
            start
        
            results{
              id
              status{
                name
                description
              }
              comment
              evidence{
                filename
                downloadLink
              }
              defects
              executedById
              startedOn
              finishedOn
              assigneeId
        
              testType{
                name
              }
        
              steps {
                  id
                  action
                  data
                  result
                  customFields {
                    name
                    value
                  }
                  comment
                  evidence{
                    filename
                    downloadLink
                  }
                  attachments {
                      id
                      filename
                  }
                  defects
                  actualResult
                  status {
                    name
                  }
              }
        
              scenarioType
              gherkin
              examples {
                  id
                  status {
                      name
                      description
                  }
                  duration
              }
        
              unstructured
              
              customFields {
                  id
                  name
                  values
              }
        
              preconditions(limit:10) {
                results{
                    preconditionRef {
                        issueId
                        jira(fields: ["key"])
                    }
                    definition
                }
              }
              test {
                  issueId
                  jira(fields: ["key"])
              }
              testExecution {
                  issueId
                  jira(fields: ["key"])
              }      
            }
          }
        }  
  `

      return graphQLClient.request(query).then(function(data) {
            //console.log(JSON.stringify(data['getTestRuns']['results'], undefined, 2))
            return data['getTestRuns']['results'];
            // testruns.push(data['getTestRuns']['results'])
      }).catch(function(error) {
        console.log('Error performing query to obtain testruns: ' + error);
      });
  }).catch( (error) => {
      console.log('Error on Authentication: ' + error);
  });
}

/**** main *****/

(async () => {

  let configFile = 'export_testruns.json'
  if (!fs.existsSync(configFile)) {
    fs.writeFileSync(configFile, "{}")
  }
  let config = JSON.parse(fs.readFileSync(configFile));
  let modifiedSince = config['modifiedSince'] || "2021-01-01T00:00:00Z"

  // obtain Test Execution issue ids
  let start = 0
  let limit = 100
  let testexecs = []
  let tes = []
  do {
    tes = await getTestExecutionIds(jql, start, limit)
    start += limit
    testexecs.push(...tes)
  } while (tes.length > 0)

  // obtain the Test Runs for the given Test Execution issue ids, modified since a given data
  let testruns = []
  start = 0
  let trs = []
  do {
    trs = await getTestRuns(testexecs, start, limit, modifiedSince)
    start += limit
    testruns.push(...trs)
  } while (trs.length > 0)
  
  console.log(JSON.stringify(testruns, undefined, 2))
  fs.writeFileSync('testruns.json', JSON.stringify(testruns));

  config['modifiedSince'] = new Date().toISOString().split('.')[0]+"Z"
  fs.writeFileSync(configFile, JSON.stringify(config));
})();

The previous script will generate a testruns.json file, having an array of JSON objects, each one corresponding to a test run.

Now we need to import it to Splunk. In Splunk we have events; our test runs will be abstract as Splunk events.

In Splunk, we start by creating a new "source type", and name it (e.g., "_xray_graphql"). Go to Settings > Source Types and choose a structured, JSON kind of source. Specify the timestamp fields (e.g., "startedOn", "finishedOn"); these fields should be present on ech test run JSON object.

Then, we define a new data input, by creating a token for the HTTP Event Collector (HEC). Go to Settings > Data Inputs and select the HTTP Event Collector, and make sure you select the source type created earlier.

We can submit one event or multiple events at once using the HTTP Event Collector instance/token we created. In this case, we'll submit an array with multiple test runs. I've created a shell script import_testruns_to_splunk.sh to assist me on this.

#!/bin/bash
 
FILE=$1
curl -k https://prd-p-ys24n.splunkcloud.com:8088/services/collector/raw -H "Authorization: Splunk 75db55f8-b89f-4bc5-0000-00000000" -X POST -H "Content-Type: application/json" -d @"$FILE"

I can then import the test runs as follows.

$ ./import_runs_to_splunk.sh testruns.json
{"text":"Success","code":0 }

These events (i.e., our test runs) will be assigned to the source we identified (e.g., "http2"). We can then use that to filter them later on.

Using Splunk to analyze the test runs

Searching for testruns is straightforward.

We can search by events (i.e., our testruns), based on the source... or on the sourcetype... or using both fields; in this case the result wil be the same.

source="http2" sourcetype="_xray_graphql"

Search for successful (i.e., "passed") test runs

We can easily search by the status reported for the test run.

source="http2" sourcetype="_xray_graphql" "status.name"=PASSED

Search for test runs having a certain comment

source="http2" sourcetype="_xray_graphql" comment = "*interesting*"

Search for test runs having linked defects

The following query will obtain the test runs having linked defects, either globally or at step-level.

source="http2" sourcetype="_xray_graphql" ("defects{}"="*" OR "steps{}.defects{}"="*")

Search for test runs for a certain Test issue

source="http2" sourcetype="_xray_graphql" test.jira.key="CALC-100"

Chart: Top tests failing

source="http2" sourcetype="_xray_graphql" status.name="FAILED" | top limit=5 test.jira.key

Chart: Top tests with more runs

source="http2" sourcetype="_xray_graphql" | top limit=5 test.jira.key

Chart: Show test results over time

source="http2" sourcetype="_xray_graphql" | timechart count by status.name

Pivot tables: show test runs count for Tests, grouped by status

Key findings

These are a sum of my findings for this brief exercise:

• we can send an event or several events in bulk by HTTP to Splunk (we can also upload the test runs as a JSON file)
• in Splunk test runs become easily searchable
• in Splunk different types of charts can be done based on fields of the test runs
• Splunk supports pivot tables based on filed on the test runs
• it's possible to use drill-down on the charts and tables

Final considerations

This was a very interesting exercise. First, it showed how easy it could be to export information from this test management tool to another solution where we can analyze data in further detail, and eventually correlate it with something else.

Splunk provides powerfull capabilities for searching, analyzing, and visualizing data. Events can be anything we want; in this case, we used events as an abstraction for our test runs.

Additional work needs to be performed to have better control over the data that is imported and how it is represented in Splunk.

Useful references

• Xray cloud GraphQL API
• Splunk Cloud Platform
• Splunk's HTTP Event Collector

This is mostly a story about learning. In this article I'll go over the purpose of my contribution, how you can take advantage of it with concrete Playwright examples, and then I'll share what I learned throughout this process.

Purpose

As team members (developers, testers) writing test automation scripts to verify expected behaviour in our web sites/apps, we implement a set of actions and one or more assertions. Many times this is done at the UI layer. After running our tests (Selenium/Webdriver, Cypress, or Playwright based) we get a report that we can use to analyze the test results, either by looking at an HTML kind of report or by uploading it to a central place/tool where we can analyze it in more detail, for example to track historical results.

But the value provided by our test scripts is not just limited to the pass/fail result and the duration of the test. We need to be able to understand what happened, so we can diagnose the failure and get to the actual root-cause. Sometimes we may need to embed additional information, either as text or to a specific variable Screenshots are also quite useful, especially upon failures.

Initally this effort was mainly targeted at enabling this information for Xray, a well-known test management tool. But this need goes beyond this specific tool, and many users out there can also benefit from it if we (I and the community) could make it broad enough.

Embedding additional information in your tests

In Playwright, you can implement a test using test() as shown in the following example for validating the successful login flow in a certain website.

const { LoginPage } = require("../models/Login.js");

test.beforeEach(async ({ page }) => {
  await page.goto('https://robotwebdemo.herokuapp.com');
});


test.describe("Login validations", () => {

    test('Login with valid credentials', async({ page }, testInfo) => {
        const loginPage = new LoginPage(page);
        await loginPage.navigate();
        await loginPage.login("demo", "mode");
        const name = await loginPage.getInnerText();
        expect(name).toBe('Login succeeded. Now you can logout.');
    });

}

But how can we provide additional information that can be embed on the generated report (e.g., JUnit XML), so that other tools can take advantage of?

Some time ago, I've helped to define an evolved JUnit XML format that could provide the means to embed additional information at testcase level. JUnit XML doesn't have a very strict format but the fact that. I've quickly shared this format through the JUnit team and no big blockers appeared. Of course that only tools that can understand this evolved format, can then take advantage of it. Other tools should simply ignore them, except for one or another that have strict schema validation (whatever that schema may be).

This evolved JUnit XML format is now supported by Xray, TestRail (in-progress), and possibly other tools.

Then the question becomes: How can we make our custom metadata into the JUnit XML report?

Well, that requires testing frameworks to have a way of specifying this and somehow associate it to the current test.

That is precisely the scope of the contribution I've made for Playwright Test, the test runner provided by the Playwright team.

Configuration

To take advantage of the new capabilities which allow us to embed additional information at test level, first we need to configure Playwright to use the JUnit reporter; we then also need to configure a few additional settings. More info here.

In the Playwright configuration file (e.g., playwright.config.js), we would have something such this code snippet.

// JUnit reporter config for Xray
const xrayOptions = {
  // Whether to add <properties> with all annotations; default is false
  embedAnnotationsAsProperties: true,

  // By default, annotation is reported as <property name='' value=''>.
  // These annotations are reported as <property name=''>value</property>.
  textContentAnnotations: ['test_description', 'testrun_comment'],

  // This will create a "testrun_evidence" property that contains all attachments. Each attachment is added as an inner <item> element.
  // Disables [[ATTACHMENT|path]] in the <system-out>.
  embedAttachmentsAsProperty: 'testrun_evidence',

  // Where to put the report.
  outputFile: './xray-report.xml'
};

const config = {
  reporter: [ ['junit', xrayOptions] ]
};

module.exports = config;

After we've configured Playwright, we can focus on writting the Playwright test scripts.

To add custom metadata, we take advantage of Playwright annotations. We can access the TestInfo object for that, either by using test.info() or by specifying it as a parameter test('', async ({}, testInfo) => {});.

test('example1', async ({ page }) => {
  test.info().annotations.push({ type: 'issue', description: 'XPTO-1' });
  // ...
});

test('example1',  async ({ page }, testInfo) => {
  testInfo.annotations.push({ type: 'issue', description: 'XPTO-1' });
  // ...
});

In this case, assuming we have Xray Test Management, it understands at least these annotations:

• test_key: map this test code to an existing Test issue in Xray having the specified issue key;
• test_id: map this test code to an existing Test issue in Xray having the specified issue id; not that much used though;
• test_summary: the summary of the Test issue that will be autoprovisioned in Xray;
• test_description: the description of the Test issue that will be autoprovisioned in Xray;
• tags: list of one or more tags that will be added as labels on the corresponding Test issue in Xray, delimited by comma;
• testrun_comment: text comment to add to the test run as associated with the Test issue in Xray;
• requirements: issue keys of one or several stories/requirements in Xray, delimited by comma.

The test_description annotation as can be a lengthy text, will be encoded slightly different on the JUnit XML report; that's precisely the reason we have identified it as text on the textContentAnnotations setting in our config file shown above.

As we also aim to embed files (e.g., screenshots, other relevant evidence) and associate them to our test result, we need to identify the annotation we'll use for this using the embedAttachmentsAsProperty setting; in our case, we'll use the testrun_evidence.

Link the test to a story or a requirement in Jira

In this case, we're covering an existing story with this test, right from the test code. For that we add an annotation requirements having the value of the issue key of the item (story, requirement, or similar) in Jira; usually a test is linked to just one story/requirement. This information will be added as a <property> on the underlying <testcase> element of the JUnit XML reporter produced by the built-in junit reporter, which then tools can take advantage of.

test('Login with valid credentials', async({ page }, testInfo) => {
        testInfo.annotations.push({ type: 'requirements', description: 'CALC-5' });

        const loginPage = new LoginPage(page);
        await loginPage.navigate();
        await loginPage.login("demo", "mode");
        const name = await loginPage.getInnerText();
        expect(name).toBe('Login succeeded. Now you can logout.');
    });

If, for example, you're using Xray for test management in Jira, then you whenever you upload the JUnit XML report (e.g., by REST API, Jenkins plugin, or other CI/CD tool), Xray will process it and will make the linkage/coverage between the corresponding Test and the identified story/requirement.

Here's an example of how it would look like in your Jira instance.

Attach screenshots and/or other files

In this case, we want to add some screenshot or even some other evidence as files associated with a given test result.

There are multiple ways to achieve it; internally they all work the same way as they'll be used to fill out an attachments array on the related TestInfo object.

test('Login with valid credentials', async({ page }, testInfo) => {
        const loginPage = new LoginPage(page);
        await loginPage.navigate();
        await loginPage.login("demo", "mode");

        // Capture a screenshot to a local file and attach it
        const path = testInfo.outputPath('screenshot.png');
        await page.screenshot({ path });
        testInfo.attachments.push({ name: 'screenshot.png', path, contentType: 'image/png' });

        // attach a text file; in this case we're writing the file first to the FS
        const file = testInfo.outputPath('evidence1.txt');
        require('fs').writeFileSync(file, 'hello', 'utf8');
        await testInfo.attach('evidence1.txt', { path: file, contentType: 'text/plain' });

        // attach a text content as file, without ever creating it
        await testInfo.attach('evidence2.txt', { body: Buffer.from('world'), contentType: 'text/plain' });

        const name = await loginPage.getInnerText();
        expect(name).toBe('Login succeeded. Now you can logout.');
    });

Here's an example of how it would look like in Jira, in case you have Xray and have uploaded the JUnit XML report.

If we're focused on screenshots, Playwright can be configured to take screenshots on every test, or even better: if a test fails.

For the latter, we would adjust our Playwright config file as follows and everything will "automagically" work, without having to change the existing test code base.

...
const config = {
  reporter: [ ['junit', xrayOptions] ],
  use: {
    screenshot: 'only-on-failure'
  }
};

module.exports = config;

Learnings

After this process, I do have some learnings to share. Some of them are not new but I think it's worth mentioning them and reflecting a bit about them.

Embrace the challenge

Contributing to a major and well-known project, with a large code base, using a language that you don't know (i.e., TypeScript), and a set of practictes you aren't fully aware of, can be daunting. Besides, you see yourself trying to contribute to a project having people with much more expertise than you. Well, does this make it impossible? Not really. I told to myself, lets embrace the challenge and use it to learn more about Playwright, TypeScript, and also about interacting/contributing to a well-established project. It all started by an initial ticket/issue raised on GH, to get discussion started.

From a working prototype to prod is a long road

Making a working prototype is sometimes quite easy; you can make a hack to the existing code that makes what you need. But making something that can get to production and be used successfuly by others requires some additional work.

We need to:

• make your contribution not so specific
• ensure to provide some automated tests along with your code changes
• ensure (good) documentation
• be open for feedback; be kind but be able to mention the points that matter

Working in a totally remote environment, without calls, is possible

Working in a fully remote environment, is natural for most open-source projects. Playwright is a projected backed by Microsoft, so there is a team behind it. Besides, there is also the broad Playwright team made by all of its contributors.

During the discovery phase/initial conversations, or even during code review, having feedback on a reasonable time frame is key for success.

I was surprised by how helpful the team was, not just because I received feedback on time but also because they helped me throughout the process. I was a first time commiter on the project and I was also writing code in TypeScript for the first time... and it all went fine!

What made the contribution easier

Besides the help from the Playwright community/maintainers, the project itself is set up in a way that eases contribuitions. First, the contribution process is described which removes some initial friction. Second, the build process makes linting on the code and on the documentation; besides, a full battery of around 8.000 (!) automated test scripts ensures a good enough coverage. There are additional checks (e.g., for dependencies) that provide one additional layer of confidence.

Async can be challenging

Sometimes you just depend on yourself; actually, most times we probably depend on others. Collaboration is something positive but we can have some delays that hardens this process.

During this process, the Playwright team was pretty fast giving feedback; it's not real-time but it was quite good.

However, sometimes we have other priorities going on, some process/legal flows we have to follow, etc. And all of this can delay the whole contribution. Delays can cause, besides a certain frustration, also additional lack of efficency due to context switches.

You start with a problem or a goal and you end up finding another ones

Sometimes you start by fixing some bug that you are well aware of, or implementing the feature that you imagined (e.g., such as the one I highlighted here). But the process is never a straight road where you go blind towards your destiny. During your trip you'll notice other things (perhaps by serendipity - "lucky findings"), and that's great because it means that you're not just coding/fixing something; you're learning through the way! I've found a minor encoding problem in character data (i.e. "CDATA") in the JUnit XML report, which we discussed, and I've fixed. Meanwhile, I found another issue that I've yet to report. All this is normal, because we as humans and smart people learn... and as we learn we seek to improve.

In sum

Contributing to a major project can be overwhelming, especially whenever we have limited context about it, its tecnologies, or even the team. But we can always learn; as testers, or as people with testing skills in their heart and soul, we're already used to it.

In the end, this was both a learning exercise and also somethign that hopefuly will be used by many of you out there.

It feels great to use open-source software; it feels greater whenever we can contribute to it!

Feel free to reach out in case you have further suggestions :)

Note: in case you want to see a full tutorial showcasing Playwright integrated with Jira and Xray, please have a look at this tutorial.

Why do we automate?

I've seen many teams come up with test automation due to these reasons:

• someone told us to
• we've seen other doing it
• we have the feeling that it will help us

Interestingly enough, all of these are not based on real needs identified by the team that serve a clear purpose.

Why should we automate?

Automation comes as a consequence of a broader strategy. The same applies to test automation, a smaller set of automation (I would recommend doing some search on test automation strategy). These are some reasons why, IME, we should automate, in terms of test automation:

• Have fast feedback loops: know ASAP if a change/fix we're adding into the product, is breaking some existing functionality or if is impacting on some relevant KPIs, such as performance;
• Have time to explore: even though I came to the conclusion that test automation and exploratory testing are not exclusive, if we have a bunch of automated checks and automated KPIs in place, then we can have some "free time" to explore the system freely and uncover things we didn't expect.

These are, IMO, the two main reasons. But there are more:

• Assist on finding the root cause of problems, if the tests are focused enough and have ways to get the context;
• Augment testing with tooling, by providing us insights that otherwise would be hard or impossible to get; performance/load testing are a good example of that.

If you're thinking about costs, or if you have been asked about, of course there are costs related to implementing proper automation. But there are even greater costs whenever you neglect it. Long story short: these are good costs for sure, as they will enable teams to become more aware and more agile with time.

The goal is something else

In my experience, automation is never the goal. Writing lines of code is never the goal.

What concerns test automation, we have to understand the needs that we aim to overcome. There are different types of testing, at different levels and layers, that we can implement. Test automation scripts are not the responsibility of a single person, unless you're the only one in that project. Test automation is code and thus should be close to the heart of developers; it also follows the same rules. Developers need to implement some checks/validations and track some quality metrics from the start, including performance, security and accessibility. If we have people dedicated just to test automation, it would be great if they can complement the already existing testing that is in place, by leveraging more complex or risky testing scenarios/environments for example.

Whenever we think about test automation, and if all that comes to "We have to implement some selenium tests to check is "everything" is ok" or "Automated tests? That's to the test automation team!", then it's a sign we have to rethink it all.

In terms of the why, a conclusion is never the why, I think. The team needs to discuss where they stand, what are their current blockers and where they want to go. How can we make the team be more efficient? How can we depict a problem as fast as we can? Can we have additional sources of information that can assist us? In my perspective, these are some examples of the questions we should ask ourselves. Then, of course, we can come to the "hows". In terms of the "who", testing and quality are responsibility of everyone, where some can contribute in different depth and with different perspectives. Testers can always contribute with their expertise on exposing unknowns and risks that may matter; that's one of their main values. It's not about the test automation scripts.

The goal is to make the team more aware about how their current system/product behaves and how changes we're crafting impact those, so the team can act quickly in case something is getting broken or something is getting negatively impacted (e.g., performance, code quality).

The goal is, and probably most of all, for the team to be more confident.

Only confident teams take risks and try out new things and learn from that, ultimately improving the product. What about your team? Is test automation a goal? Or a consequence of something broader, that involves the whole team and making that team more confident and more agile?

The speed of change is increasing, the demand for change is also increasing. Knowing that quality is multidimensional, and that time is also part of those dimensions, we need to handle quality matters wisely, by being prepared as much as we can. The decisions we take today, or that we don't take, will affect quality. Teams continuously face challenges; some may be more aware of this than others.

The performance challenge

Let's look at a real life example, based on feedback collected from some teams I've worked with. As mentioned in a previous article, some time ago, we did an assessment with several teams. One of our goals was to understand how these teams were approaching testing.

From all the conversations we had, we could realize several things (I'll just enumerate a few):

• having a product "without bugs" was a concern
• performance was also a concern, with different levels of importance, in some teams

Let's focus on the performance concern as I think it can be used as a reference.

Depending on the nature of the product, how it is deployed and maintained, how it used, and how often, performance as a quality criterium can become more or less relevant.

The "we have already dealt with it" team

One of the teams had struglled with performance many times in the past. Once in a while, a performance issue would popup. There were several reasons for it (internal and external), and some of them were actually kinda of good ones - usage in more high-demanding scenarios, with thousands of concurrent users and with unthought usage scenarios. I remember the team addressed each performance issue, one by one like in a firefighting mode. Performance wasn't being tracked consistently; there were some ad hoc prrocess to measure some aspects of performance, but a dedicated performance environment, where performance could be tracked continuously only came later on.

Whenever the team decided to tackle performance, not just by reworking some core aspects of the product but also by implementing a whole performance approach with a dedicated environment, gains started to arise and performance issues became very rare. Confidence increased a lot but we know that pursuing quality is a neverending story.

The "data" team

This very small team focused on processing data was currently in firefighting mode. A dedicated performance environment is missing and so are continuous measurements on performance. The team is overloaded trying to address performance issues as these keep coming, as the product is being used in numerous and uncontrolled ways. This in fact needs some thoughts: of course we need to measure performance upon changes in the product but we also need to have come clarifications about product usage. A clear commitment (spec if you want), about the minimals the product shall handle was missing. And more than that: there were no ways of controlling/limiting the source data which could ultimately affect that specific product usage but also the overall product related infrastructure.

"Not worried at all" kind of teams

Several teams were in this category. They hadn't faced any significant performance challenges so far, therefore they were not worried; performance was not a concern on their daily activities.

Meet the "Quality Mountains" model

Even though what contributes to quality (i.e. quality criteria) changes with time, teams will face some common challenges. Sometimes it may be performance, other times it's security, other times it's something else. We need to be prepared to handle those challenges by not just shifting left, but also by embedding that as a concern in our day to day activities.

Performance, for example, is an issue that sooner or later will come to almost every single team. The later it comes, the harder will be eventually to address it as it may require rebuilding the architecture of the application, for example. Don't expect easy fixes, like changing some flags in the compiler or in the JVM; those are possible but most times they are just workarounds, which don't solve performance problems that are due to the way the solution was built.

I realized that similar challenges will come to teams at different times, with different intensity and different relevance. Not all teams will have the same kind of challenges but we can see similar challenge patterns across teams, such as performance, security, accessibility, among other.

Therefore, I came with the "Quality Mountains" model, which I tried to sketch in a very rough way :-)

Each mountain represents a quality challenge. There are mountains of different types (here annotated with distinct letters), representing different quality aspects you have to address. No mountain is the same, even whenever they are of the same type. Mountains exist ahead of us, even if we don't see them coming. Sometimes we may already be climbing one these mountains and drowned dealing with all the technical and non-technical aspects we have to overcome.

The sooner we prepare to climb them, the smooth our trip ahead of us will be. Sometimes we may use some telescope, or send some probes, to see what's coming ahead (up to a point).

Our overall software development & maintenance trip is not flat; flatiness is an illusion.

Wrapping up

For some teams, a given challenge, such as performance, may be more or less relevant in the context of their product and its usage scenarios. That's fine. However, we need to realize that we're in a road, at full speed ahead and challenges will come, no matter what.

So, we cannot postpone handling quality because quality challenges, no matter if we're talking about performance, security, or other, they will come at fast speed. Are we prepared to handle them somehow? Can we do something about it? Can we start perhaps embedding performance/security/and other quality related practices to diminish risk, its probability and its impact?

What about taking some time to reflect within the team about those risks we're ignoring or not properly handling right now? Then define some countermeasures, such as deepen knowledge in certain areas, so our (individual and team) skills are improved and we can tackle the problem with thoughful actions.

Remember that challenges ("quality mountains") will come, even if you don't see them ahead. Are you and your team already preparing for them?

During my talk, I covered the richness of "quality", all the possible meanings we can have related to it. But also, what "value" means and where does it come from.

I came up with a new model: ATOMYQ: Another Test Oriented Model with Yummy Quality :-)

But why do we need another model? And what does it mean? Well, first models are ways to abstract some part of reality; they are ways for us to explain ideas and start conversations. They don't need to be accurate but they are nevertheless quite useful, because they can highlight concepts and how they relate to one another.

Meet ATOMYQ

ATOMYQ is a model that centers testing around 3 main players: Product, Team and (each) Person.

Team

All starts with the team. The team as a whole has a set of diverse skills that require learning and practice. Skills are used to craft our product, in all its dimensions, and all its assets. Unfortunately for us there are constraints. Constrains that will influence our skills, their intensity, their extensity, their reach. Testing also makes use of skills, many skills in fact (technical and "soft" skills). All of those are engineering skills. Testing is also applied into our other skills whenever we exercise them, i.e., we are only excellent coders if we care performance while coding algorithms, for example. All of this is the reason why team skills are so important, and why we should care about improving and fostering a knowledge culture. Investing in the team, is investing on the product and its value in broad sense.

Testing uses different sources of information to uncover bugs and opportunities in our product. Well, let me clarify the previous sentence. First, we use data... and some of that data is actually valuable information; a written specification is just a source of data, not a scientific, closed fact. Second, testing doesn't exist in the void... or alone. Testing is performed by humans that can make use of skills, tools, automation and all possible means to assist them.

Ideally, testing should be performed by the team, in everything that the team does or doesn't. But testing, as a brain driven activity, ultimately runs inside each person. Each one of us will have different insights, different backgrounds that are essential to explore our product and its idealization.

Product

Many times we think we pack value into the binares we build or the code we write. Well, tt's just not about the code we build... it's everything related to product, including documentation, all enabling kind of materials, and more.

What we build, how we build and what we decide not to build are the foundations of product attributes. Our product as a whole has a bunch of emergent properties and attributes, some of them are called quality attributes in the sense that they contribute somehow to a given view of quality.

Nowadays I see the release frequency, i.e. how frequently we release updates on our product, also as an attribute of the product. The quality adjective that can perhaps frame it is: freshness.

On the ATOMYQ diagram we can depict bugs and opportunies as living in the product; I struggle to position them to be honest, but that will require some additional thoughts.

Person

Each person, as a unique individual, weights differently product attributes to come up with value.

This not only changes with whoever is involved but also depends on the context, and on the time. Example: if a person was robbed/scammed up recently, maybe security now is one of the attributes that matter most for that person, at least for the next couple of days/weeks!

Quality also informs testing, and thus influences it, as we'll pursue testing focused around the quality attributes that matter most.

Other things worth mentioning

Our product assets, which are diverse, come from the team as a whole, and not from a single individual. It's the skills within the team that shape our product. Testing is mostly done by the team members but not only; other people, including our users, also test our products.

Testing is not used just to find bugs; I like to think on testing from a positive perspective and think more about opportunities for increasing the value of our products.

As a team sport, testing also shapes how we think and thus improves our skills because as we become aware of potential problems, it changes what we do and how we craft our product assets in the future. So yes, if we think on testing as something happening at team level, connected to the team skills, then yes, testing improves quality on mid/long term. This may be controversial, I understand, but it's my experience has someone that worked having different roles within product development teams, or whenever working with some external ones.

If we think testing as something done by a given person having the "tester" role, then we can see more testing as a bug reporting kinda of activity. Finding bugs is indeed important but we need to move from there and actually improve quality. If testers become more and more quality coaches, they start influencing the team and we'll see improvement on the skills within the team, ultimately leading to the overall improvement of quality. Still testing itself is crucial, but as a whole team concern. Well, that's how I see it right now.

In sum

ATOMYQ presents the interactions about different players in organizations that are helping others achieve some goals, through the value provided by their products.

Quality is multidimensional, time dependent and view dependent. We don't pack quality. We work with quality in mind, so what we do, shaped by the skills that we have and that the team has, will lead to better product assets.

The goal is working torwards increased team confidence, aligned with better team understanding of what value means and what blocks us from delivering/caring for/improving it.

Testing for sure help us as it is at the center of ATOMYQ, close to opportunities and bugs we cannot ignore. However, testing is not atomic. Instead, it's something we do, more or less explicitly, continuosly because ultimately we use it to improve our knowledge and the art and engineering that is building products, that people use, love, and even pay for.

Skills are essential; testing itself is a broad skill that can help us not only become more informed but also look at things differently, wisely, with value and risk in mind. Looking not just about where we stand, but where we want to go and what we can do to get us there.

Happy holidays/merry christmas, with quality time. And I leave "quality time" definition to yourself, as I'm sure that noone better than you will know the best answer that fits your context :)

I started with this question: "Improving quality is a doomed project... or is there a way out?".

During the talk I enumerated some real exercises tthat we've done to overcome this challenge and that you can perhaps try out in your team. In this article I won't go into the details about each one of these exercises; maybe I'll leave that to another post. However, I want to share a bit some thoughts on how to engage, on how we can split that into 3 steps discussed ahead.

Engaging

Making a talk around the "engage" theme made me reflect for a while. Well, that is always a good thing :)

I looked back and thought a bit about some activites I got involved in one team.

First, what do we mean by engage? We may think on it as a way of reaching out someone in order to make something happen. However, this may be quite a restricted vision of it. Of course engaging has a goal but we want that to become a shared goal, something compelling, that touches our heart and soul. That is like the fuel to ignite something bigger. There are many different ways of engaging and pursuing that goal, that we'll see ahead.

During my talk, I tried to make a connection between a well-known good pattern for writing test automation code "3 A's" (Arrange, Act, Assert) and one that we could try applying in our teams: Acknowledge, Act, Assess.

Acklowledge

Acknowledge is about understanding where we stand, what are our blockers for achieving higher quality in the products we're making, deploying, and supporting.

How are we turning ideas into value and is that value being cared for on production? The best way is to engage to initiate a discussion and a brainstorming about the whole software lifecycle.

Who is involved, when, how, and how deep? What is the flow of information and processes? Where is the knowledge (tricky question)? What are the artifacts we're making and for what purpose? Who is getting blocked, how and when? Drawing our overall process and pipeline can be quite useful and insightful; doing it together brings additional clarification and visibility about some main blockers right away.

Visualizing some aspects of how we are working and having the time to reflect is key. Doing that as a team is essential. This exercise was detailed on a previous post.

In the end, we have a set of actions, hopefuly prioritized.

Act

Act is about making sure those actions get done. How we get them done is actually something interesting. We can act by doing, but we can act by exemplifying, act by spreading the love about a certain topic (e.g., unit testing), act by increasing knowledge or bring awareness about something.

But one way of acting, that for me is one of the most interesting ones, is act by engaging to give opportunities so people can shine. We had a developer within the team that wanted to make a shift to test automation; we knew it, so this colleague was given the opportunity to address one key concern we had related to performance. This person, with the help of the team, was help to get an environment up tracking performance on a continuous base, helping out the whole team.

Assess

Every thing we do needs continuously to be improved. Sofware and quality are not static. Teams are not static. The world isn't static. Therefore we need to always keep reflecting about what we're doing and think on ways o improve it further. Sometimes, we may even have to give a step back and discard some technique, process, tool, artifact, whatever.

In this team we worked, from continuous discussions we had it was clear that we need to improve collaboration and shared understanding. Therefore, we tried out ensemble programming (also known as "mob" programming). And that was a win for us. But we also learned we needed to fine tune it and that's what we did and keep doing. We have to adapt practices, because people are not APIs; we are humans.

So can we improve quality?

In my experience, quality can always be improved. There's no specific technique, process or tool to achieve it. Problems usually are related to lack of knowledge, lack of true collaboration, and lack of proper tooling.

We need to start by acknowleding where we stand, to identify the items we need to address that can benefit the team as a while, and prioritze them.

We then need to actually do something about those; we need to act, act by doing and act by exemplifying, and also act by empowering others. Finally but not least, we need to always evaluate our progress, because we need to be able to adapt and correct our route as soon as possible.

Improving quality is not a straight road. It's a route, where may need to turn at different places, at different moments. Sometimes we may event have to stop or take a step back.

We need to seed knowledge and care it by collaborating within the team. Because knowledge grows better when you care for it; whenever the team cares for it. And as with any seed, if you can depict, use or even build tools that can help you along the way to be more productive.. the better!

You may be planting knowledge. But your crop will be... quality improvement!

Questions

During my presentation, several questions came. I think they are quite interesting and I wanted to share with you my perspective that may help others out there, or even make you reflect a bit.

You shared some experiences and exercises that people can try in order to improve how they work. What was the one that had the most impact? Did any of them surprise you?

All the exercises made with and within the team had impacts. Advocating continuously about unit testing, leveraged by real example-based workshops made quite a difference on improving the number of unit tests and related coverage, and the confidence. However, changing the way people work/collaborate by adopting ensemble programming sessions increased not only confidence but also shared understanding (i.e., knowledge). Ensemble sessions were a surprise because they represented quite a change from the way we were working before, mostly based on solo programing + pull requests + testing a bit aside. I thought it would be quite hard but it wasn't and people got satisfied. As mentioned during the talk, we use a mix: ensemble programming, pair programming (+ a tester), and solo programming. Most of the work gets done through these ensemble sessions though.

What do you mean by "untrustworthy things to test?" Could you give an example?

In the testing debt quadrants, one of the quadrants is aimed for those things that we cannot really trust. And why wouldn't we trust? Because we may lack information Imagine a product that integrates somehow with another product, that you usually don't test. That integration can be leveraged on a API but it can also make use of some shared CSS for example. You may also not understand fully the use cases of the external product, so certain parts of how that integration becomes used is unknown. Therefore, thinking on a more end-to-end user journey, bits of your product may be used in ways that you didn't foresee. Other examples could include the fact that you don't have a representative test data, or maybe your data is not big enough.

How could we improve QA/Dev relationship?

What would you recommend/advise to testers on how they practice effective collaboration / engagement with the dev team when dev team is actually in the middle of building system?

That's an excellent and recurring question but we can generalize it for any other relationship within the team. Between devs<>testers, first we need to talk a bit about our blockers... as a whole, and for each one of us. Doing some exercises together is quite helpful because it makes people talk about a certain topic and start having a shared understanding on the concerns and challenges. At that time, people we'll start offering to address some of these challenges, no matter the role they have. We hear a lot about "quality is a shared responsibility"; that's true but getting to the point where people understand it requires working on it daily, on every aspect that we do. Therefore it's important to start having a more true ownership about what we build and how we build, deploy, and operate it. We should avoid having conversations where a certain quality aspect is assigned to a specific person. We want devs to excel in what they do and how they do, removing them from the burden of dealing with major issues; testers can help making that happen. Whenever a dev is working on a story (well, it's only that dev but I'm simplifying it), that code gets tested later on and then is sent back a few times... well, we can take that moment to say "well, instead of sending this back all over again, let me try help out next time and as soon as you start working on the story, ping me and I'll join you to help clarify things from the start". The whole idea is to make the tester not the "bad news messenger" but instead "the window opener", that will show what behind it and agree about the way to move forward on the world behind that window.

How do you convince devs to get testers into such pair programming sessions? I often hear things like "testers slow us down during pair sessions...I don't have time to explain stuff to the tester."

Testers are invited to these sessions. Most times they are present, some times they may not be. Testers engagement also depends on what is being done: if it's a more coding related aspect, maybe the tester does something else. The idea is to have a kind of open format, where testers can join... and if they are not there, devs can easily ping them to ask for their insights. Having a open stage is a good thing. To get testers into these sessions in our case was not that hard. Some time ago we had concluded that we needed to improve dev<>testers collaboration, even while the team was working mainly using PRs (pull requests). Back then, we worked with devs to make them more open to invite testers to help out giving feedback early. This took a bit of time to get it rolling but after some time devs appreciated becaused they saw less stories being sent back for fixing. Moving from that point where some collaboration was already going on, having testers in the ensemble sessions was natural; however, as mentioned earlier, they don't need to stick all the time.

How hard was it to form the inital groups of testers and developers? Did all go smooth or were there some starting probems (and if so how did you solve them)?

It went smooth AFAIK. These groups are not fixed. We tried to have some balance within the groups, so we don't have all experts in one group and all juniors in a separate one.

One of the more subjective questions raised was: What does quality mean to you and your team?

We wanted to understand the team's perspective on quality and cross-relate it with the practices followed within the team and the overall status of the underlying project.

The answers to: What does quality mean to you and your team?

In order to understand the answers, we need to provide a bit more context about each team and the software product being handled by them.

Team 1

This product can be described as providing a very basic UI for processing an input and producing a well-defined output; it's a small team working in a product that exists for quite a while.

Security on the usage of the software; working without problems; always works in the same way and produces the same output

The team has been addressing some security issues that have been found (internally and through crowd-sourced security resources). The perspective on quality is a match with the ongoing security concerns and also with correctness.

Team 2

This is not actually a product; it's a more tactical team (with shared resources) that helps with small projects that can serve multiple teams or that can start some module from scratch, later on to be incorporated in another product.

Conformity level between initial requirements and what the team delivered

For this team, it makes sense to provide such an answer as the team works with specifications, addresses these in a short time frame, does the handover, and moves on.

Team 3

This is a medium team working on a product that provides a web UI. The product is SaaS based and has many concurrent users. The team embraces some DevOps practices, handles support, deployment and operations.

Absence of bugs (mostly bugs from support feeds) and to performance issues; (we're) mostly focused on customer feedback from support; How the team is performing/working

The feedback about quality is essentially around feedback coming from support, which may result in bug reports on feature requests. Besides, as it is a SaaS product serving large thousands of customers in multiple locations, performance is a concern, especially because the team is responsible for deploying to production and maintaining it. This kind of end-to-end perspective is interesting: on the left side, listening to customers... and on the right side, caring for production, using monitoring and observability (to an extent).

Team 4

This small, young team works on an innovative product. It's a product where user experience and UI are important, that runs on local machines, supporting multiple OSes (operating systems), and receives frequent updates.

Easy to use (without requiring documentation) and the app is working without issues (no crashes or bugs)

This perspective on quality matches the product context as it has to be easy to use and it shall "behave fine" in user machines, without crashes. Knowing that users will use different versions of OSes, with all sorts of configurations, is challenging for features and stability.

Team 5

This medium-sized team works on a product that exists for several years. It provides a basic UI but the core features are related to data processing, whose size and conditions are unknown upfront. The product is deployed in servers, having different configuration/infrastructure scenarios. Feature releases are done every month, while bugfix releases are pushed more frequently. The team handles support issues often.

Be able to make a release without bugs and without performance issues

The feedback on quality meaning aligns with the core concerns of the product. Besides correctness, performance is one of the main concerns due to data aspects (e.g. size, format) and also to the fact that data handling/processing is done in customer infrastructure, that may not be properly sized.

Team 6

This is a medium/large team working in a product that provides a UI and that is installed locally. It has to support diverse customer scenarios. It is used by millions of users and also in numerous ways. The product has a long tail of feature requests and bug reports.

Keep up with customer expectations, concerning features and bugs

This answer has a reasoning behind it: the high customer interaction with the team through support and other channels. Also, there's a public roadmap that customers track and try to push.

Team 7

This team works in a product that is actually a component used by other products/teams. It has no UI; it received some data, processes it, and then generates an output. It's a very small team.

Generate output without errors and in a fast way

The concern of this team is around delivering always the same output, and do it as fast as possible. Thus, correctness and performance arise as the main concerns.

In sum

We can see that the actual meaning of quality is distinct from team to team.

This is due to several factors:

• team context (e.g. maturity, skills, collaboration)
• product context (e.g. what is important for continuous product success?
• current context (i.e. what's important right now? What are we currently struggling with?)

Answers may be a bit vague (e.g. "without bugs/errors", "keep up with customer expectations") in the sense that they hide multiple quality criteria, that we would need to decompose further to have more clear insights. Some of these become a bit more clear when we uncover the testing being done, or that the team wished to have.

The quest for uncovering quality proceeds :)

Maybe it’s an emotion or it perhaps a comparison?

Is there a simple way to define it?

Before answering this question, let me start by sharing a story.

A real story about someone that got fired and didn't care that much (there was a reason for it!)

I always remember this story from 2007, when I was working in East Timor.

I was delivering a product for one of the biggest companies in East Timor. In fact, I was doing the full hardware and software installation, testing, talking with customer, well... a bit of all. At that time, we had to be able to solve as much as possible and deal with all the possible and imaginary constraints :) I loved East Timor and I wish I may come back there once again, as an explorer!

But the story is about this person that got fired because he had done a mistake. Actually, the boss wastrying to discuss with him about what happened and see his reaction to decide how to proceed.

That person wasn't concerned at all about losing the job. East Timor was a poor country and there weren't much job offers available. But that folk wasn't worried at all about getting unemployed. That for me was totally strange. "How can he survive without a job?" - I asked myself.

And then someone explained me the local context. Many people didn't actually need money. Why? Because they exchange things between them, like food supplies. Electricity and water were not a real concern back then. Thus, the value I assign to money was totally different from the value that person assigned to money, which was close to 0. And that changes your whole perspective about things, about life and the things you do with it. So, finding what matters will influence everything you'll do.

We may use this story reflect a bit about quality, value and testing. I've previously written about the relation between them using The Quality Ice Cream Truck model.

What is quality?

Quality is value to some person -- Jerry Weinberg

Quality is value to some person (quote from Jerry Weinberg, who has also a nice post on this topic). Well, in other words it means that quality is subjective. It depends on the eyes of the beholder. A couple of weeks ago I was having a conversation with my wife about values in life and she was counterarguing with me "well, those are not values... those are feelings".

That made me think for a while. I do think values and feelings are different things but they do have a subjective root.

Thinking on quality, we were saying that quality is value to some person or stakeholder... and indeed we’re talking about feelings, what makes those stakeholders satisfied or happy. Even if it's happy enough!

Quality, defined as simple as possible

So, putting it in simple words, quality is about making people happy, no matter if we’re talking about the internal product team, business, organization, or the end users.

Starting with end users, they become happy if:

• It addresses their needs
• It addresses their expectactions

And what do we exactly mean by that? What are those expectactions? Does the system perform what it says? Maybe there are performance expectactions and other we should care about?

• End users also become happy if they have seamless and simplified experiences

The team becomes happy if:

• Changes can be made safely (e.g.through good "testability")
• Changes can be easily made ("maintainability")
• One can understand the architecture
• Software can be operated easily (i.e. "operability")
• One can understand what is happening (e.g. "observability", "monitoribility")
• It provides the means for RCA (Root Cause Analysis)

Organization becomes happy with:

• Low costs with support and maintenance
• Low headaches with customers
• Low exposure in social media due to problems
• Ability to innovate fast
• Ability to overcome problems fast

All these people will approach quality from different angles, which change with time.

Quality and happinness are multidimensional

Quality is also multidimensional.

Using a balance as an analogy, each one of us, uses different weights for different quality criteria... therefore, the overall quality as measured by that balance, will be different for every single person. The variables, along with their weights, that make a certain person happy are different from person to person.

Besides, quality is not static and may easily get outdated.

Imagine an Instagram profile with few posts; people would say it has "low quality" and its interest decreases with time.

That’s why we need to provide value consistently, or else our product will get outdated & forgotten. That implies that we deliver value and also that we understand if we’re actually delivering value. In other words, we need to understand as much as possible about our users and how they are, or not, using the product.

We need to find ways of measuring value. Hmmm that could be a whole new topic but we can try having a simple approach instead of a perfect, non-feasible one.

• If a page/feature becomes unused then its value is questionable.
• If our users use our product while it’s free but we cannot convert them, well maybe we need to test our value proposition.

Wrapping it up

This simplified definition for quality - what makes people happy - can be a nice way of framing quality and explaining it. What makes each of us happy is different, in terms of the factors that contribute to it and their weights.

Besides, it changes with time. At a given moment we can be amazed by some website performance but later in time that can have less importance, when compared against the website content itself for example.

And our happinness also depends on getting value consistently (more on that in an upcoming post), or else we lose interest. Whenever talking about quality in (software/hardware) testing, we're in fact also looking at these aspects. Is this a rough/simplified definition for quality that works for you? What's yours?

Before Playwright, Selenium was the main browser automation library and SeleniumLibrary (i.e. robotframework-seleniumlibrary) was the Python package you would install for assisting on web-based testing using Robot Framework along with some specific keywords.

Meanwhile, Playwright appeared. It provides a completely different approach for automating the browser, making use of asynchronous calls and having access to and capabilities of changing some browser internals. Just as quick overview, Playwright talks to Chromium-based browsers using the Chrome DevTools Protocol (CDP), allowing you to browse and interact with web elements and also emulate devices, location, handle network events, and much more.

Architecture

In order to use the Browser Library, we need to install it as usual using "pip" and then we have to "initialize it" with "rfbrowser init". The latter command will:

• install the latest version of the playwright NPM package in our Node.js environment
• install local browsers, for chromium, webkit, and firefox browser types

Then we are ready to use Browser library in our RF tests.

The following diagram gives an overview of the underlying architecture and the flows involved when we automate our browser(s) using this library on top of Robot Framework.

This architecture may seem a bit cumbersome but there's a reason behind it. When Browser library project was started, there wasn't any Python library available for Playwright. The team decided to make a gRPC wrapper that would interact with a Node.js process where the official JavaScript "playwright" package would be running.

All of these components can run in the same host; eventually, the browser can be on a separate host. However, the Node.js process must run alongside with RF.

Test cases are written as usual in Robot Framework (RF) but now making use of keywords provided by the Browser library. RF usually runs in Python and Browser library can be extended using JavaScript.

Browser library, when initialized, launches a Node.js process implementing a gRPC server that will translate incoming requests to the Node.js Playwright library. Whenever using keywords of the Browser library, gRPC calls are made to the Node.js server. Messages are exchanged using Protobuf (i.e. Protocol Buffers) messages, on top of gRPC with a specific protocol/messages, which in turn works on top of HTTP/2. The Node.js gRPC server will talk directly to the browsers, using the Playwright API and specific browser protocols, such as CDP (to be discussed on an upcoming blog post).

We could analyze what's going on in a simple RF test scenario that tries to obtain the page title. For that, we'll use Wireshark and install the protobuf message specification. We can see the calls between the Browser library, in our Python code, and the Node.js server that acts as a proxy to Playwright. The following screenshot shows the "Get Title" gRPC call, as a consequence of a "Get Title" keyword in RF, and the respective response.

Running tests in parallel

It's possible to use pabot to run Browser enabled tests in parallel. In this case, a browser instance will be created per each pabot process. Enhancements are being discussed to provide additional speed improvements and reuse browsers more efficiently. This may affect the previous architecture.

Comparing Selenium vs Browser libraries syntax

First, and to understand better what lies under the syntax, concepts are different, and Browser library can be said to provide finer control over the browser. Tatu Aalto, along with Mikko Korpela, giKerkko Pelttari, René Rohner, explained this during RoboCon 2021.

In SeleniumLibrary, all happens on top of a browser instance. We create a new browser using the "Open Browser" keyword, which in fact launches a new browser and thus takes some time and requires considerable resources. If we want to have a clean session, we close the current browser instance and have to crete a new one. On the other hand, in Browser library there are 3 layers: Browser, Context, Page.

Our browser instance is reused as much as possible, and it makes sense to create it during the init phase. Then we have "Contexts", which are configurable "incognito" like instances: they start in a clean state and are fast to create/delete. "Pages" are our typical browser tabs. If we want to have a clean session for our test, we can create a specific context for it. If we want to reuse a session between multiple tests, then we reuse the context and eventually stick its creation to the Suite Setup.

There's another interesting difference: in SeleniumLibrary we need to open and close browser and pages explicitly. Browser library handles some of this automatically: if a context or a page is created in the Test Setup/Suite Setup, then it will be automatically closed in the respective Test Teardown/Suite Teardown; the browser will be automatically closed at the end of test execution. There's a setting that allows us to finetune the behaviour of this whenever importing the library.

What concerns the syntax itself, at first sight you may not see a big difference if you have a layer of abstraction. In the following example, the .robot file with the test case is mostly the same. The only subtle change in this case is that "Close Browser" was replaced by "Close Page", even though we don't actually need to use it as mentioned earlier :)

*** Settings ***
Documentation     A test suite with a single test for valid login.
...
...               This test has a workflow that is created using keywords in
...               the imported resource file.
Resource          resource.robot

*** Test Cases ***
Valid Login
    [Tags]  ROB-11  UI
    Open Browser To Login Page
    Input Username    demo
    Input Password    mode
    Submit Credentials
    Welcome Page Should Be Open
    [Teardown]    Close Browser

*** Settings ***
Documentation     A test suite with a single test for valid login.
...
...               This test has a workflow that is created using keywords in
...               the imported resource file.
Resource          resource.robot

*** Test Cases ***
Valid Login
    [Tags]  ROB-11  UI
    Open Browser To Login Page
    Input Username    demo
    Input Password    mode
    Submit Credentials
    Welcome Page Should Be Open
    # [Teardown]    Close Page

The actual differences are hidden in the implementation details, in a file named "resources.robot".

Using SeleniumLibrary,

*** Keywords ***
Open Browser To Login Page
    Open Browser    ${LOGIN URL}    ${BROWSER}
    Maximize Browser Window
    Set Selenium Speed    ${DELAY}
    Login Page Should Be Open

Login Page Should Be Open
    Title Should Be    Login Page

Go To Login Page
    Go To    ${LOGIN URL}
    Login Page Should Be Open

Input Username
    [Arguments]    ${username}
    Input Text    username_field    ${username}

Input Password
    [Arguments]    ${password}
    Input Text    password_field    ${password}

Submit Credentials
    Click Button    login_button

Welcome Page Should Be Open
    Location Should Be    ${WELCOME URL}
    Title Should Be    Welcome Page

While on Browser library,

*** Keywords ***
Open Browser To Login Page
    New Page    ${LOGIN URL}
    Login Page Should Be Open

Login Page Should Be Open
    Get Title    ==    Login Page

Go To Login Page
    Go To    ${LOGIN URL}
    Login Page Should Be Open

Input Username
    [Arguments]    ${username}
    Type Text    id=username_field    ${username}

Input Password
    [Arguments]    ${password}
    Type Text    id=password_field    ${password}

Submit Credentials
    Click    id=login_button

Welcome Page Should Be Open
    Get Url    ==    ${WELCOME URL}
    Get Title    ==    Welcome Page

Keywords and semantics are slightly different in this case; we can say that it's easy to understand Browser library syntax and adopt it. My recommendation would be to have a look at the official documentation as it provides detailed information. Of course, Browser is different from SeleniumLibrary, and you'll find unique keywords on Browser that have no counterparts in SeleniumLibrary. One of those examples is "Promise To", one of the keywords that allows you to implement Promises (i.e. keywords that run on the background and that you can wait for them later on).

One difference that pops up, is that while SeleniumLibrary provides specific keywords for assertions (e.g. "Title Should Be"), while the Browser library approach is different: there are assertion operators (e.g. "==", "!=") that can be used for some keywords as long as they accept these as arguments (e.g. "Get Title == Welcome Page"). Meanwhile, the assertion engine has been split to a separate library: robotframework-assertion-engine.

Wrapping up

Robot Framework's Browser library is an interesting library that brings Playwright browser automation capabilities to RF users. This project uses a mix of technologies under the hood; that leads to additional requirements, including having a Node.js environment. I found out this mix interesting, as it involves different languages, frameworks, and protocols... and a workaround (using gRPC to communicate to a Node.js process) to access Playwright automation library.

The current architecture, as of v4.0.x of "robotframework-browser" package, could be eventually simplified in order to use the "playwright" Python package directly. As Kerkko Pelttari mentions, the current architecture allows to have greater control over the Playwright features available in the Browser library implementation. Depending on an upstream wrapper package for Python would add another dependency that could bring increased delays for releasing new versions; "wrapper libraries" have some lag from the main project (i.e. "playwright" NPM package in this case). Besides, changing the current architecture to use "playwright" package would also require significant refactoring. Anyway, we never know what the future brings, so it's always good to go over the release notes and the official documentation to be up to date. Playwright provides visible speed gains because it doesn't require starting new browser instances; we can simply create new contexts... and that is fast. In a simple scenario, I reduced the overall elapsed time by 50%.

If you can, I would recommend watching the Browser library talks that were part of RoboCon 2021, to learn more from the Browser library authors themselves :)

Learn more

• RF Browser library
• Playwright documentation and API
• Playwright repo
• RF SeleniumLibrary documentation
• Analyzing gRPC messages using Wireshark
• protbuf specification used between Browser library<=>Node.js server
• "browser" channel on RF Slack

Background

Before starting, it's important to understand the conference context. EuroSTAR 2021 theme is "Engage". Engagement is a broad topic. It's about involvement, commitment, challenging, seeking understanding. We deep dive problems and challenges, we embrace them and we overcome them, no matter if they live within us or in our teams. Engaging is, in my perspective, about giving a step forward.

Infrastructure and supporting assets

The review was done using an online tool where we can easily jump between submissions. The tool itself has an interesting and scary bug that may mislead you to think that your already reviewed submissions/scores are lost... however, I found out that it is just a UI glitch :) a forced browser refresh solved the issue.

To assist me as a reviewer, I had scoring instructions that I read before starting reviewing the submissions. These instructions were always open in one of my screens.

Criteria

There were 5 main criteria for me to score:

• engaging
• new ideas
• scope
• relevant to theme
• overall feeling

I felt a bit of doubts on "engaging" vs "relevant to theme", as the theme is "engaging" and thus some overlap exists. I used the scoring instructions several times, to decide how to score each one of them. Nevertheless, I would prefer a more clear separation between them as they're not independent variables. The "scope" criterion also left me some doubts, as we'll see ahead. Finally, there was an open text field "comments" where we could leave a qualitative assessment.

How I did it?

Day 1

I had a setup a quiet room (i.e. my office), so I could read and think carefully. I went over each one of the submissions and scored them; I skipped the "comments" section on purpose so I could not loose much time with them in the first iteration but also to try avoid a bit of bias. Whenever scoring, I regularly looked the scoring instructions if some criteria was not clear to me. I noticed that I was probably assigning "scope" not the meaning it should, as the scoring values were almost identical. Even so, I decided to look at it more carefully on a second iteration. Besides leaving the scores on the online tool, I also took notes in a spreadsheet; thus, I would have a backup :) and I could use it for analysis.

After scoring all submissions, I decided to go over the submissions once again to leave qualitative comments, that I structured in these bullets:

• First impressions: Mostly based on the title and takeaways.
• Takeaways & The Day After: Is it clear what to learn in the end of this session?
• Final assessment: My overall feeling, i.e. a brief summary.
• Would I attend: A yes or no, or eventually a perhaps.
• Cliffhanger/invite to discover: Was there any mystery left? Any invite to learn something intesting during the session?
• Suitable for: beginner, intermediate, advanced.

I was curious to understand if I had some bias that would be reflected more towards time. Thus, I traced a chart with the average scored and a trend line. The result was not shocking. I also made a column with the max score I add assigned in a specific criterion... and I didn't see any relevant change with time.

I slept over and I was curious to see what would happen whenever I iterated on this.

Day 2

I went over the submissions once again, but in reversed order. Why? To avoid some possible bias related with time and exhaustion.

In terms of criteria, I also did a subtle change. While in the first day I considered "scope" as the criterion for being adequate for the time slot, in the second day I considered more the message clarity because it would be a factor that would help the fit for the available time slot.

Even though overall scoring is a bit lower, the results are not very different from the 1st iteration, where the chart shows also a similar trend line.

In the spreadsheet, I calculated the absolute difference on the average and on the total score.

I flagged in red suspicious cases, that would affect my confidence:

• absolute average scoring difference >= 0.5
• absolute difference in individual scores > 1
• absolute total scoring difference > 2

Gladly, only a few reviews appear that I need to have a look at once again. After going over them (i.e. scoring) in this localized 3rd iteration, I seem to be in my confidence interval except for one submission that remains flagged. Why? Well, because in the total score it had 3 points of difference. I read that submission all over and scored it once again. In the end, it was once again in my confidence interval.

Remember the "Comments" section? Well, I decided to go over it and look at "would I attend" bullet and compare it with the "overall feeling" criterion, that implictly should have that reflected. I notice that my decision to attend was correlated with the "overall feeling" criterion but it was not a consequence of it.

Conclusions and final thoughts

On the process

I had quite a few doubts at start, especially this one: am I going to be fair? But also, I had doubts about how I would perform in this task, my biases, etc.

After looking at the results, my first iteration was trustworthy. I scored the submissions a bit lower during the second day, when I performed the 2nd, 3rd and 4th iterations. However, my first iteration was balanced and scores should reflect that.

I don't assign 5 or 0 easily. I need to be quite impressed or reject something completely, which for me was not the case.

Scoring talks can be relatively fast but if you want to implement a process such as this one or your own, you'll spend a lot more time. Even if you don't, if you consider all the points that can affect a single criterion then it will consume more time indeed.

This whole process of revieweing submissions is also a discovery process about how each one of us works and is able of handling tasks. What works for someone, may not work for someone else.

On submitting a talk

Talks touch us in subtle different ways, so we value them differently. It makes sense, from a quality perspective: we assign value accordingly with our own definition of quality.

There are some generic rules that apply though in most cases and questions that you can think of:

• Having a clear message, not that long not that short is crucial; if we get lost, we tend do discard it rightaway.
• Why should we attend? What can we expect from it?
• How will that talk help the attendee?
• How does that talk align with conference?
• Is it presenting something new, or your own learnings and challenges?

I've done a few submissions on the past and I have been unlucky most times. Yes, true. Therefore, I still have much to learn in this matter. I remember when I did a scientific submission for a Usenix conference few years ago, and even though it was a different audience, I had to iterate on the paper a couple of times before it was accepted and I was invited to present. Therefore, as everything, we need to iterate and learn.

Going over this review process, being on the other side of the fence, made even more clear some points mentioned above. By understanding the challenges a reviewer faces, when looking at dozens of submissions, I am a bit more aware of the things I need to have in mind for any submissions I may perform in the future.

It was an interesting thing to do and I've learned a bit more on the overall process of submitting (and reviewing) talks for testing conferences. I also appreciate the effort reviewers and program committees have, no matter what conference we're talking about, because this is a process that requires considerable effort if you want to make it right and fair.

Note: the 2021 EuroSTAR Conference is taking place in September this year instead of November and the programme is usually launched at the end of April.

In Part 1 I've introduced the idea of "Omni Testing" as a term that better expresses the concept of continually testing throughout the DevOps infinite loop of discovery and delivery. In this article I'll talk about shifting testing efforts versuys what Continuous Testing really means and how Omni Testing can be used to better describe how we look at testing nowadays.

We all heard about Shift-Left Testing and Shift-Right Testing in recent years. Probably you may have got the idea that it was the thing, that single idea that was going to massively improve your testing. Due to this misconception, for example, Janet Gregory prefers to think about testing as something always happening and holistic (i.e. considering all parts of the whole and their interconnection).

Let's be clear: you cannot shift something that is always there and happens all the time; however, what you’re focused on at a given moment in time can, and probably should, change. Within a team, testing happens continuously and in parallel, so different people may be performing quite different testing activities and from different angles.

If we have pure waterfall development in mind, testing happens as a phase after implementation. However, in real-life, testing is not constrained anymore. Of course, some teams perform more intensive testing before releasing; in that sense, people may refer to shift-left testing efforts as a way to clarify requirements altogether from the start and introduce more unit kind of tests. Similarly, with DevOps, you may "shift-right" testing efforts to production and use as a feedback loop to product development.

However, testing itself is not shifted-left nor right; testing efforts can shift though. Having "Agile Testing Quadrants" (by Lisa Crispin & Janet Gregory) and previous work from Brian Marick (i.e. testing matrix), we can see that there are different types of tests, with different goals. As Lisa and Janet point in their book Agile Testing – A Practical Guide for Testers and Agile Teams, risks are an inherent and important part of projects where testing is used as a mitigation strategy.

As you plan each epic, release, or iteration, work with your customer team to understand the business priorities and analyze risks. Use the quadrants to help identify all the different types of testing that will be needed and when they should be performed.

Thus, the type and level of testing performed, where and how it is performed, are dynamic.

Testing is not a specific task assignable to one person and delimited in time; everyone is involved in testing, it happens in different ways, at the same time and continuously; and this leads us to Continuous Testing.

From Continuous Testing to Omni Testing

Continuous Testing has been around for some time; the “Continuous” wording is not new. Unfortunately, it is being pushed by some tool vendors and some people to sell it as a way to continuously run “automated tests”.

Well, first there aren’t automated tests in a strict sense; you have automated checks but testing frameworks having been calling them as such from the start. Second, “continuous” in testing is not related to automated scripts/checks being continuously run or not. Even if it was, it wouldn’t make sense, as in CI “automated tests” are triggered on code commits, pull-requests, merges or due to some predefined schedule.

So what does Continuous Testing mean? I advise you to start by this post from Lisa Crispin and complement it with this course from Elisabeth Hocke.

We know that automation plays a key role in DevOps and also, though to a minor scale, on Continuous Integration (CI) practices which are closely related to Continuous Testing.

The CI concept was depicted by Grady Booch and largely widespread with Extreme Programming as one of the 12 core XP practices (Kent Beck, Ron Jeffries, Ward Cunningham, et al.

During Continuous Integration, "automated tests" (i.e. automated checks to be more precise: code that runs and performs checks to validate expected behavior) are run. These will help assure, to a point, that code that has been integrated isn’t breaking anything and previous expectations are still met.

As teams grow and code is committed more often, CI provides the means for immediate feedback so the team knows right away when a possible “bug has been added”. It can quickly be fixed before anyone builds on top of it. These automated scripts perform unit and integration tests but can also cover other types of tests such as security tests and more.

As builds are continuously being started by the CI process, these kinds of tests are also constantly being run.

However, “Continuous Testing” is not limited to these “tests”. Lisa Crispin and Janet Gregory put it like this instead:

Collaborative testing practices that occur continuously, from inception to delivery and beyond, supporting frequent delivery of value for our customers. Testing activities focus on building quality into the product, using fast feedback loops to validate our understanding. The practices strengthen and support the idea of whole team responsibility for quality. -- Lisa Crispin and Janet Gregory

In my opinion, this is the real reason behind the "Continuous" wording before "Testing".

Well, at first sight, “Continuous ” makes sense whenever everything else is using the “Continuous” prefix. But you don’t hear about “Continuous Coding”, do you?

Thus, a better adjective should be used instead: I think Omni (Testing) may describe Testing in a more complete way. Do you agree?

Continuously evolve our way of understanding the world around us

We moved from testing in waterfall, to testing in Agile, to testing in a DevOps context. Testing, similarly to other processes, evolved along the way; its essence hasn’t.

We still use testing for the same reasons, because we aim to understand what we test and use that information to make decisions.

Whenever we go to a shop and buy something, whenever we download an app and try it out, whenever we meet or go out with someone: we’re trying to somehow understand what’s in front of us. We’re information seekers because ultimately we want to take actions leveraged on data, i.e. informed decisions.

In software development, which may also include hardware development, by the way, testing is done as a means to ensure that the deliverables produced on our products/solutions iterations have a certain level of quality. And this includes evaluating not only expected behavior but also that “customer” needs are covered in the best way, thus we look at certain quality attributes.

Testing is centered around humans and it requires them. However, machines, tools, algorithms, and AI can leverage the unique power we have within us to provide better testing. That’s around 3.5 billion years of continuous evolution and accumulated learning about life on Earth.

I know what you may be thinking: maybe there is no Agile Life, no Continuous Life, and we also don’t call it Omni Life right? Well, true; similarly to what happened with life, testing evolved, it’s widespread, it lives from the outskirts of software development. But, life is restricted in many ways: to certain conditions, to our planet (as far as we know).

Testing, on the other hand, is a broader concept; no limits attached... and being Omni is part of its real essence.

Testing, as we foreseek, is something not limited to time, not limited to space, not limited to a specific team member, not limited to a given type, technique or approach; it is a process we use to understand something, to obtain more information about it so that we can take actions on it.

Testing is not just about continuously "running” (checks); testing is always there, it is part of the whole (e.g. product development), it encompasses different kinds of activities and involves everyone.

That’s why Omni Testing can, perhaps, be a better way of describing Testing as we aim it to be.

Testing is universal. It’s one thing every single human has performed in their life.

For many of us working in software or hardware product development, testing is even more "natural". However, we sometimes misinterpret it or see it just from a narrow perspective.

Lately, we have been hearing about Continuous Testing, Agile Testing, and DevTestOps/DevTestSecOps. Before that, Shift-Left and Shift-Right Testing and "Test Automation” were also around.

This tells us that Testing is:

• broad
• important
• always present
• intermingled with what we do
• used interchangeably, in different contexts

Because of all of this, I think that Testing is a process unbound, in time, location, people and context; yet, it always has a purpose – actionable information. No matter if you’re using it to verify if you retained knowledge after attending a course, to evaluate if you like some food or a perfume, or to understand how a product works and if it meets some customer needs. Even before software was invented, testing was already part of our lives and helping us shape our decisions.

Omni Testing

Why Omni? Is it a "new thing” or a "new kind” of testing?

What types of tests and approaches does it embrace? Who is involved in it?

"Omni” comes from the Latin word omnis and means "all,” "the whole of, "all things,” "everything.”

More and more, teams everywhere are aiming to have a holistic approach to testing. We’re aiming to have ongoing conversations, ongoing experiments and investigations, ongoing checks, ongoing information, and ongoing feedback. The whole team is involved in testing and has quality in mind, from the start. For that, the team starts by assessing all risks, performs all sorts of tests, using all sorts of techniques and approaches.

(Software) Testing is multi-faceted; it can be used in so many ways, although in Omni Testing we try to perform it in a way where:

• it produces relevant information ASAP (i.e. accordingly with the agreed test strategy and the overall agreed meaning of quality)
• it focuses on what matters first
• it is efficient and quickly adapts

Thus, it is not dependent/limited to:

• specific roles
• specific assets
• specific test levels, types, techniques and approaches
• specific methodologies
• specific phases
• specific tools

Defining Testing, again

Defining "testing” although may seem simple at first sight, it isn’t. Why? Because it is a broad concept and * sometimes used incorrectly or in a very restricted sense.

Recently, I had an opportunity of attending a workshop on exploratory testing with Nancy Kelln at TestBash 2019.

To the question "why do we perform testing?" the group came to these results.

• Spend time/resources
• Understanding the system
• Uncover gaps
• Provide valuable info to team members
• Give everyone feedback
• Finding risks
• Make the end user happy
• Translate information to others
• Ready to ship?

Probably, others would come to different conclusions. For some, testing is just about checking and "finding bugs". However, it’s much more than that.

I do agree with Lena Pejgan on DevOps that pictures such as the following one won’t help, as they depict "test” as a phase (which besides not being true is also not very agile).

There’s an in-depth explanation about Software Testing from Alan Richardson, where he goes over the definition from Cem Kaner (The Ongoing Revolution in Software Testing, 2007):

Testing is an empirical, technical investigation conducted to provide quality-related information about a software product to a stakeholder. -- Cem Kaner

What the previous definition subtly highlights is this investigative and exploratory nature of testing. This is why we cannot neglect this crucial aspect. Maaret Pyhäjärvi emphasizes Exploratory Testing, where we as curious, smart, and context-aware humans are driven and challenged by it, even when we do test automation, to provide valuable information that can drive decisions and fix problems ASAP.

Exploratory Testing is an approach to testing that centers the person doing testing by emphasizing intertwined test design and execution with continuous learning where next test is influenced by lessons on previous tests. -- Maaret Pyhäjärvi

In testing, as Alan Richardson correctly points, we build models and we compare them with reality. Anne-Marie Charrett has a short, clear video on Models & Software Testing which may be useful to you in this context.

When focusing on software testing, you’ll probably find some terms: "Traditional", Waterfall, Agile, Continuous, Exploratory and Modern Testing (you may want to look at Alan Page and Brent Jensen "Modern Testing Principles"; there’s a course on MoT).

Let’s look at this testing definition:

Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes: questioning, study, modeling, observation and inference, output checking, etc. -- James Bach & Michael Bolton

I would rephrase this slightly, adding my perspective.

Testing is a targeted process of challenging something (e.g. a product) and our understanding about it, using a set of activities leveraged by specific human skills.

Where,

• "targeted process" – a set of activities, having a common goal (i.e. quality attributes in mind), that will produce value-related information
• "challenging" – includes trial/experimentation/exploration, checking, ...
• "activities" – build models, create test ideas, execute actions in the SUT, investigation, ask questions, provide feedback, ...
• "human skills" – creativity, curiosity, perseverance, analytical thinking, communication, courage, continuous learning mindset, ...

Testing is exploratory by nature; we try things and test ideas, we explore to seek understanding, to obtain information as Dan Ashby states in his adapted model of James Bach’s Heuristic Test Strategy Model. We also use other approaches, such as script-based tests (i.e. checks) which sometimes are automated to verify "compliance” with predefined rules. We use all the outputs we collect as data that we process. This data becomes information, which is then used to improve our learning of the system/product and provide feedback to "stakeholders" (e.g. business, product team).

Whenever testing something we have in mind certain quality attributes (e.g. correctness, usability, scalability) – "requirements" we demand or expect from it.

To perform it, we test at different levels and use a bunch of different techniques that help us shape what we call "tests".

We test everything, software, hardware, specifications, established ideas, documents, and specifications.

Thus, testing is a complete, omni process that uses best-of-breed human skills to answer tailored questions to clarify our understanding about something and that can help shape it before, during and after it is built.

That being said, how does it relate with Shift-Left/Shift-Right and Continuous Testing? We’ll dive further into this topic on the next blog post of this series related to Omni Testing.

I made my code and my findings available as open-source in this GitHub repo. Feel free to checkout the code, fork it, run it, play with it, whatever :)

If you want, you can skip the details and jump to the conclusions.

Background

As part of Automation Week, Ministry of Testing challenged testers to address some problems related to test automation; these would include implementing automated checks/tests for the UI of a dummy booking platform.

The idea would be to share the problems, the approach, the learnings with the community. Selected reports would be able to present during Test.Bash('Online') 2020, and I was one the lucky ones to do so.

For me this was an opportunity to learn Python, even though I've made some limited open-source contributions in the past. It was also a way to explore Model-Based Testing (MBT) further. My goal was also to find use cases and limitations, pros and cons (?) of MBT.

Overview of the Web UI challenges

Next follows a description of the UI challenges for the sample website application Restful Brooker Platform kindly provided by Mark Winteringham / Richard Bradshaw. As you'll see, challenges are pretty straightforward. The last one requires a bit more care to implement though.

Challenge 1:

Create an automated test that completes the contact us form on the homepage, submits it, and asserts that the form was completed successfully.

Challenge 2:

Create an automated test that reads a message on the admin side of the site.

You’ll need to trigger a message in the first place, login as admin, open that specific message and validate its contents.

Challenge 3:

Create an automated test where a user successfully books a room from the homepage. You’ll have to click ‘Book this Room’, drag over dates you wish to book, complete the required information and submit the booking.

Approach for implementing automated tests

I decided to follow two different approaches, so I could compare one another:

• the first one using pytest, with "standard" automated checks made of sequential actions/expectations
• another using Model-Based Testing (MBT), using AltWalker which in turn uses GraphWalker

Both make use of the Page Objects Model (POM) facilitated by the pypom library. As pages can have different sections/regions, we can abstract those precisely as classes inherited from the Region class. This will make code cleaner and more readable.

class FrontPage(Page):
    """Interact with frontpage."""

    _admin_panel_locator = (By.LINK_TEXT, "Admin panel")

    class ContactForm(Region):

        _contact_form_name_locator = (By.ID, "name")
...

In both implementations, you'll see references to a faking data library. I've combined controlled randomization of data to provide greater coverage; this is especially valuable in the MBT implementation as the model can be exercised automatically "indefinitely" (to a certain point).

Standard tests using pytest

This implementation is what I would call the "traditional" (i.e. common way) of implementing automated tests, where you implement a set of sequential actions and one, or more, asserts/expectaction checks.

The following code is an example one such tests (you can find more in in the file standard_pom_tests.py).

def test_contact_form_successful(self):
    page = FrontPage(self.driver, BASE_URL)
    page.open()
    page.contact_form.wait_for_region_to_load()
    page.contact_form.fill_contact_data(name="sergio", email="sergio.freire@example.com", phone="+1234567890",
                                        subject="doubt", description="Can I book rooms up to 2 months ahead of time?")
    self.assertEqual(page.contact_form.contact_feedback_message,
                      f"Thanks for getting in touch sergio!\nWe'll get back to you about\ndoubt\nas soon as possible.")

In this case data was initially hard-coded. However, by using faker library we can create a custom test data provider for the contact and our test method can be rewritten as:

def test_contact_form_successful(self):
    page = FrontPage(self.driver, BASE_URL)
    page.open()
    page.contact_form.wait_for_region_to_load()
    name = fake.valid_name()
    email = fake.valid_email()
    phone = fake.valid_phone()
    subject = fake.valid_subject()
    description = fake.valid_description()
    page.contact_form.fill_contact_data(
        name=name, email=email, phone=phone, subject=subject, description=description)
    self.assertEqual(page.contact_form.contact_feedback_message,
                      f"Thanks for getting in touch {name}!\nWe'll get back to you about\n{subject}\nas soon as possible.")

Using a controlled custom test data generator can be quite helpful as we'll see ahead. It's also a way of handling test data, as long as it is not fully random (as you may want to reproduce things afterwards).

Model-based tests using AltWalker and GraphWalker

With MBT, usually we have the model (either made using a visual model editor or from the IDE) and the underlying code. In our case, models are stored in JSON format under the models directory. The test code associated with vertices and edges is implemented in the file test.py.

Using Model Editor (or GraphWalker Studio), we can model our application using a directed graph. In simple words, each vertex represents a state and each edge is a transition/action made in the application. Tests are made on the vertices/states.

Modeling is a challenge in itself and we can model the application and how we interact with it in different ways. Models are not exhaustive; they're a focused perspective on a certain behavior that we want to understand. MBT provides greater coverage and also a great way to visualize and discuss the application behavior/usage.

Addressing challenge 1 with MBT

For the first challenge (i.e. contact form submission), we start from an initial state, from where we just have one possible action/edge: load the frontpage. Then we can consider another state, where the frontpage is loaded and the contact form is available. Two additional states are possible: one for a successful contact and another for an unsuccessful contact submission. We can go to these states by either submitting valid or invalid contact data.

One curious thing comes out from the model: after a successful contact, we can only make a new contact if we load/refresh the frontpage again. Was this an expected behavior? Well, we would have to discuss with the team.

With GraphWalker Studio, we can run the model offline and see the paths (sequence of vertices and edges) performed.

The code for each vertex and edge is quite simple as seen ahead.

Example of e_submit_valid_contact_data code, showcasing usage of faker library:

def e_submit_valid_contact_data(self, data):
    page = FrontPage(self.driver, BASE_URL)
    page.contact_form.wait_for_region_to_load()

    name = fake.valid_name()
    email = fake.valid_email()
    phone = fake.valid_phone()
    subject = fake.valid_subject()
    description = fake.valid_description()

    data['global.last_contact_name'] = name
    data['global.last_contact_email'] = email
    data['global.last_contact_phone'] = phone
    data['global.last_contact_subject'] = subject
    data['global.last_contact_description'] = description

    page.contact_form.fill_contact_data(
        name=name, email=email, phone=phone, subject=subject, description=description)

Example of v_contact_successful code:

def v_contact_successful(self, data):
    page = FrontPage(self.driver, BASE_URL)
    name = data['last_contact_name']
    subject = data['last_contact_subject']
    self.assertEqual(page.contact_form.contact_feedback_message,
                      f"Thanks for getting in touch {name}!\nWe'll get back to you about\n{subject}\nas soon as possible.")

If we were using just this model, then the contact details could be temporarily stored as regular object variables (each model, in code side, is an object). However, that would limit us in case we want to have shared states between models, as we'll see ahead.

def e_submit_valid_contact_data(self, data):
    page = FrontPage(self.driver, BASE_URL)
    page.contact_form.wait_for_region_to_load()

    # variables could be saved in python side, in this object, but we'll need to share them between models which use a different class & object
    self.name = fake.valid_name()
    self.subject = fake.valid_subject()
    ...

One can make this model a bit more detailed and complex, by making explicit edges/transitions for the process of submitting one field as invalid. This makes the graph harder to read though and it will only be relevant if we want to distinguish those cases.

But we have to ask ourselves? Is it worth implementing this distinction? What are we trying to achieve/model exactly?

Addressing challenge 2 with MBT

In order to validate if the contact/message appears correctly in the admin page (2nd challenge), we start from a vertex/state related to a successful contact. Note that this vertex has a shared state with the first model shared earlier, which allows AltWalker/GraphWalker to jump from one model to the other one.

We can then go to the admin panel, authenticate if needed, go to the inbox/messages section, open and check the details of the last contact message. We can see several edges corresponding to actions that can be done, allowing us to transverse the graph and thus go to different application states.

Some edges have "actions" defined in the model, to set an internal variable that can be useful later on, in case we want to conditionate the paths somehow.

Example of action defined in e_admin_correct_login:

logged_in=true;

Some edges have "guards", so they're only performed if those guard conditions are true. In the following example, the edge is only executed if not yet logged in (which uses a variable set before in an action).

Example of guard defined in e_click_admin_panel (from v_contact_successful to v_admin_login):

logged_in!=true

In this exercise and on the previous one, we take advantage of using model variables (e.g. last_contact_name, last_contact_subject) to temporarily store information about the last contact so we can make the asserts on the vertex later on. The contact data details are filled in code side and are populated back to model variables used for this purpose. Passing data between edges/vertices code and the model can be done using an optional argument (e.g. "data") on the respective methods. Variables will be serialized/deserialized (use it wisely).

def e_submit_valid_contact_data(self, data):
        ...
        data['global.last_contact_name'] = name
        data['global.last_contact_email'] = email
        data['global.last_contact_phone'] = phone
        data['global.last_contact_subject'] = subject
        data['global.last_contact_description'] = description
        ...

On model side, variables can be local to each model or they can be global. Global variables are useful in case we need to access some information from another model. Having a ton of global variables can be hard to manage though.

Addressing challenge 3 with MBT

Challenge 3 (i.e. new booking) can also be addressed using a simple model, having a variable named total_nights, defined at model level, for controlling the intended number of nights to book.

The previous model depicts a sequential set of actions and corresponding states, so it mimics a typical sequential automated test as seen in the pytest implementation. Even though feasible, and as there's only one path in the graph, this model, as it is, doesn't provide exceptional value except that it turns visible our own model of the system.

Note: another possible model could deal with the fact that the contact and date selection don't need to happen in sequence, and also provide the ability to jump back to the initial page. Well, many variations can be done depending on what we want to verify and the risks we have in mind.

Conclusions

Traditional, sequential tests can be easy to implement and read (its code). However, they can become harder and harder to read (depending on code quality aspects). More, these tests are good for checking happy paths and doing negative testing. However, their focus coming from the hard-coded path that they exercise, is also their limitation.

With MBT I'm not focused on simple happy paths or negative tests. Instead, I'm trying to look at my feature from different angles, like "all" possible ways of interacting with it. Therefore, coverage with MBT increases because many more paths are exercised. By walking though those paths, we can expose hidden risks that otherwise we would miss.

Modeling is a process that in itself is also an uncovering mechanism for unknowns. During the exercise, I could see that certain actions could only be performed if I was at a given "state"... and that could make sense or not. For example, to make a new contact request, I had to refresh the front page. There, while modeling, as a process that we iterate over, we are also performing exploratory testing. Having models can be beneficial for discussion, even if they do not have automation code linked to them.

Surprisingly, the code related to MBT is simple and clean. At the start I had some doubts about it. I would say that one important challenge is related to how to pass information between models, in case we want to have multiple models interacting with one another, through shared states. By combining controlled test data generation with MBT, we can keep our model being exercised until an error arises or a certain amount of time has elapsed. This is a great way to spend time with automation code: we're not running the same tests, we're running many different tests/paths along with different data.

Please find ahead a sum up of my thoughts on traditional tests vs model-based tests. It's not an exhaustive comparison but it's a starting point for additional learning iterations.

Traditional, sequential automated tests/checks:

• are simple and focused
• are good for happy path and negative testing
• have restricted coverage, even if we use data-driven testing
• can become hard to visualize (we need to infer the "visual model" by looking at the code)

Model-Based Testing:

• Focused on a model, not on a concrete example
• Doesn’t replace "traditional" tests/checks
• Greater coverage, beyond happy/negative tests
• Visualization fosters discussion/reflecting
• Ability to reuse and combine models
• It’s a way of performing exploratory testing!

MBT: Challenges and Tips:

• Many ways to model: keep it simple!
• Write models, even if you don’t automate them
• Increase coverage further with test data randomization
• Temporary data management (model vs code)
• Harder to debug

Learn more

• GitHub repo with code for the two approaches
• AltWalker
• GraphWalker
• Visual model editor for AltWalker and GraphWalker
• AltWalker Model Visualizer for VSCode

A couple of weeks ago, I was lucky to be able to attend an online workshop from Black-Box Software Testing on State Model Based Testing. The workshop instructors (Alexandru Rotaru and Dorin Oltean) conceived it as being hands-on, which was great because makes learning more efficient.

In the first day we spent some time modeling a simple web application; in the second one, we implemented test automation code to support our model. Actually, we implemented the coded as a group using Ensemble Programming practice. We were lucky to have Maaret Pyhäjärvi also attending, which was great as introduced this concept to us and we all were able to see its benefits in practice.

What is State Model-Based Testing?

If we look at the following model, we can easily understand in few seconds that we're talking about requesting an Uber and there are certain actions we can do meanwhile, that may affect or not the state, whatever it means.

This is the power of visual models.

MBT (Model-Based Testing) is a testing technique that uses models as basis. A model is a "formal, simplified representation of a relationship, a process or a system" (quoting Alex Rotaru).

In other words, we can see that MBT:

• it's a formalization
• it's a way to describe something relevant to the system and its usage/behavior
• it's not exhaustive (i.e. it's not totally accurate)
• it's focused on testing, although we can see similirities to DDD (Domain Driven Development)

In SMBT models are built as directed graphs, composed of states (i.e. vertices) and transitions (i.e. edges). This way, it's easy to visualize and have a discussion around it.

After we build out our model, we use a tool to generate a skeleton for our test automation code. The actual test code needs to be implemented (some commercial tools provide this ability to an extent).

Finally, we can run tests exercising our model. The idea is to go over the vertices (states) in the model using the possible edges (transitions/actions) and use is that in a way that maximizes coverage; I'm oversimplifying it though. To "walk" in the model (i.e. to go from one vertex to another one using a specific edge), we need to have a "path". The MBT tool can generate an incredible high number of paths, using a "generator" algorithm, and stopping it when a certain "stop condition" is achieved.

And that's it: we have now a way to perform interactions in our SUT accordingly to the model we've defined earlier.

Note however that MBT can be used even without automation as a way to talk about the system behavior and support manual testing activities (scripted or even exploratory).

Why Model-Based Testing?

But why do we need MBT after all?

Well, if we think on any system (web or non-web based), it's really hard to test all possible usage scenarios.

Implementing traditional test automation, implementing a test script one after another for validating a specific use case takes considerable time. And more than that: their coverage will always be limited, even if we use data-driven testing.

Therefore, MBT is an excellent technique to increase coverage through more diversified test scenarios. It can also be used to quickly understand legacy systems without having to deep dive into the code specifics.

Doubts

I had a bunch of doubts at start, as usual.

• When should I use SMBT?
• What are the scenarios where SMBT is not the best fit?
• What can we map as being a vertex (i.e. a "state")?
• What are the edges (i.e. the "transitions")?
• Where should we implement the assertions?
• Semantically, what is the test scope here? An automated test script exercises a specific use case, and what happens with MBT?
• What's the relation to requirements, features, user stories?

Some of these doubts are starting to fade away as I go through real experiments but there's no right/wrong answers for most of them.

Meanwhile, several doubts arise. One that is not yet totally clear to me is related to state management:

• Where to deal with it? In the model susing the vertices themselves, as variables set in the model, or as internal variables managed in the underlying test automatiob code? Where's the border?

Modeling and its challenges

What we do when we test is to create a model of our systems in our brain and then we refine it through experimentation and validation.

This model is not unique, though.

Imagine yourself, for a moment, talking about some dream house that you've seen and that you would love to buy. Now think about someone else talking about that same house.

Do you think that you would be looking at it from the same perspective? Or would each individual see themselves having their own experiences, mostly different, in that house? More, one could think on the house as either being "under construction", "ready for living", or "inhabited" while the other one could think on it as being "with people resting", "having a party", "having lunch", etc.

Therefore, how we model a system can be done in many ways. But where to start?

If we have a website (e.g e-commerce site), we can model it from the UI perspective, where each state corresponds to a page. Transitions are made through interactions users make (e.g. go to homepage, find owners).

However, this is not the only possible modeling approach; we can think on the state of a transaction and model that for example. So, yes, we can use MBT to model beyond just UI interactions and use it to model algorithms, for example.

Modeling requires experience because it's a simplification on reality based on a perspective that matters. All of that is subjective, have you noticed?

MBT and unknowns

Testing is about learning, learning more about what we're building, why we're building, how its being built, and how users tasting that special "meal" that is our product. Does it taste well? Is it missing something? Was it delivered at the right time, to the right users?

Well, with testing we try to uncover all these unknowns that surround our products and that live inside them. Model-based testing can be used as a great technique to expose some unknowns and confirm certain behaviors of our products.

Models are easy to understand and visualize, and also can be easy to automate and check. By writing an executable model, we can (in)validate assumptions and then refine it. We can even fork and create additional models to tackle different aspects.

Modeling has also one interesting consequence: it promotes discussion. By promoting discussion, unknowns can be tackled on people's heads. Therefore, pairing or doing it as a group (ensemble programming) can provide additional benefits.

Tooling

There are several tools out there, including open-source ones. Probably the most popular is GraphWalker, which is tailored for Java skilled users (as it generates Java code and then can run it).

Python and C# users may prefer to use AltWalker instead. AltWalker uses GraphWalker on the backstage.

Commercial tools are also available; I've seen some but not yet tried them myself. I would say that there are at least two great differences:

• visual editors are better
• test automation code generation supports many more languages and frameworks

Tips

If you are new on this topic, I already have some tips that may be useful.

• Don't try to make complex models as they will be hard to read and maintain
• Have few vertices and edges; focus on the essential
• Spliting the model in several and use shared states, if it makes sense
• Even if we're talking about the same feature, having different models can provide different angles to it and to expose different bits of it

Learn more

I would recommend the following resources to get your hands dirty :) Any of the two examples is simple to understand and to get running.

• GraphWalker e-commerce site (Java example)
• altwalker e-commerce site (Python example)

If you want to know more about some specific open-source tools, that also talk a bit about the rational of model-based testing, please check:

• GraphWalker => the wiki pages on Github provide several examples and explain concepts
• AltWalker
• Visual model editor for AltWalker and GraphWalker
• AltWalker Model Visualizer for VSCode

In the Quality Ice Cream Truck all is around value as we'll see ahead.

What is value?

Many of us think that value is something highly well-defined, static, and probably around visible features. However, value is subjective, it changes with time... and it's just more than features. Quality defines value, by defining what is relevant for the stakeholders. Here I'm talking about quality criteria (e.g. correctness, usability, scalability, etc). By agreeing what composes value, then the team can use that to drive other collaborative activities.

Distilling the "Quality Ice Cream Truck" model

First, I hope this model visually resembles an ice cream truck, at least for me it does :) It's an evolution of a previous model on the relationship between Quality, Value, Testing and Risks by Rich Rogers.

Value is at the center and it's the actual driver of our ice cream truck; it's what we and our users want to "taste" and enjoy. We know that people like different flavours, sizes, consistencies, etc, of ice creams. It makes sense since value is derived from our own definition of quality.

Knowing what is relevant will shape our testing, which in turn will provide us information on our quality. Testing informs us on (negative) risks, opportunities and also gaps. We can consider a defect as a risk in this model, since we can decide how to handle it (i.e. its treatment, from a Risk Management perspective). Even though we may consider that testing is about risk assessment, and treatment as a natural consequence, I would like to highlight opportunities and gaps for a moment because they're forgotten so many times. Having in mind value, testing looks at all spots where we can increase and optimize it. That's why I look at testing not just as bug preventing/finding but as all other ways that can ultimately give us tips to increase and accelerate the delivery of value.

Testing does not affect or assure value directly. Testing produces information, hopefully shared and discussed as a team, that we can act on. By acting on the information, we may affect value and try to maximize it. Value is directly affected by how we code/build the product and how we operate and maintain it.

Since value is served on production, we need to understand what is happening there to fine-tune it ASAP. Operations and support informs us about how value is being delivered, served and tasted. We do use monitoring tools to provide some insights even though they're limited. We can go a step further and use observability to analyze raw data, trace requests, see patterns, understand what is happening, i.e. to explore and ask questions to the system that we don't know in advance. This allows us to highlight risks and confirm the value of our product by looking at its usage. It can also give us tips on possible opportunities and gaps that we can try to address.

Final considerations

There are for sure more relations and more entities that we could represent in this model but I tried to simplify it to a point. As we saw, information comes from multiple sources. From those, testing outputs are essential to provide insights about quality. To improve the product itself and its value, we need shared understanding and collaboration, so that necessary changes, no matter if it's in the product itself or in the surrounding processes, can be implemented... no matter by whom.

I challenge you to make your own version of quality, testing, value and activities around it. Putting it on paper/screen can foster some healthy discussions and bring to the table all the different aspects that contribute to delivering value to customers... which is always a team effort leveraged by the continuous understanding of what value represents.

Useful references

• A model of the relationship between Quality, Value, Testing and Risks
• The Quality and Testing Information Model: Getting Important Quality-Related Information to the Right People

I'm curious by nature, maybe due to my deep and intrinsic "test nature" :) but I also have some bits of a "crazy scientist": I like to experiment by myself, watch, taste and learn.

Last year, I launched this blog. I told to my self: "It's time to do it... so let's do it in an Agile way!". I looked at different SaaS (e.g. Wordpress, Wix) and choosed Wix. As I started adding some content and working with the site backoffice, I realized it was getting really slow and full of bloatware. I confirmed this qualitative feeling with some metrics(obtained using Lighthouse) that made me even more upset.

There were many problems showing up in the report besides performance. How could I solve this? Well, sometimes you have to take a step back; it will hurt a bit but it will provide you so many benefits on the run.

Time go back to the basics

I asked myself:

What do I really need?

And the answer was:

• share some blog posts, once in a while, in a very easy way
• provide great experience for visitors, in terms of readability, accessibility, performance

Some nices-to-have would be:

• ability to move the blog easily to another host
• easily track and revert changes
• ability to easily make experiments, safely

After applying the changes herein detailed, some things I got besides the basics:

• on-demand live environments, per commit and per PR (pull request)
• easy backups (it's Git so...)
• ability to switch hosting provider (e.g. Vercel, Heroku)
• automated checks, per branch/PR using Checkly
• ability to easily bring in another contributor

Things I got rid of:

• low-speed
• accessibility issues
• bloatware
• lock-in on Wix
• Wix banner at top
• Wix cost

Let's see how I dit it.

How it started

I was reading about JAMstack; (more info on JAMstack WTF) and I ended up in this blog about E2E testing. Suddenly I discovered Next.js, Vercel and Checkly.

The idea behind JAMstack is to provide a development environment for web sites, using Javascript, APIs and Markup, where the latter is served as static, generated files without requiring explicitly a web server. Even though this has some constraints that limit its broader usage, it provides many benefits in terms of performance and security that can be applied in many cases. In case you've worked with blog kind of sites, you may have crossed with tools, such as (Jekyll)[https://jekyllrb.com/), that generate static sites. What makes JAMstack fun is that you can add dynamic behaviour to your static sites, using Javascript.

My experiment

From the Next.js documentation, I could try out some live examples of this framework and also have access to their respective source. It was really quite straightforward to create a blog and deploy it to Vercel in no time, either by cloning one of the repos or from the command-line.

git clone https://github.com/vercel/next.js/tree/canary/examples/blog-starter.git
# or ...
yarn create next-app --example blog-starter blog-starter-app

Then I could do some changes, evaluate the results with Lighthouse and also try the tools by myself. Let me take the opportunity to introduce them at a glance.

Lighthouse

Lighthouse is a tool that looks at certain quality aspects of your site, including performance, accessibility, SEO, PWA (Progressive Web Applications) support. It can be run from the command-line, CI or directly from within Chrome dev tools.

Lighthouse provides a score for each high-level item and then allows you to drill-down and look at the individidual checks where you may find additional information and recommendations, so that you can improve your site. It's quite simple and effective, for testers and developers; it makes easier to track some quality aspects from the start.

Some care though:

• score will depend on how and where you execute it from; if you're using Chrome, then it is highly recommended using an anonymized window so it doesn't get affected by any extensions that you may have
• if use a tool such as Lighthouse Keeper or other one that integrates with your toolchain, than results may be slightly distinct

Vercel

Vercel is a cloud-based orchestrator platform, tailored for "building" and deploying/hosting static sites; Vercel is also the team behind Next.js. As a developer, you can write the code, push it and Vercel will do the rest:

• watch a given branch and checkout the code from the Git source control provider
• build the site static files
• create a temporary environment and deploy static files there
• assign a custom DNS entry to the environment

Vercel has several integrations, including with GitHub; thus, one can also track the deployment on GitHub pull-requests.

Checkly

Checkly is a simple, yet powerful tool for DevOps teams; it can "accelerate" the delivery of software by providing feedback about your site based on browser (or API) checks, implemented using Puppeteer scripts. The checks can run from multiple locations and besides tracking the assertion results, one can also take screenshots and track historical assertion results and response times. On top of this, we can be alerted upon failures.

Checkly also integrates with GitHub, which is awesome; thus, we can also automatically run check and track their results whenever doing pull-requests. This provides valuable information before merging.

Implemented workflow

My workflow will be mostly based on git-flow but without the "develop" branch; I'll use branches to implement fixes/new features on my site. I will then create PRs and merge then if appropriate.

My website will be hosted on GitHub; I will also enable integrations with Vercel and Checkly, to ease deployments and have almost real-time feedback on my pull-requests.

In Checkly, I can quickly track my site status and performance.

I've created a check that goes over the main pages. This can be split into distinct checks but in this case I don't see the need for it as I don't aim to grow these.

const assert = require("chai").assert;
const puppeteer = require("puppeteer");

const browser = await puppeteer.launch();
const page = await browser.newPage();
const baseUrl =  process.env.ENVIRONMENT_URL || "https://myblog.bitcoder.vercel.app"
var url = baseUrl
await page.goto(url);
var  title = await page.title();
await page.screenshot({ path: "homepage.png" })
assert.equal(title, 'Sergio Freire\'s personal blog on software testing, agile');



var url = baseUrl + "/about"
await page.goto(url);
title = await page.title();
assert.equal(title, 'About');
await page.screenshot({ path: "about.png" })

var url = baseUrl + "/contact"
await page.goto(url);
title = await page.title();
await page.screenshot({ path: "contact.png" })
assert.equal(title, 'Contact');

var url = baseUrl + "/articles"
await page.goto(url);
title = await page.title();
await page.screenshot({ path: "articles.png" })
assert.equal(title, 'Articles');

await browser.close();

We can run the check manually right away.

Additionally, we may integrate it with GitHub so our check(s) is/are run automatically in PRs.

I've also implemented another check that will transverse my blog posts. In this iteration, the blog post URLs are hardcoded. But I aim to automate this properly.

![checkly](/assets/blog/improving-performance-security-accessibility-and-more/checkly_posts_script.jpg

Example

Imagine that I wanted to add a new feature on my website, or to fix something, or to make an improvement in general.. I would start by creating a branch.

git checkout -b fake_improvement

Then I would make all the code changes I wish and commit them.

git add ...
git commit -m "my changes"
git push -u origin fake_improvement

In GitHub I could see the new branch and decide to create a PR for it. Whenever looking at the pull-request, we can can see a section from Vercel giving us a link to the "build logs# and also to a preview environment having our branch code deployed. We can then go to that environment and see how our web looks like and perform some exploratory testing on it. But we can also have immediate feedback about the automated checks implemented using Checkly. In the following screenshot we can see that our two "automated tests" failed; this happened on purpose as I deleted one post and changed the title of the website.

We can also see the details (and re-run) of the test.

By having near real-time environments and test results, it eases me performing changes in the future. After fixing the problems, I can finally merge my branch (as easy as clicking on merge on GitHub) and if deployed successfully it gets assigned to my domain automatically (by Vercel).

Further improvements for the future

Checkly

• explore its potential further
• automate obtaining all blog posts
• check all site references for dead links

Lighthouse

• evaluate how to integrate Lighthouse in GitHub, to have information on the PRs
• evaluate how to have checks based on some criteria for the Lighthouse provided metrics

Blog

• review styles (e.g. cover, code blocks)
• review image sizes/formats

Key learnings

• don't be stuck in the past
• don't be afraid to try, especially if you can make a quick experiment and obtain some quick results
• Lighthouse is a great audit tool for web sites/apps (mobile/desktop), essential for testers and developers
• there are some tools (e.g. Vercel, Checkly, etc) that can easily integrate with your (source-code) versioning control provider and your delivery pipeline

Conclusion

Your project is never complete; it won't ever have all the things you wished for, nor the ones you didn't even think about.

Sometimes, you just have to start. You cannot wait for a ton of evaluations that will provide you all the things you need to decide; no evaluation is ever complete, IMO.

If you're working in a project from the start, then you can keep your eyes open and use tools such as Lighthouse to provide some quality insights. If your project is already alive, then a tool such as Lighthouse is also valuable for your team; as a tester you can look at some improvement opportunities and discuss them within the team and developers can also proactively fix.

After I've implemented my blog site, I realized that it was not meeting its ultimate purpose: being accessible, readable, high-performing, clean. If you leave a bad impression, it will be hard for visitors to return.

Choosing the mix GitHub + Verce + Next.js + Checkly + Lighthouse provided me the necessary bits to overcome my and my reader needs. This doesn't mean it's the best mix or that I won't change it in the future. But now I know that I've simplified what I had, improving key quality metrics, removing lock-ins and making it less expensive.

At the start of this article I've mentioned one secret of Agile, remember? Well, the secret is that, IMO, being Agile is not about adding more features, in an incremental way. Instead, it's about incrementally adding value. In other words, this means that sometimes it's about reducing what you have, making it cleaner, focused. And this may imply putting most (if not all) of what you have into the garbage and start it from scratch. Actually, you never start from scratch because your previous experience can help you get where you want, faster.

Some other curious stuff

• Vercel also integrates with Lighthouse. I actually discovered it whenever I was having a coffee, using my smartphone. I added it and obtained the results right away in few seconds. How impressive is that?
• Migrating my posts from Wix to Next.js took me a bit as I had to rewrite them in markdown. But now I'm getting used to it :)

Useful references

• Lighthouse
• web.dev
• Next.js
• Checkly
• Vercel

Feedback loops can be lengthy and error-prone

Having worked around product development all my professional life, I dealt with requirement elucidation and analysis for some years. Writing or reviewing an extensive and detailed specification for a system or even for a specific feature of a product is hard.

This scenario is normal in waterfall projects where the release date is normally several months ahead and it's seen as a deadline, to you and to your team. After having the specification sorted out, to a point, we start creating some other documentation for the architecture of what we're going to build. Then, implementation starts and verification is performed in bulk before the release is moved to an acceptance-testing phase with the customer. But that moment can be too late to receive feedback from customers or stakeholders. Maybe you just built the wrong product: it happens.

Agile aims to shorten the feedback loops, namely the one between the customer and the development team. By delivering small pieces of value continuously, customers can provide feedback and features can be better tailored to the actual needs. Clearly this process is more dynamic than waterfall; however, do you think it's immune to problems around what is being built versus actual needs? Well, no.

One underlying "problem" is that we as humans use language to communicate, express feelings or needs. As the communication chain increases, it also increases the noise level within it. I remember playing a funny game as a kid: a bunch of us stood together and started saying a message (e.g. a gossip) to the kid next to each other. The message would be said in very low tone and passed from kid to kid until reaching the one that originally said it. When the message was received at the starting point and was expressed loudly, we would laugh for a while. In fact, most times it had nothing to do with the original one.

Uncovering unknowns in communication

How we express ourselves is noisy and so is its interpretation. And this happens everytime we communicate, and not only at a given moment in time. One way to deal with it is by having frequent customer interactions along with the team right before the project starts and also during its lifetime. Why is it important?

Well, for several reasons:

• It helps create a shared understanding and uncover many unknowns
• It helps clarify concepts, terms and needs
• It helps identify and assess risks
• It helps prioritization and understanding what’s important and what is not

At a project's start, you can do a “three amigos” session... well, you can involve more than 3 people representing business, development, and testing units. I think having someone from UX and support is important, independently of whether you have them as separate roles or not. Probably you may have experienced design thinking before; “design sprints” are built on top of this concept. Design sprints provide a well-defined approach to have prototypes that can be tested by users, even before they’re implemented.

I tried a smaller variant of the design sprint concept but the idea is the same: Have customer present with the team Understand concepts, needs, where’s the real value and what is the ultimate purpose Diverge to get some ideas of possible solutions Converge to evaluate if some solutions are functional viable

During the project implementation and rollout, we should have frequent interactions with our stakeholders or the customer. Customers can be present in daily Scrums; I've worked in Agile-based projects with frequent (almost daily) customer interactions. I also know how it hurts whenever you can’t have that feedback on-time or even until the “last minute” of your deadline. No matter if we’re starting the project or you’re mi , questioning is really the.

Needs vs Solution(s)

Most of the time, people will come to you with the solution. And they can even give you that as a “requirement”.

Hey, we need a button here that does this. Hey, we need to generate a report with these columns.

Humans are creative and you know what? Everyone likes to “innovate” and have that “I had this amazing idea” feeling.

The problem is that ideas are like cocoa: we need to process them to have chocolate (i.e. highlight the needs).

The 5 whys technique provides a way of getting to the root cause, or in our case - the need. Questioning is in fact a skill testers have, and we all have, that can help us uncover what’s behind some closed door. Remember that we aim to optimize our overall effort building and testing the right product.

Pains are also Needs

It’s not just about uncovering and delivering “new features”; it’s also about discovering the pains and the gaps.

Pains are blockers to deliver value: they take time, effort, and costs. They distract us from thinking about ideas that could improve our product and increase the value provided by it. So, we need to discover these pains, which involves on-going conversations with the stakeholders/customers. Pains are gaps or background noise in our “requirements”, if we’re working on an existing product.

Some ways of uncovering these are:

• ask what the different personas do in their daily product usage
• if you can, observe (or monitor) people using the product.

How is all this related to testing?

It’s really important that we uncover unknowns from the start or else we begin to generate assumptions or misconceptions that will guide our implementation and our testing to paths. Only if we understand the actual needs, what is the ultimate goal, we can implement and test what we’re building properly. Many times we get excited implementing unnecessary features that will just increase our tech and test debt… and also the “needs debt”, if we can call it that. Having different perspectives from the start, helps clarify concepts, needs, concerns and thus uncover unknowns. Testing is about looking for information and understanding which starts at the big bang: right at the moment the universe of unknowns around your product begins. Please feel free to share your thoughts on how you uncover needs and help build a better-tailored product.

I remembered attending a testing-related lecture that presented a way of visualizing the known and unknown in a matrix which left me very curious.

After a quick search, I reached this video from Donald Rumsfeld that introduces and explains it briefly. There is in fact a great correlation between this perspective and testing. Testing, as I see it, is this hunt for information and understanding leveraged by unique human skills. It is quite diverse and broad (please check my articles on Omni Testing), as information has so many different variables.

Testing is highly related to uncovering unknowns; even if we have a high degree of confidence in the system and we know what to expect, we perform testing because we will only know that for sure after testing!

I decided to start a series on the unknown to exercise and refine some of my thoughts and share them with the community. In this series, I will uncover bits of the unknown, its relation to testing, and how we can use it to obtain information that may help shape what and how we build our products.

The unknown may be vast, scary, useless, and even intangible. At least that’s what we may think at first glance. But, the unknown can also be surprising, exciting, valuable, and discoverable. These are the two faces of the unknown. We can look at the unknown from the “dark side” or from the “bright side”.

The same happens with testing products: we can look at it from any of those perspectives. I prefer the positive one but I do understand that sometimes it’s hard to avoid that “dark side”, especially if we jump into a project with a high testing debt and low testability.

As I walk this journey I would like to understand a bit more about the unknown, including discovering important questions like:

• What is the extent of knowns and unknowns in products?
• What types of unknowns there are?
• Where are the unknowns?
• Tactics to uncover/expose unknowns?
• Is there any approach to testing and/or techniques that can better address unknowns? Are we ignoring something that could provide us more valuable information?
• How are we addressing the unknowns in scientific areas?
• What have we learned about the unknowns so far?
• Are our knowns solid? Or are they an illusion, mostly temporary?
• How to search within the unknown in a proficient way?
• How to increase the probability of lucky findings?
• How do humans interfere in the process of acquiring knowledge?
• How do we achieve conclusions?

By being aware of our limitations and also of our capabilities, we can achieve better testing. By “better” I mean it in a broader sense (e.g. effort, the relevance of information).

Stay tuned as I uncover bits of the unknown, together with you. If you have questions, you’re not alone. Remember, the “truth is out there” :)

A few years ago I started working partially remote in my current company. I have been a product manager and PO for a couple of years and then I became a solution architect. As a PM/PO I had to engage with the team frequently and that has its own challenges, namely, it requires discussing and explaining things often. Being with the team eases interactions and provides a better understanding of team dynamics and challenges within the team. Even though we were at the office, we had collaboration in our DNA; this means we are open to help each other complete their tasks and goals. We also used and use tools to ease collaboration at many different layers (Slack/Teams, Jira, Confluence, Bitbucket). Making the transition from office to my home (or to any other place), was mostly easy but I did it gradually. My company is remote-friendly and that helps a lot; if you're already used to collaborate and use collaboration tools, then it's even more straightforward. Nowadays, as a solution architect, I have many tasks that I need to perform by myself but many of them involve the team, as we work together to achieve goals and each one brings something to the table. Working remotely may seem a huge challenge to you; it seemed to me. I dealt with it and nowadays, it's

How is a typical day working remotely?

This is my "typical" remote day, especially if I'm working from home.

• have breakfast
• take a bath and dress up as you would do normally if you had to go to office
• go walk for a bit, sometimes having a coffee outside (this can help mark the ?* transition from personal to work mode)
• start my work around 9am (aprox.)
• have a break to see my birds or wash dishes (washing my dishes normally is quite brain-estimulating, curiously)
• lunch at 12h30pm
• walk a bit (occasionally, run for 30min)
• back to work mode at 1h30pm
• some very small breaks in the afternoon around the house/watching my birds stop working around 6pm (aprox.)

Remember: it's important to keep mentally and physically healthy. Having said this, here are my first tips for working efficiently remotely; there are many more; these are the first that came to my mind.

Dress up for work

Exactly. Take your bath and dress up as you would normally do if you had to go to the office. Prepare for your day of work.

Delimit your work time

Imagine yourself going to the office and coming back home. Try to have the same kind of habit, even though you're working remotely. It helps a lot to establish your working hours (e.g. 9am to 6pm); don't be afraid to adapt them a bit and add some randomness to them. One day you can start at 9h00, the other day at 9h15. Nobody will die because of it; we know that in IT it's hard to have fixed schedules but we can also use them to our own advantage. If you need to start a bit later, no worries. If you feel you may start sooner, also no worries. However, having some sort of working window will help you. Whenever I started working remotely, I almost forget to have lunch. Sometimes I would lunch at 2pm or later, which for me is totally not normal. Trying to adhere to a common working schedule will help you professionally and personally.

Have breaks

Working remotely doesn't mean you have to be working 100% of the time, without stopping for a bit. Having breaks (or the so-called "bio breaks") helps you being more productive. Unfocus and focus.

How to do this in practice?

• have a coffee/tea
• take care of your pets and give them some hugs
• go outside, have some air for a bit, if possible
• walk a bit

Delimit your workspace

Have your own dedicated workspace helps: a specific room inside your house or a co-working office. Avoid mixing your personal space with the working place, that way you can better focus.

Find inspiration

In your "lonely moments", you'll need inspiration. Once in a while, I choose a totally different place for working; if viable, I go outdoors for 2 or 3 hours. It helps a lot in my mood and also my inspiration.

Focus

Avoid distractions or work in places where they persist. For some people, it may be hard to stay focused. The good thing is that we can train ourselves. Think of what works best for you to stay focused and practice it. Remember: it's important to keep focused; however, it's also important to have some small breaks.

Collaboration

Depending on your background and your team background, collaboration can be easier or harder. I think that what is really important to have in mind is that work will be mostly done asynchronously even though you may have real-time tools. Using collaboration tools helps a lot because they allow you to work asynchronously and notify whoever if necessary. Thus, you can work in a Google doc or in a Confluence page and ask someone to review it; that person will be notified and will help you whenever possible. Another thing to ease collaboration and to be available to others is by understanding each one individual goals and also overall team goals. You can use Trello, Jira to give visibility to what tasks everyone needs to achieve. We are using a kind of Scrum-based approach: we use sprints, tasks and have planning meetings where we discuss things as a team.

"Live" chatting

If you use a live chat tool in your company (e.g. Teams, Slack) then you should be aware that: the fact that you or anyone are online in that chat tool doesn't mean that you/they are available all the time Chat tools can also be used as asynchronous tools. You can leave a message to a person/group and it may be answered later on. If what you're asking is urgent let others know, otherwise don't assume it. People will answer when they can. Remember that they can be completing other tasks or having their own break.

Some recommendations:

• use your online "status" to tell other ones about your availability, in an implicit way
• configure your tool to not send you notifications all the time (for example, if you're busy or if it's totally outside your working hours
• And please... don't call anyone without asking if that person is available first of all :)

Be with the team

Try to be with the team physically whenever possible. Even whenever you're remote, you can still "be with the team". You can be fun and still maintain relations with team members. It's not all about work; it's also about the individual. Have the time to talk and also to listen. Avoid loneliness/isolation.

Meet humans

You're working remotely right? That doesn't mean you should be isolated from the world. In times as we're facing currently with COVID-19, we should restrict ourselves. In more standard times, it's important to be with people in whatever activities (work or non-work related). Things you can do: lunch or have coffee with friends attend meetups go to the gym, dance or other activities If you can't or isn't recommended to meet humans on person, well you can always give them a call or attend some online events/calls.

Personal things

The fact that you're at home gives you also the ability to deal with personal things; sometimes you need a plumber to go to your house or other things. That's ok; that's the benefit of being at home. If you were at the office, you would be stressed about dealing with it. The fact that you're remote also allows you to adapt to your schedule. You may need to leave or go get your kids... that's part of life. Be sure to balance well professional and personal life; if you need to work a bit outside your normal schedule, let your partner be aware of it.

In sum...

There are is so much more than the above. If you see yourself suddenly working remotely, take a breath. You can make it. I started doing it gradually; at first, it was a bit hard (I love human contact/interactions). I gradually adapted myself. I think I now work quite efficiently remotely (more than at the office sometimes). Don't forget to socialize even if it's done in a virtual way. Set your workspace, set your working hours. Take care of yourself and work as a team, knowing that others in the team are also having similar challenges as you do. Every single of us is different; learn your differences, your challenges. There are ways to handle them. You'll succeed, no doubts about it.

What is testing debt?

We all heard about the technical debt concept before - the development decisions made to provide benefits in the short-term that will, however, inhibit benefits in the future; in other words, imagine doing a quick "hack" to provide a feature instead of properly choosing the right architecture to support it. Testing debt is similar.

Quoting Rob:

Testing debt occurs when a team intentionally or unintentionally chooses an option that yields benefit in the short term but results in accrued testing cost in terms of time, effort or risk in the longer term. -- Rob Meaney

Peter Varhol, puts it slightly differently:

The effort needed to find and fix issues that remain in the code when an application is released. -- Peter Varhol

In simple words: testing debt is all about the testing that you should have done but for some reason, you didn't.

Testing Debt Quadrants

During Rob's workshop, we mapped our debt using Testing Debt Quadrants.

The testing debt quadrants provide a way to visualize debt, including:

• untested things
• testing that is slow or impractical and that can lead it to be forgotten/not performed
• items/components that are complex to test
• items you cannot rely on

Therefore, we splitted the cardboard into 4 sections:

• impossible/impractical
• slow
• complex
• untrustworthy

Note: instead of seeing all of the above as problems, they are in fact opportunities.

The process starts by adding post-its, identifying a "piece of debt" (i.e. something that we are not testing thoroughly or the way we should). As a team, we identify the ones that are most relevant (i.e. the ones that contribute the most to the testing debt). For each one of these, we discuss possible ways of addressing them and add them to new post-its. And now we have concrete actions that we can take to reduce our debt and improve the overall testing.

From theory to practice: how we did it?

After the conference, I wanted to try this exercise internally and see if it was valuable or not. I also wanted to make visible some pain-points on our testing process. We did this with a small group, including myself, part of our testers and also the team lead. Even though my role was essentially a facilitator, as I've been the product owner of the product we are testing, I do have particular insights about it which I cannot neglect :)

We started by doing a retrospective on our current development & testing process. By drawing it, we could easily see who was doing what and when. It also allowed us to depict some improvement opportunities right away. I see this kind of diagrams as starting points for discussion; some care should be taken though, or else this can lead to long discussions.

We then moved to the Testing Debt Quadrants and starting writing and adding post-its to the 4 quadrants. Each one did this alone and in the end, we could see some overlap (which I'd say it is "normal" to happen). Where we position each post-it in the cardboard is somehow subjective; however, as we reviewed the whole cardboard, we move some of them and agreed on it. Note that, for example, some items can be slow and complex at the same time. Where the post-its end up doesn't need to be scientific; use "positioning" as a means for further discussion/insights. We saw that we had some unfinished work that could tackle some of the identified problems. We just need to turn that into reality.

From discovery to actions

Having discovered and discussed part of the items that contribute to our testing debt, it's important to have actionable, prioritized items. Thus, we decided to create a very simple Trello board to track their implementation and/or discuss anything around them.

It's interesting that as per our initial discussion during the "visual mapping" of our development/testing process, we could depict some opportunities to improve testing considerably. Thus, the board contained actions not only found during the Testing Debt Quadrants but also from the initial sketch we did about our overall process.

Things to have in mind

Whenever looking at its full extent, it's easy to get lost and start discussing a ton of different topics.

Some recommendations for us and for anyone trying it out:

• timebox it, so we don't extend it endlessly
• focus (no interruptions)
• involve other roles
• think on improving gradually, a step at a time (this will avoid finding the "perfect solution")
• revisit the cardboard regularly

I would also recommend having the conversations in such a way where there is no pinpointing. This should be a team exercise, open but not individual-focused.

And now what?

I think mapping your testing issues, which are also opportunities to improve your overall testing process, using Testing Debt Quadrants is a simple and valuable exercise. All the conversations you have are important to have a joint overview of how testing is being addressed and how it can be improved. Just make sure that these conversations aren't pointless and, instead, lead to actions that you can track and revisit together somehow.

Warming up

At the start, I felt a bit like a "stranger" in mid of the crowd: it was my first time there and I'm a bit always like that for a couple of minutes. Gladly, after few minutes I was starting conversations or joining ongoing ones. Every single person is different and not everyone may be in the mood to discuss something, but that's normal. Each one of us can be in a different mental state, at a given moment. We just have to acknowledge it and proceed. In general, people are open to a conversation; like me and you, they're attending also to exchange ideas, talk about their problems and their goals. Of course, if you see someone around that you've seen elsewhere, perhaps in a different conference, then you feel a bit more like comfy. I was really glad to see some recent, yet recurring testing friends :), such as ]João Proença](https://twitter.com/jrosaproenca) and Lisa Crispin, that besides being kind also are natural helpers. It was also great to meet Janet Gregory once again and exchanging some ideas with her. Rob Meaney became a new addition to my testing friends, especially after he met personally the one who "bugged him with questions" some days ago, in a webinar on operability :) We also had a lovely dinner together with Jeremias Roessler where we discussed a bit of all.

Mob learning in Lean Coffee, Speed Meet and during breaks

I would say that the best time to learn is whenever you interact and learn as a group. Breaks and lunch are great opportunities to discuss topics, so we should take advantage of them to improve our knowledge. I took some time to talk to Maaret Pyhäjärvi and discuss some challenges my own team faces along with tactics and strategies to overcome them. I can only say that Maaret provides very insightful tips based on her long time experience in the testing space, working with so many different people. Sometimes we have good ideas but the problem is on how to deliver them; experience is key. I also had a brief chat with Alex Schladebeck on observability, as we're both somehow newcomers in this space. It's interesting to discuss the doubts, challenges as we start to pave this road.

Speed Meet was quite interesting and cool, as a way to learn a bit about other attendees and what they're interested in but also to learn about ourselves as we have to sum up who we are, what drives us and where we want to go. I found people with great skills, some in speaking, some in automation, some in BDD, well... so many! Unfortunately, I did not have the time to further explore these quick connections, due to lack of time and also because I quickly forgot the names, interests/skills of them. I need to find a way to better handle this in the future. I had a Lean Coffee session nicely handled by Marianne Duijst (who made some great sketch notes during ETC by the way), was something kinda new to me. We discussed several topics together and we're able to help each other. It's an interesting way of having discussions around topics that the group wants to, adapt and move to a different topic when the group is ready to.

Talks and workshops

I already left my feedback on the talks and workshops to the speakers, so I'll take this opportunity just to talk about a very small subset of them.

Gáspár Nagy's workshop on BDD and Example Mapping was very straight to the point. We did some exercises together and we learned that everyone is able to depict new "rules" (i.e. acceptance criteria) and new examples for them, that would be hard to obtain otherwise. Thus, as a team and with a minimum "structure", it's possible to have a better understanding of the stories/features we aim to deliver. And BDD is all about that. It's curious (or not really, if you understand BDD) that we didn't even touch the "automation part" of it, that sometimes is only where people focus on without understanding the true nature of BDD.

João Proença's talk on deleting "automated tests"/scripts, was also interesting even though we don't pay so much attention to it as we should. Those old, "unknown", crapy automated scripts add pain and costs to your delivery pipeline. So, as your test automation grows, it is crucial to optimize it so you understand what is happening and get short feedback loops, which ultimately are crucial to enable CD. João will present this same talk at TestBash Brighton 2020, if you're lucky enough to attend it.

Rob Meaney's workshop on testability was very useful. His experience, based on concrete examples, allowed everyone to better understand what is testability about and how we can address it. I learned about "testing debt" and about ways to make it visible and addressable. Immediate takeaway: I will apply the testing debt quadrants this week with the team.

In the end...

Well, it was my first and my last EuroTestConf and I feel already a bit sad about it as I loved its format. At these conferences, you come with a bag of a few friends and you leave with a bag full of them. Even if you make just one connection with whom you can exchange ideas, it will be simply great. I learned about several topics: testability, BDD, exploratory testing among others. This is great as they were in fact my initial goals.

I took one day off, to "land" my ideas while performing some sightseeing and appreciating Amsterdam and it's beautiful surroundings, including the Muiderslot castle and Zaanse Schans.

Even though European Testing Conference has reached its end, the future is bright and everyone will have many opportunities to deepen their testing skills and connect with this amazing community elsewhere. Thanks again Maaret Pyhäjärvi and all the ETC staff for organizing it throughout the years.

Honestly, I was so glad to see my humble contribution to SeleniumLibrary during the workshop; glad that it is useful for the community (available from the upcoming v4.2 onwards).

There were two important messages that I consider to be quite interesting and relevant, especially considering the scope of the workshop : the fact that the maintainer of SeleniumLibrary project, Tatu Aalto, mentioned that he does very very few GUI based tests in his own case the general agreement that GUI/selenium based-tests, though interesting, should not be the main focus of testing; concerning automation, API-based tests give a lot more value (of course, on top of unit tests); this requires that testability is built into the product, right from the start During the workshop, René was very helpful and active, helping keeping it interesting for everyone.

What concerns the event, overall RoboCon was interesting, fun and motivational. There were many different presentations though, at least this year, I found RPA to have emerged as its own kind of topic (first conference day had many RPA tracks). E2E, GUI, and microservices testing were also covered in the presentations. I found out the talk about acceptance testing a real-time audio application (by Akseli Lukkarila, Yousician) fascinating as it depicted a scenario where instead of seeing docker containers you would see wooden boxes as different audio-dedicated test environments.

There were also other interesting presentations, covering data-driven testing (René Rohner) to more implementation topics, such as how to implement test automation in a more traditional organization with all typical challenges that we would expect (Anaïs van Asselt). In-between talks, we had a bunch of breaks, giving the opportunity for people to rest, stretch their legs and do some networking.

Even though there were people from foreign countries, most attendees were Finnish. Thus, most conversations in the breaks happened in Finnish (at least my impression); I think this could be improved if the organization promoted English as the main language that should be used (note that all presentations and the workshops were in English). This would ease people joining on-going conversations, IMO.

I also think that all presentations should have the Twitter handle of the presenter or any other relevant contacts. But these are just some small things, some nice-to-have. Talking about improvements, one of the challenges RF seems to yet face is good support for IDEs so they can provide autocomplete thoroughly. Gladly, we all became aware after Pekka's keynote that showed that this should be addressed right after RF v3.2 is out later this Spring.

But what have I learned?

These are some things that I learned at RoboCon or that were somehow reinforced in the back of my mind:

• RF is used in the most diverse contexts
• there is a vibrant and helpful Robot community, namely in Slack
• there may be automation libraries for almost anything and if not, it is probable that someone will make those; you may find some in this list
• some libraries may become deprecated/unmaintained; some care should be taken whenever choosing libraries
• open-source means receiving but also giving

What concerns more techie things, I wanna try out some libraries afterward, namely:

• robotframework-seleniumtestability
• robotframework-datadriver
• RF Swarm
• among other

I want also to explore ways of extending RF and some libraries in the ecosystem.

After RoboCon ends, a kind of call-for-action is somehow triggered in us. I'm eager to learn more about additional topics, libraries and... who knows... perhaps make some related open-source contributions later on? :)

I still had some last time to do Santa's work :)

See you soon :) You may find and follow me on Twitter and other networks.

It was quite interesting because I could try new things and help others. On the other hand, it was also quite challenging because the company was highly hierarchized and most of the teams were working in waterfall, for highly complex products, with a long history and... old technologies. I loved working there, since there were a bunch of departments working in quite different things: messaging systems, intelligent networks, mobile and TV apps, hardware equipments and more.

The context

At a certain point, I was assigned the task of helping an "external" team implementing successfully test automation on fiber optics "solutions" (we'll come back to this).

To give a bit more background, this team was a testing team, part of a broader testing team that in turn was part of a department. The department had several development teams, mostly hardware related but also software related. Each software team was working in a particular area; some in the CLI (command line interface) of the equipment, some in the management systems, some in the firmware, etc. When a customer bought a product, they were buying the whole stack of products or a combination of some. Some product existed by themselves while other only while in the context of the whole stack. To make this even harder, some customers had their own configurations (i.e. variants of the product(s)).

Testing was essentially being done end-to-end, sometimes at the management software level. Regression testing was being performed manually against each build; as time was scarce, only a subset of tests could be executed; it was simply impossible to execute everything because meanwhile a new build was knocking on the door.

Challenges

We had many challenges, technical and non-technical; in hindsight, our biggest challenge was human related which is also the hardest to overcome, IMO.

Some challenges included:

• lack of communication
• testers seen as the end of the line and the ultimate responsible for quality
• siloed teams, i.e. teams working separately (devs vs testers, testers vs testers, devs vs devs)
• many assumptions
• missing or outdated documentation
• lack of coding/development know-how
• lack of expertise in test automation
• no CI/CD
• very few automation, except for a few scripts

Besides this, our target "solution", or SUT, was too complex and was apparently impossible to automate because we were talking about physical hardware with many physical interconnections, optical and electrical. Some of these networking equipments are not easy to automate from the outside.

Strategy

As we had a strict timeline and we want to show results to all stakeholders (yes, this is probably one of the most important aspects), we needed to be very pragmatic. Thus, and since the team was very small (4, myself included), we defined clearly the SUT and told everyone that we wouldn't be testing everything; it wasn't viable. We also stated that this allowed us to do more things in the future and that this part was essential.

We looked at the whole fiber optics system (customer and operator side) and choose the operating system of the main backend/operator side equipment as the main target for testing. We then looked at all the available interfaces and thought from an automation point-of-view: "Can we use this interface to communicate to the equipment? Can we build an API somehow on top of that? How easy is going to build that API and how stable it will be? What are the cons?".

What seemed to be impossible at first sight, soon became feasible. We were thinking "hmmm, we can interact with that equipment if we do this or if we use that...". Building these layers of communication allowed us to build the foundation where we could build our test automation on.

Coding skills were low at the team; no problem, we all can learn. Ruby was chosen as it is a scripted language quite easy to learn; besides, there are a ton of libraries that can easily extend it to do whatever we need. Thus, the team doesn't have to deal with low-level kind of details and may see results sooner.

Cucumber was used to implement the testing scenarios; we were not adopting BDD but we were taking advantage of the Gherkin syntax and the natural ability of having reusable, executable steps across different scenarios.

Since the team was using Jira, we stopped building out reports manually in Word, PDF, Powerpoint, whatever, and sending them by email; instead, we used Jira and proper dashboards that could be accessed even by C-level roles.

Even so, we met on person with all the team members, from the different teams, to show where we were and the gains we were already obtaining.

In sum,

• simplify our problem, i.e. make our SUT well-defined and remove the moving parts
• build APIs if necessary; abstract the hard parts and make them easy to use
• focus on regression testing
• think on risks and in what risks you're neglecting because your time isn't enough
• mentor other teams for "testability"
• provide real-time visibility of progress
• show what you were not doing before vs what you started to achieve

Lessons learned

• automation can be implemented successfully even on teams with few or no coding skills whatsoever
• documenting, even if briefly, reusable executable steps/keywords can help providing awareness of what's available
• building reusable libraries, with a main responsible/"owner", can motivate and empower people
• using Ruby+Cucumber was a good fit; in general, using languages and frameworks where people feel comfortable and that don't require too much bloatware can help achieve results faster and provide that feeling of progress sooner
• successful automation requires testability as a main concern from the start

If you're going to implement test automation in on-going project that didn't have testability as one of its concerns from the start, it is going to be hard but it's not impossible.

In conclusion

Even though systems may be complex, people are probably the biggest blocker for automation :)

So, if you're going to implement test automation in an on-going project make sure that (in so special order):

• you have good communication between everyone (think on ways of promoting this)
• that quality is something you all agree on and is everyone's responsibility
• everyone is involved in the "test automation project", by sharing goals and tasks
• everyone agrees, as much as possible, with the immediate, mid and long-term goals
• everyone understands the importance of testability

When the previous points are established, you may proceed with the more technical part:

• start simple, decompose your problem and focus your automation efforts
• if you have parts of your (aim to be) CI pipeline that are not automated, work on those (e.g. making sure the build process is fully automated and have no local dependencies)
• don't automate everything or blindly; think from a risk perspective (more on RBT here) and on the repetitive tests/checks that you have to perform on every single build

Learn more

To know more about the details, you can watch the following video presentations; they're a bit old though :) Slide deck is also available. If you want to know more about the challenges we've faced, let me know and I'll be more than glad to go into more details.

Presentation videos follow.

Welcome to my website and my first blog post here. I have been postponing the creation of a website, including owning the domain for quite a while. I wanted to have something cool and elaborate. However, this got delayed, postponed, forgotten. Ever had this kind of feeling in your projects? Probably so, if they suffer from those waterfallish kind of problems. I aim to improve this site with time, by continuous learning and iterating on it. Part of my goals include writing here some stuff once in a while. Topics will include testing, Agile, technology, whatever comes to mind and that may be relevant, at least for some of you.

I hope you enjoy it, feedback is always welcome!