Can AI (ChatGPT) be helpful in testing?

Lately there is a lot of talk on using the artificial intelligence in many fields. Recently social media were flooded with a wave of graphics generated by AI (Artificial Intelligence) – among others by the Lens application. Therefore, it is an interesting topic to explore in the context of using this technology for testing.

In the article I will be covering the usage of the AI in testing, concentrating on the ChatGPT tool. In the text you will find a couple questions and issues related to testing and I will present how ChatGPT reacted to them. I will check the AI responses concerning i.e.:

• Creating test cases,
• Automated testing,
• Accessibility.

What is ChatGPT?

Let’s ask ChatGPT itself what it is and what sources it uses to give responses:

Applying AI in testing

Test cases

Let’s begin with the issue, which most of the testers encounter at the beginning of their career path. Let’s see how ChatGPT deals with two questions regarding the creation of test cases. I asked ChatGPT about logging to a webpage and a mobile app test cases. You can see the responses below.

Currently I am working on a mobile app project in a medical market. That is why I have added more details to the query above.

Here is the result:

ChatGPT’s response was surprising to me, because – apart from a couple differences – I could use this test case in the project.

Questions on automated tests

We will now look at how AI will deal with a harder subject regarding automated tests.

Let’s begin with something easier, so a query on how to open a webpage using Selenium.

The code above can be considered correct for the Python language.

We did not specify the name of any programming language in our query. Let’s see what happens when we specify it and indicate a specific language.

I have checked if the above code works. Below is the result:

I did not have any problem with running the above code, and the outcome was as expected, which is Sii website has been properly opened in the browser. 

Of course, we can continue to improve our query. Below I have modified it, so that ChatGPT tries to verify if the webpage title is correct.

Naturally, in real life we should not use such code – instead use an assertion. Therefore, now will ask ChatGPT for an assertion.

In the examples above ChatGPT has no problem with indicating libraries, which should be imported for the code to work properly. Still, I was surprised with the fact, that the tool has also managed to prepare the code with the inclusion of assertion in TestNG.

Question on Robot Framework

Unfortunately, ChatGPT not always generates response in such a way. In case of a query regarding Robot Framework, the tool did not indicate the necessary library to import.

In this case the SeleniumLibrary should be imported.

Questions on Selenium, Cypress and Playwright

In case of Selenium the most frequently asked question concerns the differences between implicit wait and explicit wait. Therefore, it is worth checking if artificial intelligence will handle that issue.

I leave the evaluation of this response to you An excerpt from the documentation on this issue may help you with it.

ChatGPT is able to respond to even more complicated queries:

A tester does not live by Selenium alone Nowadays, Cypress is also frequently used and Playwright framework is gaining popularity. Can ChatGPT be helpful in this case?

In case of Cypress the code generated by the tool was working and performed the test correctly:

describe('www.sii.pl', () => {
  it('should open the homepage', () => {
    cy.visit('https://www.sii.pl');
    // Assert that the page has loaded successfully
    cy.title().should('eq', 'Strona główna - SII');
  });
});

Example:

Unfortunately, the situation was not that colorful in case of Playwright. ChatGPT prepared the following code for me:

const { chromium } = require('playwright');
describe('www.sii.pl', () => {
  it('should open the homepage', async () => {
    const browser = await chromium.launch();
    const page = await browser.newPage();
    await page.goto('https://www.sii.pl');
    // Assert that the page has loaded successfully
    expect(await page.title()).toEqual('Strona główna - SII');
    await browser. Close();
  });
});

Taking into consideration the format, which ChatGPT has adopted, in my opinion the code should look like this:

import { test, expect } from '@playwright/test'
test.describe.only('www.sii.pl', () => {
  test('should open the homepage', async ({ page }) => {
    await page.goto('https://www.sii.pl');
    // Assert that the page has loaded successfully
    expect(await page.title()).toEqual('Strona główna - SII');
  })
})

Of course, in my research I have also checked other issues concerning automated tests and programming, including queries regarding:

• Appium,
• HTML,
• JavaScript.

ChatGPT and the issue of accessibility

For some time now I have been very interested in the topic of the accessibility. The WCAG standards are connected with that topic.

ChatGPT responds in different languages. For me, the response in Polish was surprising. I also think, that creating web content accessible also for infants is very hard

Of course, also in the case of A11Y (accessibility) we can ask more specific questions.

It should be mentioned here that ChatGPT does not always work as expected. In the example above, I asked the question in English and the answer was given in Polish.

Controversies concerning AI

The main goal of the article was to present AI possibilities in the context of testing, however I think that the controversies concerning artificial intelligence are worth mentioning.

Copyrights

Media and authors report, that AI very often uses other people’s work. Midjourney creator said to the Forbes magazine:

There isn’t really a way to get a hundred million images and know where they’re coming from. It would be cool if images had metadata embedded in them about the copyright owner or something. But that’s not a thing; there’s not a registry. There’s no way to find a picture on the Internet, and then automatically trace it to an owner and then have any way of doing anything to authenticate it.

From these words, in my opinion, confirmation of the use of other people’s works by artificial intelligence can be deducted. It has a particular significance in the context of the yet another case of right infringement of millions of creators, who decided to file a class action lawsuit.

Education and critical thinking

Schools in New York have restricted access to ChatGPT. Below are the words of Jenny Lyle, spokesperson for the U.S. Department of Education:

Due to concerns about negative impacts on student learning, and concerns regarding the safety and accuracy of content, access to ChatGPT is restricted on New York City Public Schools’ networks and devices.”

While the tool may be able to provide quick and easy answers to questions, it does not build critical-thinking and problem-solving skills, which are essential for academic and lifelong success.

I do agree with these words. I live in a city without access to any sea or lake, and ChatGPT tries to convince me, that there are beautiful beaches in Reda.

Can AI be relied upon?

Questions arise:

• Can AI replace a human being?
• Can responses generated by these tools be trusted?

Programs using artificial intelligence still often give wrong answers or provide outdated information.

Above, I have presented and example of the code, that needed to be corrected in order to work, as well as a response, in which ChatGPT proposed ensuring website accessibility to infants.

In my history of using the tool I have also probably encountered a situation of providing outdated information. I asked about differences between Cypress and Playwright. The response was as follows:

Playwright supports a wider range of browsers than Cypress, including Chromium (used by Google Chrome), Firefox, and WebKit (used by Safari). Cypress currently only supports Chromium.

I decided to look into that response and verify it. On the official website, Cypress informs about the support of many browsers and engines, i.e.: Chrome, Firefox, Edge, Webkit. In this case, we may be dealing with a ChatGPT limitation. ChatGPT contains information until 2021.

Therefore, it is a confirmation, that a human and their control are essential.

Other IT areas are also considering the possibilities of using AI in their work. On YouTube you can find materials on the use of artificial intelligence in the process of creating a website or application. It is worth looking into what they say.

Summary

In my opinion artificial intelligence can only be used as a support in daily work of, for example a tester, and we should not rely on it immensely. Data obtained through AI should be verified and approached with great caution. I am a big opponent of creating content based on ChatGPT responses on a copy/paste basis.

An open issue is still the question of ethics and copyrights. To me it seems that it is something we have to face in the next few years. In relation to the above doubts, I believe that AI cannot replace a human being.