Tutorials

Automated web application testing with AI and Playwright

End-to-end tests are essential but fragile. Learn how Heal.dev and AgentQL make them robust with AI-powered selectors that adapt as your web app evolves.

End-to-end tests are one of the most valuable testing strategies in any codebase. They are the ultimate integration tests that ensure the application's critical paths are thoroughly tested. With them, testing teams ensure the high quality releases and reliable user experiences while reducing the need for human testers to perform repetitive tasks and manual testing.

However, these are often overlooked when building web applications, with good reasons! The most common justifications for not using end-to-end testing are:

  • Cost of implementations: Even if modern testing processes are easier to use than ever, they still use traditional testing methods. Users still need to spend significant time and manual effort choosing selectors on a page, and variable testing time can lead to slow iteration cycles.
  • Cost of maintenance: End-to-end tests are often coupled with the implementation of the application's user interface, meaning they require a maintenance when modifications are made that impact appearance and user behavior, like UI updates or new features and capabilities.
  • Flakiness: Because a website's performance is impacted by a wide range of parameters, from CPU scheduling to network stability, end-to-end tests tend to be flaky. Web applications behave slightly differently from one run to another, leading to a significant loss of time when executing such tests.

This tutorial explores AI-powered testing to build more stable tests, leading to greater testing efficiency, shorter testing cycles, and faster release cycles. To do so, you will use:

  • Playwright is a browser automation tool and testing library by Microsoft. Its robust locator API significantly reduces flakiness when interacting with an element.
  • AgentQL has AI-powered features that turn any web page into an API you can interact with using Natural Language queries. Instead of targeting elements on the page imperatively using CSS selectors like .submit_button12lsi or XPath , you describe what they are semantically, like "submit_button". These self-healing tests dramatically reduce maintenance as they no longer break when the page's layout or content changes.
  • Node's built-in test runner
  • Node's native support for TypeScript (still experimental)

This means the package.json will be particularly small:

{
  "name": "agentql-tutorial",
  "type": "module",
  "version": "1.0.0",
  "main": "index.js",
  "scripts": {
    "test": "node --env-file=.env --test --experimental-strip-types test/**.ts"
  },
  "author": "",
  "license": "MIT",
  "dependencies": {
    "@playwright/test": "^1.49.0",
    "agentql": "^1.6.0",
    "playwright": "^1.49.0"
  }
}

Using Node.js test runner with Playwright and TypeScript

Setting up tests with Playwright and Node.js

Start with a test for the TODOMVC application hosted by the Playwright team that will:

  • Add elements to the To-do List
  • Confirm that the elements are present

The Todo app landing page

The whole test file is available here, or you can follow along below

Setting up the test context

You need to provide a Playwright Page for each test to run your tests. That way, if you create multiple tests, they will stay independent from each other:

import {
  after,
  afterEach,
  before,
  beforeEach,
  describe,
  test
} from 'node:test';

import { chromium } from 'playwright';
import type { Page } from 'playwright';
import { expect } from '@playwright/test';

import type { TestContext, SuiteContext } from 'node:test';

const TEST_TO_PAGE = new WeakMap<TestContext | SuiteContext, Page>();
let browser = null;
before(async () => {
  browser = await chromium.launch();
});

after(async () => {
  await browser.close();
});

beforeEach(async (testContext) => {
  const page = await browser.newPage();
  TEST_TO_PAGE.set(testContext, page);
});

afterEach(async (testContext) => {
  await TEST_TO_PAGE.get(testContext).close();
});

Notice that two imports are made with import type instead of just import. This ensures Node.js knows what imports to remove when transforming code from TypeScript to JavaScript. For instance, Page is not exported by the playwright module, but its type is! So, such import needs to be removed before running the code; otherwise, it will crash with an import error.

Now take a look at the hooks (before, after, beforeEach, afterEach) in the code:

let browser = null;
before(async () => {
  browser = await chromium.launch();
});

after(async () => {
  await browser.close();
});

The before hook is executed once before each test, and the after hook is executed once after all the tests. This is the perfect place to create and destroy resources needed for the test session!

Start the browser instance you will use for testing: close it in the after hook to ensure it is permanently closed. If closing the browser depends on tests in the suite, it might not happen, as the tests might crash and prevent the browser.close() method from being called.

Additional hooks run before and after each test. In the beforeEach, create a Page for each test. In the Node.js test runner, each test has a single TestContext object. You can use this object to link resources related to the test here:

const TEST_TO_PAGE = new WeakMap<TestContext | SuiteContext, Page>();
beforeEach(async (testContext) => {
  const page = await browser.newPage();
  TEST_TO_PAGE.set(testContext, page);
});

afterEach(async (testContext) => {
  await TEST_TO_PAGE.get(testContext).close();
});

This creates a new page before each test and links it with the current test using a WeakMap.

Note: you could also have used the Playwright test runner directly, as it would have handled these fixtures!

Write the test

test('should allow to add a TODO item', async (testContext) => {
  const page = TEST_TO_PAGE.get(testContext);
  await page.goto('https://demo.playwright.dev/todomvc');
  const newTodo = page.getByPlaceholder('What needs to be done?');

  // Create 1st todo.
  await newTodo.fill('Use AgentQL');
  await newTodo.press('Enter');

  // Make sure the list only has one todo item.
  await expect(page.getByTestId('todo-title')).toHaveText('Use AgentQL');

  // Create 2nd todo.
  await newTodo.fill('Use Heal.dev');
  await newTodo.press('Enter');

  // Make sure the list now has two todo items.
  await expect(page.getByTestId('todo-title')).toHaveText(['Use AgentQL', 'Use Heal.dev']);
});

The first two lines of the tests are here to:

  • Get the Page object
  • Navigate to the URL

Then, use a locator to find the input for new To-do List items:

const newTodo = page.getByPlaceholder('What needs to be done?');

This locator represents the component on the page. Playwright will handle all interactions with it for you!

await newTodo.fill('Use AgentQL');
await newTodo.press('Enter');

This tells Playwright to type Use AgentQL in the input field and to press ENTER to submit the task.

Now check that the task has been properly added to the list:

await expect(page.getByTestId('todo-title')).toHaveText('Use AgentQL');

This asks the Playwright assertion library to check that within the elements on the page with the id todo-title, one has the text "Use AgentQL," meaning that the item has been added to the list.

Add another item, "Use Heal.dev," and check if it is present in the list, too:

await newTodo.fill('Use Heal.dev');
await newTodo.press('Enter');
await expect(page.getByTestId('todo-title')).toHaveText(['Use AgentQL', 'Use Heal.dev']);

Running the tests

Run the test with the following in your terminal:

node --experimental-strip-types test/without-agentql.ts

You don't need the --test flag when running a single test file. Node.js will run these standalone files as tests automatically!

Running these tests with --experimental-strip-types allows you to run .ts files without node-ts or any translation!

Self-healing tests with AgentQL's smart locators

You just wrote a test for the To-do List application! But what happens if the placeholder of the task input changes? Or if the testId used for the elements in the list is modified? The test will break!

Of course, you can always maintain the test as part of your development cycle and manually change the locators, but what if you could use AI to handle that?

Using AI to identify the target elements is a great way to stabilize these tests. AgentQL uses machine learning models that enable you to test using the meaning of elements on the page rather than their implementation details.

Adding AgentQL's AI-powered testing tools to the test

You can follow along or find the complete file here.

First, import AgentQL into your test:

import type { PageExt } from 'agentql';
import * as AgentQL from 'agentql';

And add another beforeEach hook to wrap Page with AgentQL:

beforeEach(async (testContext) => {
  const page = TEST_TO_PAGE.get(testContext);

  const pageExt = await AgentQL.wrap(page);

  TEST_TO_PAGE.set(testContext, pageExt);
});

This adds the following methods to the page:

These powerful methods can invoke AI to locate elements and data on the page by focusing on what the elements expose as features or data instead of their implementation.

getByPrompt('Entry to add todo items') replaces getByPlaceholder('What needs to be done?') in the test:

test('should allow to add a TODO item', async (testContext) => {
  const page = TEST_TO_PAGE.get(testContext);
  await page.goto('https://demo.playwright.dev/todomvc');
  const newTodo = await page.getByPrompt('Entry to add todo items');

  // Create 1st todo.
  await newTodo.fill('Use AgentQL');
  await newTodo.press('Enter');
  const todoItemsQuery = `{
    todo_items[]
  }`;

  // Make sure the list only has one todo item.

  const { todo_items: itemsBefore } = await page.queryData(todoItemsQuery);

  Assert.deepStrictEqual(itemsBefore, ['Use AgentQL']);

  // Create 2nd todo.

  await newTodo.fill('Use Heal.dev');
  await newTodo.press('Enter');

  // Make sure the list only has one todo item.
  const { todo_items: itemsAfter } = await page.queryData(todoItemsQuery);
  Assert.deepStrictEqual(itemsAfter, ['Use AgentQL', 'Use Heal.dev']);
});

This means that as long as there is an input to add a task to the list, AgentQL will provide a locator for it, even if its placeholder changes.

This also works for extracting data from the page:

const todoItemsQuery = `{
  todo_items[]
}`;

const { todo_items: itemsBefore } = await page.queryData(todoItemsQuery);

This AgentQL query returns the elements on the page that match the description todo_items without impacting how the page is implemented.

Both methods are self-healing and resilient to page changes, making your test more robust!

Running the test

node --test --experimental-strip-types --env-file=.env test/**.ts

Note: Use the --env-file flag to target the file containing AGENTQL_API_KEY=<MY_API_KEY>

What’s next?

This example mitigates some of the top challenges of end-to-end testing: the cost of maintenance and flakiness. But there are still things your testing team can do to keep these tests running over time for more complex applications.

Update relevant tests as part of your feature development process

Using AgentQL's smart locators that target elements by their on-page context rather than hardcoded selectors makes tests more stable over time. But this only works as long as user interactions on the page stay the same. For example, the next version of this To-do List could require users to add a time estimate to the entry. You would still need to update the relevant tests to address new user stories like this.

Build a testing infrastructure

Hosting end-to-end tests locally is difficult. Browsers require a certain amount of CPU and memory (and GPUs on some websites). Also, scheduling browser-based tests can be challenging. You must ensure the network state is stable to load the target website and assets and even to connect to AgentQL servers!

Integrate complex testing workflows

Many testing building blocks are painful to build right. For example, testing sending and receiving emails, designing clever ways to test randomness, and writing assertions. How do you automate “check that a modal is open” in a stable way? That remains tricky!

The Future of End-to-End Testing is AI-Powered Testing Tools

Next generation reliable testing tools will embrace self-healing testing, relieving testing efforts by reducing maintenance overhead and ensuring tests remain stable as applications evolve. Heal.dev is an test automation platform that leverages AgentQL to make this a reality by:

  • Writing reliable end-to-end tests
  • Maintaining your tests with AI-driven insights
  • Managing the entire test infrastructure for you
  • Adding implicit assertions to catch defects before they reach users

By combining AI-powered tools like AgentQL with modern testing platforms, we’re entering an era where stability, maintainability, and efficiency are no longer trade-offs. With the right approach, you can build tests that:

  • Adapt to UI changes without constant updates
  • Reduce flakiness by targeting functionality rather than fragile selectors
  • Scale effortlessly with AI-assisted assertions and infrastructure management

But stability is just the beginning.

True test automation isn’t just about catching bugs—it’s about enabling teams to move faster confidently. By integrating AI-driven solutions, developers can shift their focus from fixing flaky tests to shipping high-quality software at scale.

If unreliable tests are slowing you down, it’s time to stop fighting your test suite—and start healing it. Check out Heal.dev and experience the future of test automation today.