Automated Visual Regression Testing With Playwright

Evaluating visible artifacts is usually a highly effective, if fickle, method to automated testing. Playwright makes this appear easy for web sites, however the particulars may take a bit of finessing.

Latest downtime prompted me to scratch an itch that had been plaguing me for some time: The fashion sheet of an internet site I keep has grown just a bit unwieldy as we’ve been including code whereas exploring new options. Now that we have now a greater concept of the necessities, it’s time for inner CSS refactoring to pay down a few of our technical debt, profiting from fashionable CSS options (like utilizing CSS nesting for extra apparent construction). Extra importantly, a cleaner basis ought to make it simpler to introduce that dark mode characteristic we’re sorely missing so we will lastly respect customers’ most well-liked colour scheme.

Nonetheless, being of the apprehensive persuasion, I used to be reluctant to make giant modifications for worry of unwittingly introducing bugs. I wanted one thing to protect towards visible regressions whereas refactoring — besides which means snapshot testing, which is notoriously sluggish and brittle.

On this context, snapshot testing means taking screenshots to determine a dependable baseline towards which we will evaluate future outcomes. As we’ll see, these artifacts are influenced by a mess of things which may not at all times be totally controllable (e.g. timing, variable {hardware} assets, or randomized content material). We even have to take care of state between take a look at runs, i.e. save these screenshots, which complicates the setup and means our take a look at code alone doesn’t totally describe expectations.

Having procrastinated with no extra agreeable resolution revealing itself, I lastly got down to create what I assumed can be a fast spike. In spite of everything, this wouldn’t be a part of the common take a look at suite; only a one-off utility for this explicit refactoring job.

Luckily, I had imprecise recollections of previous analysis and shortly rediscovered Playwright’s built-in visual comparison characteristic. As a result of I attempt to choose dependencies fastidiously, I used to be glad to see that Playwright appears to not depend on many exterior packages.

Setup

The really useful setup with npm init playwright@newest does an honest job, however my minimalist style had me set every thing up from scratch as a substitute. This do-it-yourself method additionally helped me perceive how the completely different items match collectively.

Provided that I anticipate snapshot testing to solely be used on uncommon events, I wished to isolate every thing in a devoted subdirectory, referred to as take a look at/visible; that shall be our working listing from right here on out. We’ll begin with package deal.json to declare our dependencies, including just a few helper scripts (spoiler!) whereas we’re at it:

{
  "scripts":  true"
  ,
  "devDependencies": {
    "@playwright/take a look at": "^1.49.1"
  }
}

Should you don’t need node_modules hidden in some subdirectory but additionally don’t wish to burden the basis mission with this rarely-used dependency, you may resort to manually invoking npm set up --no-save @playwright/take a look at within the root listing when wanted.

With that in place, npm set up downloads Playwright. Afterwards, npx playwright set up downloads a variety of headless browsers. (We’ll use npm right here, however you may want a distinct package deal supervisor and job runner.)

We outline our take a look at atmosphere through playwright.config.js with a few dozen fundamental Playwright settings:

import { defineConfig, units } from "@playwright/take a look at";

let BROWSERS = ["Desktop Firefox", "Desktop Chrome", "Desktop Safari"];
let BASE_URL = "http://localhost:8000";
let SERVER = "cd ../../dist && python3 -m http.server";

let IS_CI = !!course of.env.CI;

export default defineConfig({
  testDir: "./",
  fullyParallel: true,
  forbidOnly: IS_CI,
  retries: 2,
  staff: IS_CI ? 1 : undefined,
  reporter: "html",
  webServer: {
    command: SERVER,
    url: BASE_URL,
    reuseExistingServer: !IS_CI
  },
  use: {
    baseURL: BASE_URL,
    hint: "on-first-retry"
  },
  tasks: BROWSERS.map(ua => ({
    identify: ua.toLowerCase().replaceAll(" ", "-"),
    use: { ...units[ua] }
  }))
});

Right here we anticipate our static web site to already reside inside the root listing’s dist folder and to be served at localhost:8000 (see SERVER; I want Python there as a result of it’s widely available). I’ve included a number of browsers for illustration functions. Nonetheless, we would cut back that quantity to hurry issues up (thus our easy BROWSERS listing, which we then map to Playwright’s extra elaborate tasks information construction). Equally, steady integration is YAGNI for my explicit state of affairs, in order that complete IS_CI dance might be discarded.

Seize and evaluate

Let’s flip to the precise assessments, beginning with a minimal pattern.take a look at.js file:

import { take a look at, anticipate } from "@playwright/take a look at";

take a look at("residence web page", async ({ web page }) => {
  await web page.goto("https://css-tricks.com/");
  await anticipate(web page).toHaveScreenshot();
});

npm take a look at executes this little take a look at suite (based mostly on file-name conventions). The preliminary run at all times fails as a result of it first must create baseline snapshots towards which subsequent runs evaluate their outcomes. Invoking npm take a look at as soon as extra ought to report a passing take a look at.

Altering our website, e.g. by recklessly messing with construct artifacts in dist, ought to make the take a look at fail once more. Such failures will provide numerous choices to match anticipated and precise visuals:

We will additionally examine these baseline snapshots immediately: Playwright creates a folder for screenshots named after the take a look at file (pattern.take a look at.js-snapshots on this case), with file names derived from the respective take a look at’s title (e.g. home-page-desktop-firefox.png).

Producing assessments

Getting again to our authentic motivation, what we would like is a take a look at for each web page. As an alternative of arduously writing and sustaining repetitive assessments, we’ll create a easy net crawler for our web site and have assessments generated robotically; one for every URL we’ve recognized.

Playwright’s global setup allows us to carry out preparatory work earlier than take a look at discovery begins: Decide these URLs and write them to a file. Afterward, we will dynamically generate our assessments at runtime.

Whereas there are different methods to cross information between the setup and test-discovery phases, having a file on disk makes it straightforward to switch the listing of URLs earlier than take a look at runs (e.g. briefly ignoring irrelevant pages).

Website map

Step one is to increase playwright.config.js by inserting globalSetup and exporting two of our configuration values:

export let BROWSERS = ["Desktop Firefox", "Desktop Chrome", "Desktop Safari"];
export let BASE_URL = "http://localhost:8000";

// and so on.

export default defineConfig({
  // and so on.
  globalSetup: require.resolve("./setup.js")
});

Though we’re utilizing ES modules right here, we will nonetheless depend on CommonJS-specific APIs like require.resolve and __dirname. It seems there’s some Babel transpilation occurring within the background, so what’s truly being executed might be CommonJS? Such nuances typically confuse me as a result of it isn’t at all times apparent what’s being executed the place.

We will now reuse these exported values inside a newly created setup.js, which spins up a headless browser to crawl our website (simply because that’s simpler right here than utilizing a separate HTML parser):

import { BASE_URL, BROWSERS } from "./playwright.config.js";
import { createSiteMap, readSiteMap } from "./sitemap.js";
import playwright from "@playwright/take a look at";

export default async perform globalSetup(config) {
  // solely create website map if it does not exist already
  strive {
    readSiteMap();
    return;
  } catch(err) {}

  // launch browser and provoke crawler
  let browser = playwright.units[BROWSERS[0]].defaultBrowserType;
  browser = await playwright[browser].launch();
  let web page = await browser.newPage();
  await createSiteMap(BASE_URL, web page);
  await browser.shut();
}

That is pretty boring glue code; the precise crawling is going on inside sitemap.js:

• createSiteMap determines URLs and writes them to disk.
• readSiteMap merely reads any beforehand created website map from disk. This shall be our basis for dynamically producing assessments. (We’ll see later why this must be synchronous.)

Luckily, the web site in query gives a complete index of all pages, so my crawler solely wants to gather distinctive native URLs from that index web page:

perform extractLocalLinks(baseURL) {
  let urls = new Set();
  let offset = baseURL.size;
  for(let { href } of doc.hyperlinks) {
    if(href.startsWith(baseURL)) {
      let path = href.slice(offset);
      urls.add(path);
    }
  }
  return Array.from(urls);
}

Wrapping that in a extra boring glue code offers us our sitemap.js:

import { readFileSync, writeFileSync } from "node:fs";
import { be part of } from "node:path";

let ENTRY_POINT = "/subjects";
let SITEMAP = be part of(__dirname, "./sitemap.json");

export async perform createSiteMap(baseURL, web page) {
  await web page.goto(baseURL + ENTRY_POINT);
  let urls = await web page.consider(extractLocalLinks, baseURL);
  let information = JSON.stringify(urls, null, 4);
  writeFileSync(SITEMAP, information, { encoding: "utf-8" });
}

export perform readSiteMap() {
  strive {
    var information = readFileSync(SITEMAP, { encoding: "utf-8" });
  } catch(err) {
    if(err.code === "ENOENT") {
      throw new Error("lacking website map");
    }
    throw err;
  }
  return JSON.parse(information);
}

perform extractLocalLinks(baseURL) {
  // and so on.
}

The fascinating bit right here is that extractLocalLinks is evaluated within the browser context — thus we will depend on DOM APIs, notably document.links — whereas the remaining is executed inside the Playwright atmosphere (i.e. Node).

Exams

Now that we have now our listing of URLs, we principally simply want a take a look at file with a easy loop to dynamically generate corresponding assessments:

for(let url of readSiteMap()) {
  take a look at(`web page at ${url}`, async ({ web page }) => {
    await web page.goto(url);
    await anticipate(web page).toHaveScreenshot();
  });
}

That is why readSiteMap needed to be synchronous above: Playwright doesn’t at the moment assist top-level await inside take a look at recordsdata.

In apply, we’ll need higher error reporting for when the positioning map doesn’t exist but. Let’s name our precise take a look at file viz.take a look at.js:

import { readSiteMap } from "./sitemap.js";
import { take a look at, anticipate } from "@playwright/take a look at";

let sitemap = [];
strive {
  sitemap = readSiteMap();
} catch(err) {
  take a look at("website map", ({ web page }) => {
    throw new Error("lacking website map");
  });
}

for(let url of sitemap) {
  take a look at(`web page at ${url}`, async ({ web page }) => {
    await web page.goto(url);
    await anticipate(web page).toHaveScreenshot();
  });
}

Getting right here was a little bit of a journey, however we’re just about completed… until we have now to take care of actuality, which usually takes a bit extra tweaking.

Exceptions

As a result of visible testing is inherently flaky, we typically have to compensate through particular casing. Playwright lets us inject customized CSS, which is usually the simplest and simplest method. Tweaking viz.take a look at.js…

// and so on.
import { be part of } from "node:path";

let OPTIONS = {
  stylePath: be part of(__dirname, "./viz.tweaks.css")
};

// and so on.
  await anticipate(web page).toHaveScreenshot(OPTIONS);
// and so on.

… permits us to outline exceptions in viz.tweaks.css:

/* suppress state */
important a:visited {
  colour: var(--color-link);
}

/* suppress randomness */
iframe[src$="/articles/signals-reactivity/demo.html"] {
  visibility: hidden;
}

/* suppress flakiness */
physique:has(h1 a[href="https://css-tricks.com/wip/unicode-symbols/"]) {
  important tbody > tr:last-child > td:first-child {
    font-size: 0;
    visibility: hidden;
  }
}

:has() strikes again!

Web page vs. viewport

At this level, every thing appeared hunky-dory to me, till I noticed that my assessments didn’t truly fail after I had modified some styling. That’s not good! What I hadn’t taken under consideration is that .toHaveScreenshot solely captures the viewport reasonably than the whole web page. We will rectify that by additional extending playwright.config.js.

export let WIDTH = 800;
export let HEIGHT = WIDTH;

// and so on.

  tasks: BROWSERS.map(ua => ({
    identify: ua.toLowerCase().replaceAll(" ", "-"),
    use: {
      ...units[ua],
      viewport: {
        width: WIDTH,
        peak: HEIGHT
      }
    }
  }))

…after which by adjusting viz.take a look at.js‘s test-generating loop:

import { WIDTH, HEIGHT } from "./playwright.config.js";

// and so on.

for(let url of sitemap) {
  take a look at(`web page at ${url}`, async ({ web page }) => {
    checkSnapshot(url, web page);
  });
}

async perform checkSnapshot(url, web page) {
  // decide web page peak with default viewport
  await web page.setViewportSize({
    width: WIDTH,
    peak: HEIGHT
  });
  await web page.goto(url);
  await web page.waitForLoadState("networkidle");
  let peak = await web page.consider(getFullHeight);

  // resize viewport for earlier than snapshotting
  await web page.setViewportSize({
    width: WIDTH,
    peak: Math.ceil(peak)
  });
  await web page.waitForLoadState("networkidle");
  await anticipate(web page).toHaveScreenshot(OPTIONS);
}

perform getFullHeight() {
  return doc.documentElement.getBoundingClientRect().peak;
}

Word that we’ve additionally launched a waiting condition, holding till there’s no community site visitors for some time in a crude try and account for stuff like lazy-loading photos.

Bear in mind that capturing the whole web page is extra resource-intensive and doesn’t at all times work reliably: You might need to take care of layout shifts or run into timeouts for lengthy or asset-heavy pages. In different phrases: This dangers exacerbating flakiness.

Conclusion

A lot for that fast spike. Whereas it took extra effort than anticipated (I consider that’s referred to as “software program growth”), this may truly remedy my authentic drawback now (not a typical characteristic of software program lately). In fact, shaving this yak nonetheless leaves me itchy, as I’ve but to do the precise work of scratching CSS with out breaking something. Then comes the actual problem: Retrofitting darkish mode to an current web site. I simply may want extra downtime.