While migrating performance specs to Playwright, I noticed that I'm getting vastly different numbers for the theme specs while running headfully vs. headlessly. From a local run:

I googled a bit but couldn't find a good read on testing performance headfully vs. headlessly, so I asked GPT. Here's the relevant bit:

(…) Since these (headless) browsers don't spend time loading images or executing JavaScript to render UI elements, they're often faster than traditional browsers.

(…) For measuring Time to First Byte (TTFB), headless testing should suffice, as it's about measuring the time between the client making an HTTP request and receiving the first byte of the response from the server. This measurement shouldn't be impacted by whether the browser is headless or not.

However, if you're measuring other performance metrics that might be affected by the rendering of the page (like First Contentful Paint, Time to Interactive, etc.), then you might want to consider headful testing, as it simulates a more realistic user experience.

The part about TTFB doesn't seem to check out, though, because I'm getting much larger numbers for that metric as well. Anyhow, the part about simulating a more realistic use experience makes sense to me, so I would like to discuss if we should start measuring via a headed browser instance to get more realistic metrics.

I'm curious what your thoughts are on this, @youknowriad, @dmsnell, @oandregal, @kevin940726 🙌

(I'm unsure yet if running headfully in CI is even possible, but I'm currently trying to verify it.)