Q: Management sets a target of "80% code coverage". Your response?

I avoid opening with "that's the wrong metric" — it triggers defensiveness. Instead I ask about the goal first: why is this target being set? There's usually a real and entirely legitimate worry behind it — "too many bugs reach production" or "we don't trust our tests". That's the actual problem; coverage is just a chosen but ill-fitting indicator for it. Then I explain the flaw with a concrete example: an assertion-less test still produces coverage — it executes code, verifies nothing and lifts the number. So once it becomes a target, people find the cheapest way to move it, coverage hits 80% and quality stays exactly where it was (Goodhart's law: when a measure becomes a target, it ceases to be a good measure). What I propose instead: (1) an outcome metric — escaped defects, since that's the actual worry; (2) risk-based coverage — "how many of the 20 critical flows are automated", which is far more meaningful and trackable; (3) keeping coverage as a tool, not a target — reading the coverage report to spot areas no test touches is genuinely useful; turning it into a KPI is not. If management insists, a compromise: coverage is tracked but never gates; the gates sit on escaped defects and flaky rate. This approach — acknowledging the worry behind the request and offering a better alternative — is scored far higher in interviews than the metric discussion itself.

Q: Nobody on the team looks at test results. How would you change that?

First I try to understand why they don't look, because the fix differs completely by cause. It's usually one of four, each with its own remedy. (1) No trust in the results (high flaky rate) — the most common cause by far. Fix: attack the flakiness, quarantine + SLA, get the rate under 1%. Until that's fixed everything else is pointless — no report design restores trust. (2) Results aren't reachable — you must download an artifact, open it, hunt around. Fix: the report link directly in the PR (inline annotations via the GitHub reporter, the HTML report on static hosting), the failing test visible immediately. (3) No ownership — "everyone's job" is nobody's job. Fix: a named owner for red (the feature team or a rota), automatic notification to that channel when the pipeline breaks. (4) The tests provide no value — they never catch a real bug, they only break themselves. This is the most painful case and demands the most honest answer: the tests themselves need rethinking. The structural change: make tests a required check (branch protection) — but only after (1) is solved, otherwise the team learns to bypass the gate and things get worse. The closing point matters: this looks like a technical problem but is really a problem of trust and ownership. Framing it that way is what makes the difference in an interview.

Q: When do you choose Allure over Playwright's built-in HTML report?

My default position: the built-in HTML report is enough for most projects — zero configuration, interactive, links straight into traces, no extra infrastructure. On a new project I wouldn't propose starting with Allure; it would be a tool looking for a problem. Concrete needs that justify Allure: (1) history and trends — seeing one test's behaviour across the last 30 runs, very useful for tracking flakiness systematically; (2) test case ↔ requirement links — in regulated sectors (banking, insurance, healthcare) audits ask you to show "this requirement is covered by this test"; (3) a non-technical audience — a report the business side reads itself, with steps, severity and attachments; (4) many suites/teams — one centralised view. I state the cost too, because listing only benefits is a weak answer on tool choice: Allure needs a separate generation step, somewhere to host results, and ongoing maintenance; annotations get added in the code; sharding and merging require extra configuration. So the decision is simple: if none of those four needs exist, keep the HTML report. When one does appear, I name that specific need and migrate — "it looks more professional" isn't a reason.

Question 1

Which metrics do you use to measure automation success?

Accepted Answer

I don't open with a list of metrics — I set the frame first: a metric must lead to a decision, otherwise it's just a number that looks good on a slide. My three core metrics: (1) Escaped defects — bugs reaching production. The most important, because it answers the suite's primary purpose directly. For each miss I run a retrospective question: why didn't we catch it? The answer produces a new test, a new approach, or an explicit "we accept this risk" decision. (2) Flaky rate — target <1%. This is the measure of trust in the suite's signal. Above 5% the team stops looking at red and the whole investment zeroes out. (3) PR feedback time — if tests block developer flow, people find ways around them. I also track the reduction in manual regression effort — the most comprehensible number for explaining value to management ("2 days → 4 hours"). And I always say what I deliberately DON'T measure: test count and code coverage percentage. Coverage is useful only when hunting for gaps; as a target it makes people write assertion-less tests to lift the number. That last part strengthens the answer considerably, because it shows critical thinking.

Question 2

Management sets a target of "80% code coverage". Your response?

Accepted Answer

I avoid opening with "that's the wrong metric" — it triggers defensiveness. Instead I ask about the goal first: why is this target being set? There's usually a real and entirely legitimate worry behind it — "too many bugs reach production" or "we don't trust our tests". That's the actual problem; coverage is just a chosen but ill-fitting indicator for it. Then I explain the flaw with a concrete example: an assertion-less test still produces coverage — it executes code, verifies nothing and lifts the number. So once it becomes a target, people find the cheapest way to move it, coverage hits 80% and quality stays exactly where it was (Goodhart's law: when a measure becomes a target, it ceases to be a good measure). What I propose instead: (1) an outcome metric — escaped defects, since that's the actual worry; (2) risk-based coverage — "how many of the 20 critical flows are automated", which is far more meaningful and trackable; (3) keeping coverage as a tool, not a target — reading the coverage report to spot areas no test touches is genuinely useful; turning it into a KPI is not. If management insists, a compromise: coverage is tracked but never gates; the gates sit on escaped defects and flaky rate. This approach — acknowledging the worry behind the request and offering a better alternative — is scored far higher in interviews than the metric discussion itself.

Question 3

Nobody on the team looks at test results. How would you change that?

Accepted Answer

First I try to understand why they don't look, because the fix differs completely by cause. It's usually one of four, each with its own remedy. (1) No trust in the results (high flaky rate) — the most common cause by far. Fix: attack the flakiness, quarantine + SLA, get the rate under 1%. Until that's fixed everything else is pointless — no report design restores trust. (2) Results aren't reachable — you must download an artifact, open it, hunt around. Fix: the report link directly in the PR (inline annotations via the GitHub reporter, the HTML report on static hosting), the failing test visible immediately. (3) No ownership — "everyone's job" is nobody's job. Fix: a named owner for red (the feature team or a rota), automatic notification to that channel when the pipeline breaks. (4) The tests provide no value — they never catch a real bug, they only break themselves. This is the most painful case and demands the most honest answer: the tests themselves need rethinking. The structural change: make tests a required check (branch protection) — but only after (1) is solved, otherwise the team learns to bypass the gate and things get worse. The closing point matters: this looks like a technical problem but is really a problem of trust and ownership. Framing it that way is what makes the difference in an interview.

Question 4

When do you choose Allure over Playwright's built-in HTML report?

Accepted Answer

My default position: the built-in HTML report is enough for most projects — zero configuration, interactive, links straight into traces, no extra infrastructure. On a new project I wouldn't propose starting with Allure; it would be a tool looking for a problem. Concrete needs that justify Allure: (1) history and trends — seeing one test's behaviour across the last 30 runs, very useful for tracking flakiness systematically; (2) test case ↔ requirement links — in regulated sectors (banking, insurance, healthcare) audits ask you to show "this requirement is covered by this test"; (3) a non-technical audience — a report the business side reads itself, with steps, severity and attachments; (4) many suites/teams — one centralised view. I state the cost too, because listing only benefits is a weak answer on tool choice: Allure needs a separate generation step, somewhere to host results, and ongoing maintenance; annotations get added in the code; sharding and merging require extra configuration. So the decision is simple: if none of those four needs exist, keep the HTML report. When one does appear, I name that specific need and migrate — "it looks more professional" isn't a reason.

Metric	Which question it answers	Which decision it drives
Escaped defects	Is the suite doing its primary job?	A new test/approach per missed bug; exposing coverage gaps
Flaky rate	Can a red result be trusted?	Allocating sprint time to flakiness above 1%
PR feedback time	Are tests blocking developer flow?	Tiering the suite, sharding, moving work to the API
Test count	— (nothing)	Nothing; growth is neither good nor bad news
Code coverage %	How much of the code gets executed?	Useful only for spotting GAPS; harmful when set as a target

Reporting and metrics

📚 Sources and documentation