E2E QA Automation

Problem

QA Tied Up in Repetitive Checks and Tool Switching

When I joined, three POs, including me, shared QA work that took one to two weeks per release. Repeating basic checks left too little time to examine policies and exceptions in new features. Feature dependencies were difficult to understand in full, so even a small change could affect critical functionality.

At the time, test cases (TCs) were maintained in Google Sheets, while tickets for bugs found during QA were created and managed in Jira. Verification meant repeatedly switching between the two tools, updating records in both places, and keeping their content and progress status aligned.

Initial Improvement

Unifying QA Records and Bug Status

Recording a bug in Google Sheets now creates a Jira ticket automatically, and Jira fix status is synchronized back to the sheet. This brings verification items and bug-fix status together for review and follow-up.

Remaining Limitation

Repetitive Checks That Remained After Connecting QA Records

Easier record management still left the same basic features to be checked again in the next release.

I used existing test cases, past bugs, and related workflows to scope recurring basic checks for automation, while assessing new policies and complex exceptions directly. The verification unit became the workflow from contract creation through collection and billing: a working screen alone could not establish that its data carried through correctly to the next task.

Developing the Idea

Automating Repetitive Basic Checks with E2E Tests

I needed to reduce time spent checking basic features repeatedly so I could give more attention to policies and exceptions in new features. I started E2E test automation to verify the same workflows for each release.

Playwright MCP was available by then, allowing me to connect it to Cursor and write test code while interacting with the browser and checking the screens. I chose this approach to build the initial test foundation quickly, and built the test code and execution infrastructure from scratch with Playwright and TypeScript.

Tests create contracts and schedules in the Admin web app, perform collections in the Driver app, then return to Admin to verify completion records and billing. Separate login state by role, project dependencies, and teardown connect setup, execution, verification, and cleanup.

For billing, the tests calculate expected amounts independently from contract terms and task amounts, then compare the subtotal, VAT, and total. In addition to checking the success message, they query the API again to verify the saved state and amounts.

The first tests used existing test-server data, which could change and cause failures even when a feature was working. Each run now creates its own contracts and schedules and records their identifiers for subsequent tests. Cleanup runs after completion or a mid-test failure, reducing the influence of pre-existing data on the results.

Extending Verification

From PR Verification to Developer Fixes

E2E testing reduced repetitive QA, but changes could only be checked after they were merged and deployed to the test server. Once several developers' changes were deployed together, it was also harder to identify which change had affected an existing feature.

I introduced independent verification environments for each PR and attached results to that PR. Developers could compare failures with their own changes, find problems, and make fixes before deployment to the test server.

GitHub Actions builds the Frontend and Backend, and Helm deploys the Frontend, Backend, and database to a k3s cluster I set up internally. Separate namespaces and databases for each PR and run keep their changes and data apart, and Playwright verifies the application in that environment.

Building from a branch name could pick up a new commit during a run and use a different version from the code originally intended for verification. I therefore selected the Frontend and Backend commits at the start of each run and used those commits to build and test throughout that run. Images and database snapshots were also tagged per run so each result could be traced to its code and execution conditions.

Enhanced Monitoring

Tracking Run Progress and Investigating Failures

Without progress updates until the tests finished, it was hard to tell whether a run was taking longer than expected or had stopped. I added CI log updates every 30 seconds showing the platform under test, execution stage, elapsed time, completed test count, and retry status.

When tests failed, I collected Playwright reports, Backend logs, Pod and Kubernetes events, and Helm status to distinguish product defects from problems in the verification environment. I saved the evidence as GitHub Actions artifacts and kept the Pod running until copying finished. Failed environments were also retained temporarily so the cause could be investigated after the run.

I shared per-PR verification results through PR status and comments, and sent results from the test server to Slack. Slack notifications included pass and failure counts by platform, durations, and run links, with failed tests and error details posted in threads. Developers could compare failure points with their changes, fix implementation errors, or update test expectations when behavior had changed intentionally.

Outcomes

An Automation Foundation Built from Scratch, a New Way to Run QA

I built the company's first QA automation infrastructure from scratch (0→1). Test coverage is 88.2%, and basic-feature E2E tests run in about an hour. Reducing repeated manual checks freed time to examine policies, exceptions, and effects on related workflows in new and changed features.

Across regular releases before and after E2E adoption, the number of tickets in scope for QA rose from 712 to 992. Average verification volume also increased from 22.3 to 29.2 tickets per day, about 31%.

As verification volume increased, the post-release bug ratio fell from 7.0% to 3.9%. E2E tests repeatedly checked basic features, while I directly reviewed policies and exceptions in new features. For bugs found during QA, I tracked developers' progress in Google Sheets and Jira and retested the affected behavior after fixes.

Early on, three POs shared QA, which took one to two weeks per release. After building the QA automation infrastructure, I took sole responsibility for release QA and now complete it in about three days on average. E2E tests repeat basic checks, while I use RAG to compare related policies and code and define the verification scope for new and changed features.

For the use of related policies and code in verification, see RAG Knowledge Base.