QA and AI
What QA Does Differently When AI Writes More Code
When AI writes more code, QA's job shifts from confirming that code passes tests to proving that features work for the business. QA moves earlier, into acceptance criteria and risk review before implementation, and later, into production monitoring, while AI-generated tests take over much of the mechanical coverage.
Developer output is up. By some measures it is dramatically up: a 2024 McKinsey study reported a 76% increase in code output among teams using AI coding assistants. Prototypes that took weeks now take days. Features that took sprints now take sessions. The stream of code flowing toward production has widened substantially.
QA headcount has not kept pace. Most organizations that adopted AI coding assistants did not expand their testing teams to match. The implicit assumption was that faster code production would somehow make testing easier, or that AI-generated code would be inherently more reliable, or that the same testing processes would absorb the extra volume. None of these assumptions has held up.
The reality is more demanding. AI can make code pass tests. QA still has to prove the feature works for the business. That distinction between technical correctness and business correctness has always been the core of quality assurance. AI makes it sharper, more urgent and harder to ignore.
Three Types of Correctness
To see why QA becomes more important with AI, it helps to separate three types of correctness every feature has to satisfy.
Code correctness means the software does what the code says it does. Functions return expected values. API endpoints respond with the right status codes. Data transformations produce valid output. Unit and integration tests verify code correctness. AI coding assistants are generally good at it: the code they generate compiles, runs and passes the tests written to verify it.
Product correctness means the software does what the product specification says it should do. The user flow matches the design. The feature behaves as the acceptance criteria describe. Edge cases identified during planning are handled. Product correctness requires understanding the intended behavior, not just the implemented behavior. AI is less reliable here. It implements what it infers from the prompt and context, which can diverge from the product manager's intent in subtle but important ways.
Business correctness means the software solves the business problem it was built for. The feature works for real users in real workflows with real data. It fits the business processes it has to support. It handles the exceptions that come up in practice, not just the ones anticipated in planning. Business correctness requires domain expertise and context that neither AI tools nor automated tests have.
QA's mandate spans all three levels, but the emphasis shifts with AI. Code correctness is increasingly handled by automated tests, many of them AI-generated. Product correctness becomes a higher-stakes checkpoint, because AI-generated implementations are more likely to drift from specifications without anyone noticing. Business correctness becomes the area where human QA professionals provide value no tool replaces.
Moving Earlier: The Shift Left That Actually Matters
"Shift left" has been a QA catchphrase for a decade, but AI gives it new urgency and a more specific meaning. QA should move earlier in the process, not just by writing tests before code, but by shaping what gets built before implementation begins.
In an AI-assisted workflow, implementation is so compressed that traditional QA timing no longer works. If a developer builds a feature in a day and QA starts testing the next day, QA is evaluating an artifact produced in a fraction of the time they will spend evaluating it. The ratio is inverted: more time testing than building. That is not wrong in itself, since some features warrant extensive testing, but it suggests the testing effort would do more earlier, when it can prevent defects instead of discovering them.
In practice, QA engineers should help define acceptance criteria before development begins. They should review the business analyst's specifications for testability, asking not just "is this clear?" but "can I write a test that verifies this?" They should identify risk areas that merit extra attention: complex business logic, security-sensitive operations, performance-critical paths, integration points with external systems. And they should define regression paths, the existing functionality that must keep working when new code ships.
The Azumo SDLC builds this into the feature workflow, which runs clarify, spec, plan, build, validate, review, deliver. The agent stops for a person's approval after the spec and after the plan. Those approval points are where QA asks whether each acceptance criterion can be tested, before any code exists.
This pre-implementation work does not replace post-implementation testing. It makes post-implementation testing more effective. A QA engineer who helped define the acceptance criteria can test against them precisely. A QA engineer who first sees the feature when it reaches staging is reconstructing context that should have been set earlier. Test-first development with AI covers how to write those tests before the code.
AI-Generated Tests: Useful but Insufficient
AI coding assistants can generate test cases, and they do it well. They often produce comprehensive unit test suites that cover the common paths, check obvious edge cases and follow the testing conventions of the framework. That is useful. It reduces the mechanical work of test authoring and can improve coverage.
But AI-generated tests share a fundamental limitation: they test what the code does, not what the code should do. An AI given a function and asked to write tests will verify the function's current behavior. If the function is wrong, because it implements a business rule incorrectly, handles an edge case differently than the business expects, or produces subtly inaccurate results, the AI-generated tests will confirm the incorrect behavior as correct. The tests pass. The code is wrong.
Human QA professionals write tests from a different direction. They start with the business requirement and work backward to the test, instead of starting with the code and working forward. They test the scenarios that matter to the business, not just the scenarios the code happens to handle. They test the boundary between what the system does and what the user expects, the gap where defects live.
The effective approach is to let AI generate the bulk of mechanical coverage, such as unit tests, integration tests for standard paths and data validation checks, and then layer human-authored tests on top for business-critical scenarios, complex edge cases and cross-system interactions. AI handles the volume. People handle the judgment.
The UAT Imperative
User acceptance testing has always been important. With AI-generated code, it becomes indispensable, for a reason specific to how AI writes software.
AI coding assistants produce code optimized for the criteria they can observe: it compiles, it follows patterns, it handles the cases described in the prompt. They are less effective at the harder-to-specify fit of the software for its actual use. Does the form flow feel natural to the business user who will use it fifty times a day? Does the error message make sense to someone who doesn't know the system's internals? Does the report format match the way the finance team actually reads reports?
Automated tests cannot answer these questions. Neither can developers reviewing code. Only the people who will use the software to do their jobs can. UAT is the process that surfaces the gap between "technically correct" and "actually useful."
Organizations using AI coding assistants should invest more in UAT, not less. Higher code volume means more features reaching UAT in any given period. Higher risk of business-logic misalignment means each UAT session is more likely to surface meaningful issues. And a compressed development timeline means UAT feedback has to flow back to development faster, so corrections land before the next batch of features arrives.
Vulnerability and the QA Imperative
A growing body of evidence suggests AI-generated code carries more vulnerabilities than human-written code. A Stanford study found that developers using AI assistants produced code with more security vulnerabilities than those coding without assistance. Research from Snyk and other security firms has reported the same pattern: AI-generated code is more likely to contain injection vulnerabilities, improper input validation, insecure default configurations and hardcoded credentials.
This is not because AI is inherently insecure. AI learns from the vast body of publicly available code, which contains enormous quantities of insecure patterns. Asked to generate a login function, an AI might produce code that works perfectly but stores passwords in plain text, because it has seen many examples of exactly that. The AI does not distinguish between code that works and code that is secure. It optimizes for plausibility, not safety.
For QA, this means security testing is no longer a specialized activity a dedicated team performs at the end of the release cycle. It has to run throughout testing. Every test plan for AI-generated code should include basic security checks: input validation, authentication and authorization, data handling, error message content (does it leak internal information?), and dependency security. QA engineers don't need to become penetration testers, but they need enough security awareness to catch the common vulnerabilities AI tools introduce. The chapter on security for AI-assisted development goes further.
Fifty-eight percent of enterprises surveyed by Gartner in late 2024 reported that they were actively upskilling their QA teams in AI-related tools and techniques. That is the right instinct. The question is whether the upskilling goes far enough: whether it covers only the tooling (how to use AI for test generation) or also the judgment (how to test AI-generated code effectively).
Shift Right: Observability as a Testing Strategy
The companion to shift-left is shift-right: extending QA's attention past the release boundary into production. This is not new, but AI makes it more important.
AI-generated code can introduce subtle behavioral changes that are hard to detect before production. A function that behaves correctly under test conditions may behave differently under production load, with production data or in production edge cases. The gap between test and production environments is always a risk. AI-generated code may widen it, because the AI's training data includes patterns that work in isolation but fail at scale.
Shift-right QA means investing in observability: logging, monitoring, alerting and anomaly detection that surface production issues quickly. It means defining quality metrics that can be measured in production, such as error rates, latency distributions, user journey completion rates and business process success rates. And it means feedback loops that route production quality data back to the development team, so patterns of AI-generated defects can be identified and fixed at the source.
Shift-left and shift-right are complements, not alternatives. Shift-left catches the defects that can be caught before deployment. Shift-right catches the ones that only show up after. Together they form a quality strategy that matches the pace and risk profile of AI-assisted development.
The Evolving QA Skill Set
The QA role is changing, not shrinking. The mechanical parts of testing, such as writing repetitive test cases, executing manual regression suites and maintaining test data, are increasingly automated, often with AI. That frees QA professionals for the work that requires human judgment: evaluating business correctness, designing test strategies, assessing risk and interpreting results.
The QA professional who does well in an AI-assisted team combines technical testing skills with business domain expertise. They understand not just how to test software but why specific behaviors matter to the business. They can judge whether an AI-generated implementation matches business intent, not just technical specifications. They connect the development team's output to the business's expectations, and that connection matters more as code volume grows.
QA teams should invest in three areas:
- AI tool proficiency: using AI for test generation, test data creation and test analysis.
- Business domain depth: the expertise to judge whether software actually solves the business problem.
- Security awareness: enough knowledge of common vulnerability patterns to catch the issues AI tools frequently introduce.
How Azumo Runs QA on AI-Assisted Work
As a custom software development company, we put QA engineers inside the delivery team for the whole lifecycle, not at a gate at the end. On AI-assisted engagements they help define testable acceptance criteria before development begins, validate the business correctness automated tests cannot reach, and monitor production behavior after release. The Azumo SDLC backs this up at review:
- Nothing passes review without evidence. Separate review checklists cover correctness, security, test adequacy and UI/UX.
- Work missing a test goes back to build.
- Teams can add their own steps and checks, such as an accessibility check, without changing the shared kit.