Skip to content

Test Strategy and Convention ​

An agent asked to test an endpoint may write a unit test around the handler while mocking the HTTP layer and database. Such a test checks handler logic, but it cannot show whether a real request reaches the handler or whether persistence works. Name the boundary the acceptance criterion requires before choosing the test type.

The test taxonomy ​

Different test types verify different things at different boundaries. No row in the table below substitutes for another.

A working taxonomy, representative rather than prescriptive:

TypeWhat it verifiesScope
UnitOne component in isolation (logic, edge cases, error paths)Single class or function
SliceOne framework layer with its wiring intact (external services stubbed)e.g. persistence layer, HTTP routing
IntegrationComponents interacting within a module (real dependencies, no stubs)Module pipeline
Interface / ContractAn API boundary or interop point holdsPublic API, Java/Kotlin interop
ArchitecturalStructural rules hold: no forbidden dependencies, no layer violationsModule graph
AcceptanceA user-visible scenario works end to end at the spec levelFeature, golden fixture
E2EThe deployed system behaves as the user expectsCLI to output, browser to API
Visual regressionRendered output matches a stored referenceUI component or full-page render
SmokeThe deployed system is alive and routing correctly after a releaseDeployed entry points
PerformanceLatency or throughput stays within a defined boundLoad profile
ManualIntent is specified in the spec, but automated evidence is not yet written or not practicalAny level

Most projects settle on a subset. The important step is to write the allowed test types into a project document before the agent starts inventing its own defaults.

Sources: Dave Farley, "Modern Software Engineering" (Addison-Wesley, 2021), tests as boundary-specific feedback loops, the basis for matching the test type to what it verifies. The taxonomy rows above are this book's synthesis.

The convention document ​

The convention document defines the types the project uses, the framework that covers each type, where the test files live, and what coverage thresholds apply. A simple table is enough:

markdown
## Test types in use

| Test Type     | Framework          | Location                                   | Level       |
|---------------|--------------------|--------------------------------------------|-------------|
| Unit          | JUnit + Kotest     | src/test/kotlin/**/*Test.kt                | pre-commit  |
| Integration   | JUnit + Kotest     | src/test/kotlin/**/*IntegrationTest.kt     | pre-merge   |
| Contract      | JUnit (Java only)  | src/test/java/**/interop/                  | pre-merge   |
| Architectural | ArchUnit           | src/test/kotlin/**/*ArchTest.kt            | pre-commit  |
| Acceptance    | JUnit + Kotest     | src/test/kotlin/**/*AcceptanceTest.kt      | pre-merge   |
| E2E           | JUnit + Kotest     | src/test/kotlin/**/*E2eTest.kt             | post-deploy |

Type answers what the test verifies. Level answers when it runs. Write both axes down in a file the agent can read, or the agent will make them up from whatever pattern it saw last.

The document belongs with the project's architecture docs, wherever those live. Direct the agent to read it before writing a test. Without that guidance, the agent has to infer the framework, file location, and run stage from the code it happens to inspect.

The decision to adopt a specific convention, and the rationale for each choice, belongs in an ADR. The ADR stays. The convention document evolves.

Sources: Michael Nygard, "Documenting Architecture Decisions" (2011), ADRs as durable records of architectural rationale. The convention-document shape above is this book's workflow rule for making test-layer decisions readable by agents.

Choose cases from the criterion ​

The number of tests follows from the behavior, not a fixed tier. A file-size limit needs an accepted file, a rejected oversized file, and attention to the limit itself. A retry rule may need separate cases for the first failure, the last allowed attempt, and exhaustion. Use equivalence partitions and boundaries to choose cases, then inspect what each test asserts. Tests as Evidence, Not Ritual explains why a count alone is weak evidence.

Sources: ISTQB "Certified Tester Foundation Level Syllabus v4.0.1" (September 15, 2024), equivalence partitioning and boundary value analysis. The example and review rule are this book's application to acceptance criteria.

The AC registry ​

The convention needs one more file, an AC registry that allocates acceptance-criterion IDs so two scenarios never collide on the same identifier. One row per component, updated in the same commit as any new scenario. In the companion repo, the registry lives at tests/ac-registry.md.

The allocation rule, the monotone-numbering discipline, and why a deleted ID leaves a permanent gap are the next chapter's subject: AC IDs and Coverage.

Sources: intent-engineering-for-coding-agents/cli tests/ac-registry.md (ongoing), monotone AC registry layout and non-reuse rule.

The convention tells the agent how to write tests. Review still needs a way to trace each test back to the acceptance criterion it verifies, including after the spec changes.