Agentic coding experiment 2

A revised approach based on thoughts I wrote down in Software Engineering in the Age of AI. These are thoughts I mulled over on April 25, 2025. (The next experiment is Agentic coding experiment 3)

What to improve from the previous experiment

In Agentic coding experiment 1, I tried to have the agent mimic as closely as possible the way humans had been working. For example, instructing it to do TDD with relatively short steps.

But according to the small realization(?) I wrote about in Software Engineering in the Age of AI, this approach does not seem appropriate. That is because quite a few implicit/explicit premises of software engineering like the following need to change considerably.

Software development is done by humans.
Labor and time are expensive resources.
Human intellectual activity has various special characteristics.

But now software development is done jointly by AI agents and humans, the share of work done by AI agents will gradually increase, development cost and time required will decrease accordingly, and AI agents have intellectual(?) characteristics quite different from those of humans.

Based on these thoughts, I came up with a process that departs a little from Agile methodologies and partially borrows from traditional heavyweight methodologies (Big M methodologies).

If machines do the design and coding, then compared to before, it is okay to do relatively more up-front design. Because the cost of change is low.
When requirements change, traceability is important so that the design and code can be revised accordingly. We need an environment that supports AI agents so they can easily manage traceability and do round-trip engineering.

The moment you open a new chat window, the context disappears, so it is like an existing developer quitting and a new developer joining. Therefore, it is important to have good documentation so the project’s context can be understood. It would be good to further develop the existing PLAN.md.

Managing formalized planning documents through collaboration between LLMs and traditional code

Define a formalized planning document (BPRD; Business and Product Requirements Document) like the one below. It should be easy for humans to read and edit, while at the same time having high machine readability.

overview:
  title: 간단한 플래시카드 모바일 웹
  summary: >
	사용자가 어휘 학습이나 다른 주제를 위해 사용자 정의 플래시카드 세트를
	생성, 구성 및 학습할 수 있는 간단한 모바일 웹사이트입니다.
  mission: >
    사용자가 이동 중에도 자신의 플래시카드를 생성하고 복습할 수 있는
	가장 간단한 도구를 제공합니다.
  vision: >
    가장 사용자 친화적인 모바일 플래시카드 도구가 되는 것입니다.
business_objectives:
  - id: BO-001
	description: >
	  사용자가 사용자 정의 콘텐츠로 자신만의 플래시카드 세트를 생성하고
	  관리할 수 있도록 합니다.
  - id: BO-002
	description: >
	  사용자가 카드의 앞면과 뒷면을 보고 넘기면서 플래시카드를 학습할 수 있는
	  간단한 인터페이스를 제공합니다.
functional_requirements:
  - id: FR-001
	title: 사용자 회원가입
	covers:
	  - BO-003
	prerequisites: []
	description: >
	  사용자는 이메일 주소와 비밀번호를 사용하여 새 계정을 만들 수 있습니다.
  - id: FR-002
	title: 사용자 로그인
	covers:
	  - BO-003
	prerequisites:
	  - FR-001
	description: >
	  등록된 사용자는 이메일 주소와 비밀번호를 사용하여 로그인할 수 있습니다.
  - id: FR-003
	title: 사용자 로그아웃
	covers:
	  - BO-003
	prerequisites:
	  - FR-002
	description: >
	  로그인한 사용자는 자신의 계정에서 로그아웃할 수 있습니다.
  - id: FR-004
	title: 플래시카드 세트 생성
	covers:
	  - BO-001
	prerequisites:
	  - FR-002
	description: >
	  사용자는 새로운 빈 플래시카드 세트를 만들고 이름을 지정할 수 있습니다.
  - id: FR-005
	title: 플래시카드 추가
	covers:
	  - BO-001
	prerequisites:
	  - FR-004
	  - FR-002
	description: >
	  사용자는 기존 세트에 새 플래시카드를 추가하고 카드의 앞면과 뒷면에
	  대한 텍스트를 제공할 수 있습니다.
  - id: FR-006
	title: 세트 및 카드 관리
	covers:
	  - BO-001
	prerequisites:
	  - FR-004
	  - FR-005
	  - FR-002
	description: >
	  사용자는 자신이 만든 세트 목록을 볼 수 있습니다. 세트를 선택하면 해당
	  세트 내의 카드 목록을 볼 수 있으며, 개별 카드나 전체 세트를 편집하거나
	  삭제할 수 있는 옵션이 있습니다.
  - id: FR-007
	title: 플래시카드 학습 모드 (뒤집기)
	covers:
	  - BO-002
	prerequisites:
	  - FR-006
	  - FR-002
	description: >
	  사용자는 세트를 선택하고 카드가 하나씩 표시되는 학습 모드로 들어갈
	  수 있습니다. 카드를 탭하면 뒷면이 나타나고, 사용자는 세트의 카드를
	  탐색할 수 있습니다.
acceptance_tests:
  - id: AT-001
	covers: FR-001
	description: >
	  사용자가 회원가입 페이지에 있을 때, 유효한 이메일 "test@example.com"과
	  비밀번호 "password123"을 입력하고 "회원가입"을 탭하면, 로그인되어 메인
	  페이지로 리디렉션됩니다.
  - id: AT-003
	covers: FR-003
	description: >
	  사용자가 로그인되어 있을 때, "로그아웃" 버튼을 탭하면, 로그아웃되고
	  메인 페이지로 리디렉션됩니다.
  - id: AT-004
	covers: FR-004
	description: >
	  사용자가 메인 페이지에 있을 때, "새 세트 만들기"를 탭하고 이름
	  "내 단어장"을 입력하면, "내 단어장"이라는 이름의 세트가 세트 목록에
	  나타납니다.
  - id: AT-005
	covers: FR-005
	description: >
	  '사용자가 "내 단어장" 세트를 만들고 보고 있을 때, "카드 추가"를
	  탭하고 "단어: Hello", "정의: 안녕하세요"를 입력하면, "Hello/안녕하세요"
	  카드가 "내 단어장" 세트에 추가됩니다.'
  - id: AT-006
	covers: FR-006
	description: >
	  사용자가 메인 페이지에 있을 때, 세트 목록에서 "내 단어장"을 보고
	  탭하면, "Hello/안녕하세요"를 포함하여 "내 단어장"의 카드를 보여주는
	  페이지로 이동합니다.
  - id: AT-007
	covers: FR-006
	description: >
	  '사용자가 "내 단어장"의 카드를 보고 있고 "Hello/안녕하세요"를 편집하기
	  위해 탭했을 때, "정의: 안녕하세요"를 "정의: 안녕"으로 변경하면, 카드가
	  "Hello/안녕"으로 업데이트됩니다.'
  - id: AT-008
	covers: FR-006
	description: >
	  사용자가 "내 단어장"의 카드를 보고 있고 "Hello/안녕하세요"를 삭제하기
	  위해 탭했을 때, 삭제를 확인하면 "Hello/안녕하세요" 카드가 "내 단어장"
	  세트에서 제거됩니다.
  - id: AT-009
	covers: FR-007
	description: >
	  사용자가 "내 단어장"의 카드를 보고 있을 때, "학습" 버튼을 탭하면,
	  첫 번째 카드의 앞면("Hello")이 표시됩니다.
  - id: AT-010
	covers: FR-007
	description: >
	  사용자가 카드의 앞면("Hello")을 보고 있을 때, 카드를 탭하면, 카드의
	  뒷면("안녕하세요")이 표시됩니다.
  - id: AT-011
	covers: FR-007
	description: >
	  사용자가 카드의 뒷면("안녕하세요")을 보고 있을 때, 카드를 다시 탭하면,
	  카드의 앞면("Hello")이 다시 표시됩니다.
  - id: AT-012
	covers: FR-007
	description: >
	  사용자가 카드를 보고 있을 때, 다음 카드로 이동하면(예: 스와이프 또는
	  버튼), 세트의 다음 카드 앞면이 표시됩니다.

Create a CLI program that checks the parts of the planning document that a machine can understand (in the traditional sense). This program checks whether there are logical problems in the graph that starts from business objectives (BO), passes through functional requirements (FR), and leads to acceptance tests (AT).

For example, something like this:

$ deva check < BPRD.yaml
UNKNOWN_BO_ID: FR-001 references BO-003.
UNKNOWN_BO_ID: FR-002 references BO-003.
UNKNOWN_BO_ID: FR-003 references BO-003.
UNCOVERED_FR: FR-002 is not covered by any acceptance test.

If you register the BPRD document format and the usage of the tool called deva in Cursor’s Rule file, the agent will create defect-free requirements documents well and also revise them appropriately. For example, if you tell it, “Please add this or that feature,” it adds the functional requirements and related acceptance tests, runs the CLI program on its own, and revises things appropriately until all problems are caught. It is a way for the two systems to complement each other’s weaknesses—LLM understands the text, and the CLI checks for logical defects (a kind of neuro-symbolic system).

Coding while maintaining context based on formalized planning documents

With the BPRD.yaml document and appropriate Rule files in place, if you say, “Implement FR-001,” it implements it on its own and writes the necessary acceptance tests. If you run a specific command (for example, npm run check-all), it runs the linter, formatter, type checker, unit testing, and acceptance testing all at once and reports the results, so the agent works by itself until all issues are resolved. (It mostly succeeds, but occasionally fails.)

For example, you would register Rule files like these:

---
description:
globs: e2e/**/*.test.ts
alwaysApply: false
---

Acceptance tests should be in `e2e` directory and named according to [BPRD.yaml](mdc:BPRD.yaml), e.g.: `AT-001.test.ts`

Whenever you add, modify, or remove acceptance tests, always run `npm run deva` and fix any problems it reports.

---
description:
globs:
alwaysApply: true
---
This project uses DEVA - The Software Development Agent.

## BPRD

[BPRD.yaml](mdc:BPRD.yaml) contains Business and Product Requirements Document (BPRD). This document is the single source of truth for the entire project. It provides traceability between the business objectives (`BO-nnn`), functional requirements (`FR-nnn`), acceptance tests (`AT-nnn`), and actual implementations. Whatever you do, always read this document first.

Whenever you modified `BPRD.yaml`, always run `npm run bprd`. Fix any problems it reports.

## Getting feedback

Whenever you modified code, always run `npm run deva` to run all the checks and tests. Fix any problems it reports.

It seems like it would be good to do more things like the following going forward.

Track whether particular functional requirements have been implemented. Add a CLI tool that analyzes and tells the agent whether business objectives (BO), functional requirements (FR), acceptance test specifications (AT), and acceptance test code are linked to one another, using methods such as leaving specially formatted comments in the source code.
Define machine-readable design documents based on BPRD documents. Add a CLI tool that analyzes consistency between the BPRD, design documents, and code.
Have the agent write unit tests and measure test coverage so that it satisfies a certain standard. Since humans will not be the ones doing the work anyway, also try experiments measuring against standards beyond branch coverage.
Measure complexity such as cognitive complexity. (Even if AI does the coding, complexity management is still important. For details, see the “Code Quality” section of Software Engineering in the Age of AI.)
Formalization using Alloy, etc. Perhaps semi-automatically. For example: if the specification is written in natural language, AI creates an Alloy model. It then shows several examples for parts where there is ambiguity, asks the user which one matches the intent, and through that process refines the spec more precisely. Reference: Specifications Written in Natural Language
Find and delete unused code. For example, if you use TypeScript, consider something like knip.dev

For the following next experiment, see Agentic coding experiment 3.

References

v0.dev/docs/prd-design - Generating V0 PRD design documents