Retrospective of agentic coding experiments

From April 2025 through August, I took spare moments to try various experiments related to agentic coding, and since it seemed like about time to do a retrospective now(August 19), I looked back and organized my thoughts.

First experiment

On April 12, 2025, in the first experiment, I tried two ideas.

Create a file called PLAN.md to maintain context, and record tasks and progress there
Separate the planning agent, implementation agent, and refactoring agent, give each distinct instructions, and collaborate with the appropriate agent depending on the stage

I tried to steer the implementation agent toward doing TDD, but it was disappointing that it just would not follow very well. This probably seems related to things like the fact that there are overwhelmingly more articles on the internet explaining TDD incorrectly than explaining it properly, and the fact that many of TDD’s important practices largely take the form of tacit knowledge.

Second experiment

While watching a YouTube conversation in which John Ousterhout criticizes Agile Design and test-driven development, and thinking about various things, I suddenly arrived at the thought that the software engineering I had studied up to now (including agile methodologies) does not fit the AI era.

That is because software engineering explicitly or implicitly assumes that 1) software development is done by humans, 2) labor and time are expensive resources, and 3) human intellectual activity has various special characteristics, but those assumptions no longer hold. (See: Software engineering in the AI era)

Looking back with that thought in mind, the first experiment felt like an attempt to make AI imitate good practices that humans used to carry out as-is (for example, trying hard to get AI to do TDD).

In the second experiment, which I organized around April 25, I tried doing excessive up-front design, which I had previously avoided, and introducing the concept of traceability spoken of in traditional heavyweight methodologies (big-M methodologies).

I expanded PLAN.md, which had previously had almost no formal structure, and devised a half-formalized document (in YAML form) called a BPRD (Business and Product Requirements Document). It is a document in which everything is structurally connected, from mission and vision all the way to business goals, feature lists, and acceptance test specifications. And I created a simple CLI tool that analyzes whether the internal consistency of this document holds, to help the agent.

The reason I made a half-formalized document was to experiment with a division of labor in which LLMs and traditional programs complement each other’s strengths and weaknesses. (Design is beneficially relating elements. —Kent Beck)

LLMs are good at reading the unstructured natural-language portions of a specification and generating code accordingly.
Traditional programs are good at accurately determining whether the structured parts of a specification conform to fixed rules, and at checking whether code generated by an LLM satisfies fixed rules.

Third experiment

I left things alone for quite a while because I was busy with various matters, and then started experimenting again little by little from July. In the third experiment, I extended and refined the second experiment further.

The BPRD from the second experiment includes the meaning of a “requirements document,” but that felt too narrow. What is needed is not a requirements document, but a specification containing the key information for the software project as a whole. I changed the name to s4 (semi-structured software specification) and improved the document format and the CLI.

From early August, it became possible to do so-called eating my own dog food—developing s4 using s4.

While organizing my thoughts about what exactly I was trying to do, I devised the concept of stateless AI-human interaction and reflected it in s4. Thanks to that, it became easier to delegate large tasks that are hard to finish in a single session to a coding agent across multiple sessions, and it also became easy to build things like a task queue for AI agents.

Byproducts

Fragments of thought

I felt that the better I know a field (for example, software engineering), the harder it is to rethink it from the ground up. See: A world where you enter a search query and get the result a month later
There is still a large gap between the degree of trust in code obtained by feeding a specification to an LLM and the degree of trust in a binary obtained by feeding code to a compiler. This gap cannot be eliminated completely (nor would that be desirable), but it does seem necessary to narrow it somewhat further. To do that, it will probably be necessary to move beyond the limitations of specifications written in natural language.