Building skilled intuition for using LLMs

Because there are so many kinds of LLMs, and versions advance insanely fast, it is not easy to get a feel for what kinds of things each LLM is good at and how good it is. I organized my thoughts on what might help improve that feel.

Skilled intuition

If you keep doing something consistently, you often develop intuition, but there are situations in which intuition can be trusted (the area that the NDM school, including Gary Klein, focused on) and situations in which it cannot (the area that the HB school, including Daniel Kahneman, focused on). Even in environments where genuine skilled intuition cannot develop, some people occasionally make successful judgments by chance. These “lucky” individuals mistake their luck for skill and become overconfident. (Reference: Conditions for intuitive expertise: A failure to disagree)

Skilled intuition for using LLMs

What would it mean to develop skilled intuition for using LLMs? It seems important to be able to judge things like these well.

• Knowing in advance what kinds of problems, asked in what ways, an LLM is likely to do well on (or do poorly on)
• While looking at an LLM’s answer (or the output produced by an agent), being good at spotting parts whose truth is questionable or that need additional verification

What would be a good way to build this kind of intuition? It does not seem like simply using them a lot with no particular plan would do it.

Conditions for skilled intuition to develop

According to Conditions for intuitive expertise: A failure to disagree, two things are needed for skilled intuition to develop.

• A high-validity task environment
• Long-term deliberate practice in situations where feedback is rapid and clear

What it means for a task environment to have high validity is this.

“

We describe task environments as “high-validity” if there are stable relationships between objectively identifiable cues and subsequent events or between cues and the outcomes of possible actions. Medicine and firefighting are practiced in environments of fairly high validity. In contrast, outcomes are effectively unpredictable in zero-validity environments. To a good approximation, predictions of the future value of individual stocks and long-term forecasts of political events are made in a zero-validity environment.

We describe task environments as “high-validity” if there are stable relationships between objectively identifiable cues and subsequent events or between cues and the outcomes of possible actions. Medicine and firefighting are practiced in environments of fairly high validity. In contrast, outcomes are effectively unpredictable in zero-validity environments. To a good approximation, predictions of the future value of individual stocks and long-term forecasts of political events are made in a zero-validity environment.

Deliberate practice does not mean “just throwing time at it,” but rather training under conditions like the following.

1. Focused attention on specific weaknesses - Target areas that need improvement rather than practicing what you already do well
2. Immediate feedback - Get information about what works and what does not
3. Activities designed to go beyond current ability - Work just outside your comfort zone
4. A high degree of repetition and refinement - Practice repeatedly with mastery as the goal

What about the situation of using LLMs?

• Validity of the environment: Because LLMs operate probabilistically and often hallucinate, the environment has relatively low validity. Even for the same prompt, they may do well or poorly depending on the case.
• Deliberate practice: (At least I have not) trained in the way defined above.

The reasonable conclusion? (At least I am not) in a situation conducive to the development of skilled intuition for using LLMs, and in such a situation, I cannot expect that excellent skilled intuition will emerge just from blindly burning lots of tokens, diligently trying the latest models, and having them do many things.

What would be good to do

For now, the two things I want to get better at are these.

• Knowing in advance what kinds of problems, asked in what ways, an LLM is likely to do well on (or do poorly on)
• While looking at an LLM’s answer (or the output produced by an agent), being good at spotting parts whose truth is questionable or that need additional verification

First, let’s focus on the first problem.

• Increase the validity of the environment: Present the same prompt to the same model twice in independent sessions, and count it as a success only if both succeed. (The more often the better, but if I were actually going to do this in everyday life, even doing it twice is hard.)
• Deliberate practice: Present two different prompts for the same task, and before seeing the results, predict in what respects each prompt will produce better (or worse) results, then compare those predictions with the actual results.

If I combine those two, I would have to perform one task four times (run two different prompts twice each in independent sessions), and I do not think I would be able to keep doing that consistently.

It would be nice to make a program like this.

• A UI that runs prompts in independent sessions
• A prompt that automatically creates “mutations” when given one prompt
• A prompt that compares multiple responses and summarizes the differences

Meanwhile, the deliberate practice above does not satisfy the condition of being “activities designed to go beyond current ability.” I need to think more about that part.

Just trying it for now

For now, I only thought it through this far and quickly threw something together. (2025-12-02)