The Needle In a Haystack Test

The Needle In a Haystack Test, introduced by Greg Kamradt

Designed to measure an LLM’s ability to retrieve a specific piece of information (the “needle”) embedded within a large and often irrelevant block of text (the “haystack”). The test systematically varies the depth at which the “needle” is placed within the context window and the overall size of the “haystack” to assess the model’s performance under different conditions.

youtube.com/watch?v=KwRRuiCCdmc

See also

• AbsenceBench: The opposit of NIAH test.