ANTE Reviewer 2

Reviewer 2 · Original review

Critical Weaknesses

The claims go well beyond what the method actually does. The title, the abstract and Figure 1 all present the agent as foreseeing an edit's effect before it makes the edit. In Figure 1, the belief written before the edit compares two possible fixes. It predicts that changing the function's output would pass the target test but break other code that calls the function, while adding a default value keeps all tests passing, so the agent picks the second fix. The method as defined cannot produce such forecasts. The forecast that execution checks has only two discrete fields: the result of the check run in the current step, and whether the command exits cleanly. An edit step is always forecast as "no verdict" plus the exit code, and the summary gets no training signal in RL. So the forecast written before an edit never says whether the edit will make the tests pass or break other tests. The agent only forecasts a test result later, when it runs the tests, and by then the code has already been changed. For the same reason, ANTE-TTO cannot rank candidate edits by their effect on the tests. The rising hit rate may also partly come from easy checks on scripts the agent writes itself. The theory applies standard bounds under idealized assumptions that do not cover the algorithm as implemented. Calling this an internal world model, and the abstract's stated aim of one that is "omniscient over both time and environment state," goes far beyond this evidence.

Questions For Authors · 1

The paper's key evidence that the world model gets better is the rising forecast hit rate on conclusive steps, that is, checks that return PASS or FAIL (Figure 5(a)). At 32B it goes from 44% to 69% during RL. These checks are tests that the agent chooses to run, including reproduction scripts it writes itself. The protocol also asks the agent to make each script print its own PASS or FAIL verdict (Appendix P). Some of these forecasts are easy. One example is predicting that a bug-reproduction script fails before any fix. Another is predicting that a script re-run with no code change gives the same result as last time. The paper's own example of the belief sharpening during RL (Appendix O.3) is of the first kind: the improved forecast is that the reproduction script fails on the still-unfixed code. BAPO also gives credit to the action at a step whose trusted forecast comes true, so the policy may even learn to run more such easy checks. Figure 18 shows that the share of scored forecasts stays stable, but not what kinds of checks they are. Could you split the hit rate in Figure 5(a) by type of check: checks run right after a code edit, reproduction runs before any fix, and re-runs with no code change in between? Could you also show how the share of each type changes during RL? Only the first type shows whether the agent can foresee the effect of its own edits.

Questions For Authors · 2

ANTE-TTO picks among candidates with a fixed score (Eq. 23). A forecast of PASS gets 3 points, "no verdict" gets 1, FAIL gets 0, and a predicted clean exit adds 1. Under the protocol, edits and views are always forecast as "no verdict", so they score at most 2, while a check forecast to pass scores 3 or 4. So any check predicted to pass beats every edit, and two different edits can only be told apart by their predicted exit code. In other words, the score favours optimistic forecasts: a useful check that is correctly expected to fail ranks below an unverified edit. The paper's own SWE example shows this (Figure 14 and Appendix O.1). At step 7, the drafted edit scored 2. The winning candidate ran the reproduction script and scored 4 because it forecast PASS, based on the wrong belief that the bug was not there. The script in fact failed. Had the forecast been correct (FAIL with a clean exit), this candidate would have scored only 1. The rule would then have kept the unverified edit that the paper says ANTE-TTO avoided. The TTO gains in Table 1 are also small (+1.4, +2.4 and +0.6 points) and have no error bars. At 7B and 32B they are smaller than ANTE's own run-to-run standard error. On the same candidates, how does ANTE-TTO compare with simple rules that ignore the forecast? Examples are picking a test-running candidate whenever there is one, or picking at random. How often is the winning candidate's forecast correct? Please report these with error bars.