Scoring the Envoy Forecasts: Individual Calibration and Institutional Aggregation in the Final Assessments Before the Doomsday Battle

1. Introduction

Were the analysts who judged the approaching probe a peaceful envoy habitually overconfident? The post-battle inquiries say so (Orlind, 18 DE), and they describe the catastrophic misreading as a calibration failure of the forecasters. On that account the analysts of the UN Planetary Defense Council (PDC) and of Fleet International were too sure that the probe was a peaceful envoy, and the Combined Fleet went to meet it in the expectation of a gesture because the people who judged such questions had judged them too confidently. This account is wrong. The records supply the reason. A forecaster is miscalibrated when stated probabilities depart from observed frequencies over many judgements, and one outcome (the probe was not an envoy) cannot show that, since a calibrated forecaster who rated an envoy at 0.6 and an overconfident one who rated it at 0.6 for poor reasons are equally contradicted by it.

Both obvious remedies cost something. Scoring only the envoy question treats one outcome as a test of a habit, so it risks blaming analysts for an event that they rated unlikely and excusing careless ones who happened to rate it so. Scoring the forecasters on everything else they wrote avoids that risk, but on the other hand it assumes that skill with trajectories and arrival windows carries over to judgements of intent, and that carry-over is the point in dispute. We place our design between the two. We score each forecaster on the many sub-forecasts (e.g., arrival windows, closing-speed bands, changes of heading) that resolved before the battle, we score the envoy question separately with a Brier score, and we model the step from working-level assessments to consolidated ones as a stage with its own data.

Our sources are declassified assessment texts written in the final years before the battle, and every one of them predates the engagement that it concerns. This study is written in Deterrence Era, Year 58.

The texts were drafted while sophon interference was unsuppressed, so each was composed with two readers in mind, the Council and the sophons (Corvane, 49 DE). That fact limits what a stated probability can mean. We return to it in Section 5. Section 2 describes the coding and the scoring, Section 3 reports both stages, and Sections 4 and 5 state what the numbers do and do not support.

2. Methods

We coded two bodies of text, one from each institution that issued assessments. The PDC texts come from the Council's strategic assessment series (United Nations Planetary Defense Council, Secretariat, CE 199–201), and the Fleet texts come from the staff estimates later released for teaching (Fleet International, Combined Staff, CE 201). Together they hold 212 texts, 127 from the PDC and 85 from Fleet International. Of these, 165 are working-level assessments signed by 64 named forecasters (98 PDC, 67 Fleet), and 47 are consolidated assessments issued over the signature of a body such as a Council committee or a Fleet staff section (29 PDC, 18 Fleet).

One coder recorded every stated probability that the probe was an envoy, as a number where the text gave one and otherwise through the verbal scale of Tarnow (31 DE), which maps the stock terms of Council estimates (e.g., "unlikely") to fixed values. A second coder re-coded 60 texts blind to the first, and the two agreed within 0.05 on 53 of them (88%).

Of the 64 forecasters, 58 stated an envoy probability in at least one text, and we use the last value each gave before the battle. Of the 47 consolidated assessments, 41 stated one, and 37 of those cite identifiable working-level inputs (median 4, range 2 to 7). For these 37 we define the input pool as the mean log-odds of the cited inputs, converted back to a probability (approximately the geometric pool of Hallowell, 40 DE). A consolidation tier is one step of drafting between the inputs and the stated assessment, from a desk to a section to a committee or staff, and the signature blocks give 1, 2 or 3 tiers.

A sub-forecast is a dated and resolvable statement with a stated probability, such as whether the probe would hold its heading to a given date or whether its closing speed would fall inside a stated band. We kept the 1,386 sub-forecasts of the 52 forecasters who made at least five that resolved before the battle (approximately 27 each), and we resolved them from the telemetry and survey logs. Following Bexley (44 DE), we scored them with the Brier score and compared it with the best constant forecast, which is the observed rate. The stated probabilities sit on ten levels (0.05, 0.15, …, 0.95), two in each band of Table 1. Calibration came from a logistic recalibration of outcome on the log-odds of the forecast, for which perfect calibration gives a slope of 1 and an intercept of 0. Standard errors allow for the (co)variance of residuals within a forecaster through a design effect of 1.51, computed from an intraclass correlation of about 0.02.

What we test is the claim that the forecasters were miscalibrated, which predicts a recalibration slope below 1 or an intercept away from 0 in the sub-forecasts, a positive link between a forecaster's Brier score and the stated envoy probability, or both. We assess the link with a Spearman correlation across the forecasters who have both sub-forecasts and a stated envoy probability, with a Fisher interval.

The probe was not an envoy, so the Brier score of a stated envoy probability p is p squared, and a coin-flip forecast scores 0.250. All forecasters answered one question with one outcome, so their scores are strongly dependent, and the standard errors that we give describe spread among forecasters and not independent evidence of skill. The envoy question is therefore descriptive only, and the test of calibration rests on the sub-forecasts that Section 3 reports.

For the 37 consolidated assessments with identifiable inputs, we regressed consolidated log-odds on input-pool log-odds and took the paired difference (consolidated minus pool) as the shift that consolidation introduced. We then regressed that shift on the number of tiers. Intervals for both regressions and for the paired shift use the t distribution with the matching degrees of freedom.

3. Results

The 212 texts gave 58 forecaster-level envoy probabilities and 41 consolidated ones. The 1,386 sub-forecasts resolved yes in 681 cases (an observed rate of 0.49), and the mean stated probability was 0.50.

On the sub-forecasts the Brier score was 0.176, against 0.250 for the best constant forecast, a skill of 0.30. The recalibration slope was 0.98 (95% CI 0.84 to 1.12) and the intercept was -0.03 (95% CI -0.19 to 0.12). Table 1 shows the agreement in five bands of stated probability. Among the 47 forecasters who also stated an envoy probability, forecaster-level Brier scores ran from 0.10 to 0.24, with an unweighted mean of 0.171.

For the envoy question the 58 forecasters stated a mean probability of 0.38 (SD 0.14), and 12 of them (21%) stated 0.50 or more. Their mean Brier score was 0.162 (SE 0.015). Across the 47 forecasters who have both (5 of the 52 with sub-forecasts stated no envoy probability), the Brier score on sub-forecasts showed no detectable relation to the stated envoy probability (rho 0.05, 95% CI -0.24 to 0.33, p = 0.74).

The 41 consolidated assessments stated a mean of 0.70 (SD 0.18), and 34 of them (83%) stated 0.50 or more. Their mean Brier score was 0.522 (SE 0.036), and Table 2 sets the four sources side by side.

Among the 37 consolidated assessments with identifiable inputs, the input pools had a mean of 0.38 (SD 0.10), and 5 of them (14%) stood at 0.50 or more. The consolidated values had a mean of 0.69 (SD 0.18), and 30 (81%) stood at 0.50 or more. The institutions followed the order of their inputs, with a slope of 1.42 (95% CI 0.83 to 2.01, R² = 0.41) of consolidated on pooled log-odds, and they raised the odds of an envoy in 36 of the 37. The mean shift was 1.45 in log-odds (95% CI 1.19 to 1.72, SD 0.79), an odds multiple of 4.28 (95% CI 3.29 to 5.58). The Brier score of these 37 rose from 0.156 for the input pools to 0.505 for the assessments, a difference of 0.349.

The shift grew with the number of tiers (Table 3). Each added tier raised the log-odds by 0.86 (95% CI 0.67 to 1.06, R² = 0.69), from a mean of 0.76 at one tier to 2.50 at three.

4. Discussion

The claim that the catastrophic misreading was a calibration failure of the forecasters is not supported. Across 1,386 sub-forecasts the recalibration slope was 0.98 (95% CI 0.84 to 1.12) and the intercept was -0.03 (95% CI -0.19 to 0.12), so neither overconfidence nor a bias in level is detectable, and the Brier score of 0.176 gave a skill of 0.30 over the best constant forecast. A forecaster's score showed no detectable relation to the stated envoy probability (rho 0.05, 95% CI -0.24 to 0.33, n = 47), so the interval excludes a correlation above 0.33. On the envoy question itself the forecasters stated a mean of 0.38, and only 12 of 58 (21%) stated 0.50 or more.

At consolidation the stated odds of an envoy rose. The 41 consolidated assessments stated a mean of 0.70, and 34 of them (83%) stated 0.50 or more, against a mean of 0.38 for the individuals (descriptive on one outcome, with Brier scores of 0.522 and 0.162). Among the 37 with identifiable inputs, consolidation raised the log-odds of an envoy by 1.45 on average (95% CI 1.19 to 1.72), an odds multiple of 4.28 (95% CI 3.29 to 5.58), and the rise grew with each tier, from 0.76 at one tier to 2.50 at three.

We write within the Luo-Ji / Ye framework, in which a civilisation that senses a chain of suspicion forming has reason to read an approaching probe as a threat. An estimate that rises toward an envoy as it climbs the chain of drafting is therefore the pattern that needs explaining.

The strongest objection is about transfer. A forecaster who is sharp on arrival windows and closing speeds may still hold a "habit" of reading probes as envoys, and intent has no sub-forecast of its own, so the carry-over cannot be checked directly. What meets the objection is the stated envoy probability itself. The forecasters averaged 0.38, and only 21% stated 0.50 or more, so the working-level texts show no lean toward an envoy of the kind that such a habit would produce. The objection survives in part, because 58 stated probabilities on one question are still one observation, and Section 5 says what is left.

From one outcome the inquiries argued to a habit of the forecasters. The sub-forecast scores replace that inference with an estimate, and the estimate shows no habit to blame. The shift analysis then finds the movement toward an envoy at the institutional step, and it appears at every tier. The data show where the shift happened and how large it was. They do not show why, and we test no cause here.

5. Limitations

The sub-forecasts concern kinematics, and calibration on arrival windows and speeds does not certify calibration on intent. The envoy question is the only direct measure of intent, and it has one outcome, so the 58 forecasters supply one observation and not 58. We claim that no miscalibration was found, and we do not claim that the forecasters judged intent well.

Declassification selects the texts that we can read. The 212 texts, the 64 forecasters and the 47 consolidated assessments are those that survive, and the texts that were written and withheld can differ in either direction, and we have no count of them.

Every text was drafted for the sophons as well as for the Council (Corvane, 49 DE). A stated probability is what the drafter chose to write, and a drafter aware of a hostile reader has reason to understate suspicion, so the individual means are probably upper bounds on what the forecasters believed. We cannot test this. It is also the one candidate that we can name for the consolidation shift, since each tier added a reader, but we hold no intermediate drafts and so cannot say which edits moved the odds.

The verbal scale fixes each stock term at one value, and four of the 41 consolidated assessments cite no identifiable inputs and drop out of the shift analysis. Tier counts rest on signature blocks, and the design effect rests on an intraclass correlation of 0.02 that we estimated from the same texts. We built the pool on log-odds, and a linear pool of the same inputs would give a different baseline that we did not compute.

For the records that the inquiries read wrongly, the consequence is practical. Statements in the post-battle literature that cite forecaster overconfidence need to be checked against the working-level texts of the Council series, where most forecasters put their larger weight on a probe that was not an envoy, and the consolidated assessments need to be read as a separate source from the inputs that they cite, since the rise in the stated odds of an envoy appears only at that step.

The first extension is a second resolved question about intent, which would turn one observation into a sample. The IDC transcripts are the likeliest place to look for a stated intent that later events settled. The second is to score the working-level estimates of the Council series against the committee texts that consolidated them, if the intermediate drafts are ever released, so that each tier can be tied to the edits that moved the odds. The last is the unsigned reader. Every text was drafted with a hostile audience in view, so a stated probability blends belief with display, and until scoring carries a term for the display we can report only the sum.