How Vigilo scores are built, validated, and published

This document defines every published metric, states the forecast formulation, and describes the validation protocol — so that a technical reviewer can verify our numbers from the public data alone. Production constants and layer internals are omitted deliberately; everything needed to audit the results is here.

1 · The accounting rule everything rests on

Every prediction is written to an append-only journal before the outcome, with a timestamp, a tier, and a confidence. Outcomes are scored against the entry as written — entries are never edited or deleted retroactively. All aggregate numbers below are computed from this journal and republished automatically every 8 hours.

2 · Classification metrics

With TP = confirmed calls, FP = calls that did not confirm, FN = events we missed (measured against an external ground truth — USGS M6+ catalogue for the disaster domain):

Confidence intervals on proportions use the Wilson score interval:

Unverifiable outcomes (opaque countries) are reported as their own category and excluded from the precision denominator — this is stated on the scorecard rather than hidden.

3 · Calibration

For journal entries carrying probability-like confidence, we publish the Brier score and its Murphy decomposition:

Reliability (miscalibration) and resolution (discrimination) are tracked separately, so a worsening score points to its cause. Tier confidences are self-calibrating: once a tier accumulates n ≥ 20 scored outcomes, its displayed confidence is its historical confirm rate (Beta-smoothed), not an author's opinion.

4 · Short-horizon forecast formulation

The composite risk forecast at horizon h blends the live signal with the country's own historical base, plus an exogenous seasonal-climatology term:

The horizon weights themselves are proprietary — fitted by blind backtest and frozen under the pre-registration protocol.

The weights were fitted by grid search on the first half of accumulated history and are re-derived blind every 8 hours on the training half, then scored on the unseen second half (public/model_validation.json). As of this writing the shipped weights remain train-optimal, and the out-of-sample edge over a persistence baseline is:

Beyond ~3 days the raw news series carries no autocorrelation (lag-7 ACF ≈ 0.01), so all longer-horizon skill comes from structurally different layers: the scheduled-events calendar, disaster climatology, external academic forecasts (ViEWS, 36-month), prediction-market odds and the two-government advisory consensus — combined as independent evidence, not extrapolation.

5 · Window probabilities and decay

v2 (2026-09-06). Our own measurements falsified the Poisson independence assumption the v1 window used: an elevated day repeats the next day ~69% of the time against a ~20% base rate — day states cluster. Trip-window risk therefore comes from a two-state Markov chain with Laplace-smoothed transition probabilities, and today's state matters:

Window probabilities come from the chain's transition structure and are state-aware: today's condition enters the forecast. Country rates are stabilised by empirical-Bayes shrinkage toward a regional pool (a three-event country borrows its region's strength; a long-history country stands on its own data), and confidence intervals use the autocorrelation-effective sample size, not raw n — dependent days must not masquerade as independent evidence. Exact transition estimators and shrinkage strength are proprietary, frozen under the pre-registration protocol. Walk-forward Brier scoring of the chain against train-climatology shows significant skill (+25.9/+2.9/+1.3% at 1/3/7 days, moving-block bootstrap). Spike decay per country is the mean-reversion half-life from an Ornstein–Uhlenbeck regression; countries without detectable reversion report none rather than a fabricated number. All of it feeds public/risk_stats.json.

A separate Reality Shift Index (public/reality_shift.json) guards against the most expensive failure of predictive systems — a confident forecast in a structurally changed world: a bounded distribution-shift measure between a country's recent signal distribution and its own full history, scaled 0–100. A shifted country is outside the regime its statistics were learned on, and its model confidence is reported as degraded — never boosted. It is an uncertainty signal, not a risk signal: calming shifts register as hard as escalation.

6 · Evidence handling

• Reprint collapsing. Near-identical headlines are collapsed by token-set key; N reprints of one wire story count as one effective source.
• Effective source count. Where source multiplicity feeds confidence, it enters as n_eff = n / (1+(n−1)ρ) — correlated sources cannot masquerade as independent confirmation.
• Organic-vs-echo separation. A topic's reproduction number R (new independent sources today / active yesterday, after collapsing) classifies it as organically growing (R>1) or a decaying echo of a single seed (R<1).
• Geographic verification. High-severity events pass an automated LLM check that the attributed country is the affected country, not merely a mentioned actor.

7 · Decision rule

Recommendations are produced by a cost-loss rule with a built-in humility constraint: action is advised only when the Wilson lower bound of the event probability clears the client's cost-loss ratio:

Less data widens the interval and makes advice more conservative — overconfidence is excluded by construction, not by policy. The same forecast therefore yields different recommendations for users with different stakes, which is intended behaviour.

8 · Estimative language, source grading, warning board

• Words are pinned to numbers (Kent 1964 → ICD-203 lineage): "likely" = 55–80%, "very likely" = 80–95%, "almost certain" = 95%+ — in both directions, product-wide.
• Admiralty grading on every event: source reliability (A–F, from source class) × information credibility (1–6, from independent corroboration) — two deliberately independent axes, so a reliable source's uncorroborated claim reads A3, not "true".
• Indicators & Warnings board (Grabo's formalism): per-country tripwires grouped by channel — MONEY / DIPLOMACY / OPERATIONS / ATTENTION / STRUCTURE; escalation requires ≥2 independent channels lit, because independent channels cannot all be wrong the same way (public/iw_board.json).
• Words-vs-deeds rule (data-poisoning defence): channels split into words (media narratives — the one thing a coordinated LLM botfarm can manufacture) and deeds (market prices, airline reroutes, government advisories, chronic base rates — costly real-world actions that text cannot fake). Narratives alone never escalate a country: every ensemble alert requires at least one deed channel, and a media wave with zero deeds behind it is flagged words_only — a manipulation watchlist, not a warning.
• Physical layer (v2) — the third rung of the words→deeds ladder: observations of the physical world that no information campaign can manufacture. Satellite thermal anomalies (NASA FIRMS VIIRS, each country scored against its own trailing baseline — z, never raw counts), country-level connectivity outages (IODA: BGP / active probing / telescope — routes disappearing is a state action, not a narrative), and market shocks (ECB FX + Brent day-moves scored as z against the trailing return distribution — money moving is costly). Physical indicators join the warning board as their own channel; several independent physics agreeing outranks any news wave (public/thermal_anomaly.json · public/net_outages.json · public/market_shock.json).
• Structured dissent: contested questions run through ACH (rank by least contradiction, Heuer) with an automatic "tenth man" pass — the leading hypothesis's evidence gaps and watch-for falsifiers are listed with the verdict; public predictions carry a written premortem before the outcome.
• Automatic WHY layer (v2): every ensemble alert gets an automatic ACH review — an LLM proposes 3-4 genuinely competing explanations (always including a benign one, and a manipulation one when media signals are present) and rates the evidence; the deterministic Heuer arithmetic picks the least-contradicted winner. The LLM never chooses the leader — hallucinated hypotheses collect inconsistencies and lose. Evidence strength follows Van Evera's process-tracing tests: a failed hoop test kills a hypothesis outright; stronger evidence classes carry greater weight. Each verdict ships with the leader's key assumptions and what would change our mind (public/ach_review.json).
• Deception screening (v2): countries flagged words_only (media wave, zero deeds) get a MOM-POP screen — motive, opportunity, means, past operations — the CIA's deception-analysis checklist, applied automatically to every suspicious narrative wave.
• Local normalcy (v2): a curated calendar of what is NORMAL per place — religious calendars, burning seasons, recurring strike rituals, holiday crowds — so anomalies are judged against the local baseline, not a global yardstick. Context annotates and raises thresholds; it never deletes evidence (public/normalcy.json).

9 · Validation protocol & anti-overfitting rules

• Walk-forward only. All fitting uses data strictly earlier than the scoring window; the test half is never touched during tuning.
• Blind re-validation every 8 h as part of the production pipeline; weight drift or edge collapse is published, not patched.
• No retro-editing. The journal is append-only; misses stay on the record; history files are guarded against being overwritten by degraded runs.
• Negative results are kept. Trend extrapolation was tested and beaten by the base rate; it was removed from the model and is documented as removed.
• Look-ahead bias. Calendar and seasonal layers use only information publicly dated before the forecast time.
• Multiple comparisons. Tier-vs-tier claims will carry Benjamini–Hochberg-corrected tests (scheduled with the day-120 threshold optimization).
• Pre-registration (v2). Every hypothesis test is registered in the repository's EXPERIMENTS.md — hypothesis, metric, success threshold, date — before it runs; negative results stay on the record. Model weights and thresholds are frozen in a dated config; backtests must use the config of the evaluated period; the most recent 30 days are a holdout excluded from all tuning decisions.
• Dependence-aware inference (v2). Significance intervals use moving-block bootstrap (errors are temporally dependent); proportions use Wilson on the autocorrelation-effective sample size. Accuracy is never a headline metric — on rare events a constant "all quiet" scores 90%+; we score with Brier / log loss, false-alert rate and median lead instead.
• Segment decision thresholds (v2). One probability, different actions: each audience segment alerts at its own P* = C_FP/(C_FP+C_FN) — a tourist's missed warning costs far more than a shifted booking; an insurer needs balanced errors (public/decision_thresholds.json).

10 · Known limitations

• Recall against external ground truth is 58% — we catch most, not all, and publish the gap.
• 92 days of proprietary history (as of this version) bounds model complexity; regime-switching and self-exciting event models are deliberately deferred until data volume justifies them.
• Media-attention inputs measure attention, not decisions; the system labels them as such.
• Country-level resolution; sub-national precision is roadmap, not current capability.