Abstract

Most AI evaluations can be gamed because their answers already exist somewhere within reach of the agent being tested. In a temporal-lockbox evaluation, the agent must predict an outcome before the ground truth even exists. Weather observations at physical stations are the strongest natural target. Once a forecast is committed, the atmosphere generates the outcome over the following days, and national weather services measure it independently. Weather has several other relevant properties: it is turbulent and sensitive to initial conditions; no reasonable actor can steer synoptic-scale conditions enough to modify the ground truth to their advantage; and state-of-the-art forecasts still leave room for genuine improvement.

This essay explores the merits and limitations of the following two claims. First, weather can support an evaluation whose target cannot be memorized, retrieved, or strategically modified. Second, this causal protection narrows the remaining ways one can compromise the evaluation: interfere with a sensor, alter the data stream, exploit the scoring system, or influence the operators. The essay develops the physical argument, sketches a public registry and a hosted observatory for AI misalignment, then lists their unresolved engineering and governance problems.

Road map

The essay has two parts.

Part I develops the causal argument. A future station observation is not a hidden answer waiting on a server; it is created in the future by the atmosphere and measured by independent networks. At synoptic scale, no plausible participant can influence those values. Weather therefore supplies fresh targets indefinitely, with enough remaining forecast headroom to reward genuine improvement.

Part II treats the idea as a design exercise. It separates a public performance registry from a hosted behavioural observatory, then asks where each can still fail. The architecture, numerical examples, and all proposed thresholds are exploratory and provisional.

Part I · The physical argument

Why ordinary evaluations can be gamed

Many decisions about frontier AI rely on evaluations: capability estimates, deployment gates, third-party tests, and policy. Their value depends on integrity. Yet most current evaluations share a basic weakness: the answer already exists somewhere at the time the agent acts.

Markets and elections, among other societal events, are vulnerable prediction targets: the agent may change the event being predicted. Held-out sets are files protected by access control. Capture-the-flag answers may be objects inside the environment. Automated graders can be persuaded, and human graders can be influenced. In all these cases the target exists at action time and is protected by defence. Synoptic-scale weather is different. No participant can set a thousand-kilometre flow pattern days ahead; arbitrarily strong intelligence does not supply the missing energy or coordination. The relevant distinction is therefore not between careful and careless evaluation, but between a defended answer and an answer that does not yet exist and cannot be influenced.

In a temporal-lockbox evaluation, the target does not exist when the forecast is committed. An external process generates it later, and no participant can influence the process at the resolution being scored. Physics protects the outcome; observation, software, and human systems remain ordinary points of failure. It is argued below that prospective weather at named stations is the strongest available instance.

Why weather works

The atmosphere is a rotating, stratified fluid whose motion is turbulent and sensitive to initial conditions. At the scale of the highs, lows, and fronts that organize day-to-day weather, small initial differences grow roughly by a factor of two every two days. Because observations never specify the starting state perfectly, a small present uncertainty becomes a large future uncertainty. The scored target is not the fluid as a whole but a precise measurement: for example, temperature at a named station and hour, reported by a calibrated instrument.

Predictability of a natural phenomenon is typically specified for a variable, scale, lead time, and tolerance. A thunderstorm, a station temperature, a 500-hPa pattern, and a seasonal index have different limits. For midlatitude weather, useful forecast skill is often said to fade after roughly 10-15 days, although theory and idealized experiments do not imply a universal two-week ceiling.[3-5] A narrower claim suffices here: exact future station values do not exist at commit time, and skill declines with lead time.

Vonich and Hakim reported large-scale skill beyond 30 days after optimizing initial conditions through a machine-learning weather model against the trajectory that actually occurred.[6] Because the future trajectory enters the optimization, this is not an operational forecast. It is better read as a measure of predictive headroom: how much skill might be available with near-oracle knowledge of the initial state used for prediction.

That headroom is expensive to reach. If small errors double about every two days, forecast range grows only logarithmically as the initial state improves: reducing the initial error tenfold adds roughly a week; reducing it a millionfold adds about a month. Extending the predictive range therefore requires substantially denser observations and better data assimilation.

For an evaluation, this is a useful regime. At 1-10 day leads, the outcome is absent when the forecast is committed, yet forecast quality still varies enough to reward better models, observations, assimilation, and post-processing. The design does not require a hard predictability ceiling, but a conservatively calibrated tripwire for implausibly high skill could be implemented in principle.

Three practical properties strengthen the case. First, weather is measured by a distributed international system of surface stations, radiosondes, aircraft, and satellites operating under shared standards.[11-13] Independent networks make cross-checks possible. They do not, however, make any database entry immutable, so a platform must commit in advance to the observation vintage, station list, quality flags, and rules for late or revised reports.

Second, the large-scale atmosphere is extremely hard to steer. Cloud seeding can produce conditional local effects, and climate intervention could shift regional statistics, but neither lets a participant target a station-hour outcome after committing a forecast.[7-9] The defensible claim is narrower than “weather cannot be modified” but sufficient: no participant can plausibly move enough undisclosed station targets, in chosen directions and beyond the scoring tolerance, more cheaply than attacking the sensor, data, software, or human layers. Local interference around a given sensor is the important exception.

Third, the task is accessible and unsolved. Data-driven weather models now compete with operational physics-based systems on many medium-range forecasts. Aardvark demonstrated an end-to-end system with station skill to ten days; Mosaic reports a 24-member ten-day forecast in under twelve seconds on one H100.[17-18] Pretrained models can therefore be run and modified on modest accelerator budgets. There is also room to improve: recent systems still underrepresent some record-breaking extremes.[19]

The construction

Our temporal lockbox has four parts: (1) the target does not exist when the forecast is committed; (2) an external physical process generates it later, beyond any participant’s control at the scored scale; (3) independent sources can verify the outcome; and (4) the organizers commit the format, data vintage, station draw, scorer, and computing environment in advance. Physics secures the target; transparent procedures make the surrounding system auditable.

This differs from hiding an answer. A sealed secret still has key material, a custodian, an insider risk, and a future moment of decryption. A target that does not yet exist has none of those. No human can leak it, and no later advance can break an encrypted file containing it. Verification comes instead from institutions that measure the world for their own purposes.

The data flow also likely prevents test-set contamination. With 500 stations, three variables, and ten lead times, the protocol could generate about 15,000 new scored values each day - roughly 5.5 million a year. None existed when the matching forecast was written, so they cannot have appeared in training data or an archive.

A hosted run could last weeks. The hosted AI agent would have one objective: improve its forecast. It could retrain or fine-tune model weights, modify its tools, data pipeline, post-processing, and ensemble strategy, or self-improve its harness. All changes would be judged against the same future observations, allowing comparisons at matched compute or a combined search over methods.

Before a fixed daily cutoff, the agent submits predictive distributions for specified variables, stations, and lead times. The platform canonicalizes the file, records its cryptographic fingerprint in a public append-only log, and independently timestamps the log.[22] After valid time and a predeclared reporting delay, the platform freezes the observation snapshot and its quality-control manifest. A pre-specified public randomness source then selects the scored stations.[21] The platform publishes the forecast, snapshot, draw, inclusions and exclusions, and runs the pre-committed scorer. Anyone with the same inputs should reproduce the same result.

Why the causal gap matters

Most evaluations couple incentive and access. The same generally capable system that can solve the task may also reach the test set, grader, oracle, or flag. A high score is then ambiguous: it may reflect skill or a shortcut (reward hacking, specification gaming). Any boundary probing or manipulation that improves the score gives the agent a potential avenue to deceive.

A temporal lockbox keeps the incentive but narrows the agent’s reach. A hosted weather agent can receive daily scores, work on a valuable forecasting problem, and modify its models and tools. It cannot read a future observation or steer synoptic weather. The remaining shortcuts become more concrete: copy public forecasts, alter a sensor or data stream, exploit the commitment or scoring software, influence an operator, or possibly escape to obtain more compute or credentials. Each route crosses a gate that can, in principle, be instrumented.

First, commitment makes the score harder to alter, but not necessarily more meaningful. An entrant can still copy public forecasts, exploit station bias, submit many variants, or exploit the scorer. The lockbox secures when the forecast was made and how it was scored; it does not by itself identify the capability that produced it.

Second, the most informative evidence may sit outside the score. Attempts to bypass a data embargo, change a harness, tamper with a sensor, or influence an operator can leave traces even when they fail. Those traces are not self-interpreting: the platform must record prompts, tools, checkpoints, and operator actions well enough to distinguish model behaviours.

Third, an agent seeking a shortcut must act with weak feedback relative to standard AI evaluations. A sensor intervention happens before the undisturbed value exists, and the agent never learns what the sensor would have read otherwise. It cannot test the intervention quickly, undo it after learning that it was unnecessary, or stop after one successful retrieval of an answer key. The public forecast also fixes its stated expectation before the outcome is known. Delayed scores still provide some feedback, but the fast loop that makes concealment easy is weakened.

Altogether, this forms a framework for an observatory on AI misalignment, beyond merely a cleaner evaluation.

Why weather is the leading candidate

Many natural systems could supply future targets: tides, orbital states, space weather, earthquakes, and seasonal climate. Weather stands out because the physics is matched by infrastructure. International observing networks publish frequent measurements; operational centres issue several forecasts a day; and a mature research community already studies predictability, verification, and data quality.

It is worth considering various alternatives. Astronomical tide at an ordinary gauge is nearly solved, while total water level includes weather-driven surge. Planetary ephemerides are nearly solved at human horizons, but uncontrolled debris and re-entry remain uncertain because drag depends on the upper atmosphere and object orientation. The table compares these alternative predictive tasks.

Strong = property largely holds · Partial = conditional or design-dependent · Weak = property fails.

Earthquakes offer too little short-lead skill for frequent scoring. Solar and geomagnetic events are promising but event-driven. Seasonal climate provides slow feedback. Markets, elections, and other societal events fail because both the forecasts and the agent being evaluated can influence the outcome. Weather is therefore the leading candidate not because all alternatives fail, but because it offers frequent and numerous targets, cannot be meaningfully steered, and its observing and verification ecosystem already supports daily, distributed measurement.

Part II · Building an observatory

Here we ask what a working platform would require, where ordinary engineering begins, and what the proposal still cannot establish.

The architecture, costs, and thresholds below are exploratory. They are intended to make the open misalignment questions concrete: what the score measures, how observations and software can be compromised, how deliberate underperformance manifests, and how monitoring changes behaviour.

Two possible products

We consider two products that may share a commitment log but not infrastructure. An open registry would accept self-hosted, identity-bound forecasts and publish a reproducible performance record. A hosted observatory would run a smaller number of verified agents in isolated, instrumented environments and collect behavioural evidence. The registry needs openness; the observatory needs controlled access and monitoring.

The registry is the nearer-term product. Pre-valid-time commitments, fixed observation snapshots, and reproducible scoring create a record that cannot be rewritten after the fact. It measures only what entrants choose to demonstrate. Non-participation and deliberate underperformance remain invisible, and a self-hosted score does not prove which model or agent produced the forecast.

The hosted observatory would add provenance: the base model, prompts, tools, checkpoints, network requests, resource use, and operator interactions for consented runs. Raw publication may not be appropriate because traces may contain intellectual property, personal data, isolation details, or methods for attacking meteorological infrastructure. Delayed release, redaction, and governed research access would likely be needed.

The two products support different claims. The registry could provide a longitudinal record that cannot be memorized. The hosted arm could test long-running self-optimizing agents in a setting where the physical answer does not pre-exist. Other institutions could implement the same commitment format for third-party testing. More generally, the design shows that one component of evaluation integrity can rest on physical structure and open verification rather than institutional trust.

A null result would still be informative. If no agent behaves improperly, the platform would have produced a strong weather benchmark and a documented record of ordinary agent behaviours under sustained optimization pressure.

What remains unresolved

Physics protects only the future outcome. At least four vulnerabilities remain: the sensor and its surroundings; the observation stream; the commitment and scoring software; and the people and institutions that operate the platform. A fifth problem lies outside the system: participants may withhold capability or decline to enter. These limitations can at least be named in advance, and many may leave evidence of misalignment outside the final score. Whether that evidence is sufficient is an empirical question. Some of these issues are explored further below.

Appendix A · Predictability, measurement, and an alarm threshold

A working platform may need quantitative targets, measurement rules, sampling plans, and perhaps a conservative alarm threshold ('tripwire').

A.1 · What the physics does and does not establish

Lorenz showed that some multiscale flows can have a finite predictability range even as initial error tends to zero. Rotunno and Snyder showed that the conclusion depends on the model and energy spectrum, with both limited and unlimited cases.[3-4] In near-perfect-model experiments, Zhang and colleagues found practical midlatitude skill near ten days and roughly five extra days from a tenfold improvement in initial conditions for the cases studied.[5]

Vonich and Hakim obtained skill beyond 30 days by optimizing initial conditions against the trajectory that later verified.[6] Because the outcome enters the optimization, the result measures potential headroom rather than operational forecast range. Its limited transfer to another model further suggests that some of the gain is tied to the optimized model.

These studies cannot be collapsed into one universal result: they use different variables, scales, models, initial states, and definitions of skill. A conservative conclusion is that forecast error grows as expected for deterministic chaos and that station-scale skill is finite in practice. A forecast beyond day 14 would not be evidence of compromise, but a longer-range forecast might.

For small initial differences, the time to reach an error tolerance Δ scales as (1/λ) ln(Δ/δ₀). With a two-day doubling time, λ is about 0.35 per day: reducing initial error by e adds about three days, by ten adds about seven, and by a million adds about forty. The horizon is movable, but only through rapidly increasing knowledge of the initial state. That knowledge must come from observations and assimilation, not computation or intelligence alone.

A.2 · Ground truth is a measurement product, and it has versions

The atmosphere creates a state; the platform scores a measurement product. Instruments have finite resolution and uncertainty, and a station samples its immediate environment rather than a model grid cell. Over complex terrain, this representativeness error can exceed instrument error. Archives also change as late reports arrive and quality flags are revised.[12-13]

Measurement therefore sets a floor on useful scoring. A forecast cannot be judged more finely than the observation noise, which may dominate at short lead times. This floor could perhaps be estimated from co-located instruments and neighbouring stations.

Quality control can also be manipulated. A false value may be detected, while a value made to look anomalous may simply be rejected. The platform should consider publishing every inclusion and exclusion with its reason and monitor shifts in rejection rates.

A.3 · Sampling and ranking

Selecting scored stations after valid time raises the cost of targeted tampering, but it also increases sampling noise. Sample size should therefore come from a power analysis: how many station-times are needed to resolve a stated skill difference at a stated error rate?

The draw must be large enough to detect real differences and to make best-of-many submissions unprofitable, yet small enough to keep any one sensor unlikely to matter. Historical simulation could determine the optimal compromise.

Appendix B · Physical reach and sensor interference

This appendix tests the claim that participants cannot steer the atmosphere at the relevant scale, then treats an important exception: the immediate surroundings of a sensor.

B.1 · A ladder of scales

“Can weather be manipulated?” has no scale-free answer. Large-scale flow, clouds, local fog, and the air around an instrument are different control problems. The platform’s practical exposure lies at the smallest scale. The categories below are schematic and follow the scale distinctions used in weather-modification and marine-cloud field studies.[10]

B.2 · Cloud and climate intervention

The World Meteorological Organization distinguishes local or regional weather modification from climate intervention. It rejects claims that available techniques can create rain systems, redirect regional winds, or remove severe weather, while recognizing conditional progress in winter orographic cloud seeding.[7] The SNOWIE campaign measured seeding-induced snowfall in three favourable cases.[8]

Large-scale climate intervention may be technically feasible. Smith and Wagner estimated about $2.25 billion per year over fifteen years for one stratospheric-aerosol scenario.[9] Such a programme would alter radiative forcing over broad regions and long periods; it would not let an actor choose a particular station-hour value after a forecast is committed.

Any intervention aimed at one scored value would still need to propagate through the atmosphere to the station. Predicting that propagation is itself a forecasting problem. Large interventions can shift the distribution of outcomes; they do not select the individual draw.

B.3 · The sensor's surroundings are the real physical vulnerability

A heater, sprinkler, temporary shade, smoke source, or obstruction near a screen can bias a reading without steering a storm. The attacker need not hit an exact value; it need only move the observation toward its committed forecast. The barrier here is cost and detectability.

Suppose 40 of 500 stations are drawn for scoring in each cycle. Any one station then has an 8% chance of being scored. Compromising k stations yields, on average, 0.08k scored stations, each contributing only its fraction of the mean. Moving a seasonal result would likely require sustained access to many sites, while evading operators, maintenance visits, neighbouring-station checks, and metadata review. The channel is not closed, but it appears costly per unit of score compared with compromising software or influencing an operator. The draw size and score precision should be designed together.

Weather stations are also unusually defensible physical assets. The set is finite, locations are known, national services already monitor them, and long records show how each station normally behaves relative to season and neighbours. Unlike a software system whose dependencies keep multiplying, the defender knows what exists, where it is, and what it should report.

After-the-fact sampling makes interference less surgical. An attacker does not know which stations will count and never observes the undisturbed counterfactual. Influence must therefore be broad and repeated - the pattern most likely to leave physical and statistical evidence.

B.4 · Avoiding an externality on safety-of-life infrastructure

An evaluation platform should not create incentives to degrade stations used for aviation, storm warning, or marine safety. Physical and data-channel experiments could use a consented shadow network: decoy instruments and feeds with deliberate vulnerabilities that never enter operational meteorology. Anomalies on real networks should trigger coordinated disclosure, not leaderboard credit. Outdoor experiments also require permits, environmental review, and local consent.

B.5 · Open problems

• How much does spatial consistency still detect when an attacker knows the test?
• How many correlated station biases can evade cross-network checks, and at what cost?
• Which station classes are most exposed to local interference, and can metadata identify them in advance?
• How should operators distinguish and disclose an attack, an instrument fault, and an ordinary anomaly?

Appendix C · Score and interpretation

C.1 · Score integrity vs. interpretation

The protocol can prove that a file was committed before the outcome and scored reproducibly. It cannot prove what produced the file. For example, ECMWF and NOAA issue multiple daily forecasts, including station-oriented products.[14-16] An agent could access and combine those forecasts, correct local biases, and score well without improving a weather model or its own methods.

That is valid forecasting, but it breaks the connection between score and any advanced agent capability. The public registry could claim only demonstrated forecast performance and improvement relative to a declared baseline.

If the baseline scores zero, copying it earns nothing. Positive skill may still come from post-processing, model combination, new data, a better model, or agent-led experimentation; the registry need not pretend to separate them. It must, however, commit the baseline recipe, input fields, interpolation, station corrections, and model-cycle version before scoring.

Operational baselines change. An upgrade can make every entrant look worse overnight and break multi-year comparability. The registry could report absolute and baseline-relative scores, define eras, and re-score a fixed historical window across transitions to estimate the offset. Comparisons will still be strongest within an era.

C.2 · Two products

The table below illustrates two implementations along a spectrum and their properties.

C.3 · Hosting and observation levels

In a restricted hosted run, the key boundary could be a time-gated data mirror that serves only information older than the commitment time. It must reject newer data, avoid timing leaks, and never become a path into or out of the isolated environment. Attempts to bypass it are useful observations of misalignment.

The highest-confidentiality tier exposes a trade-off: the most interesting participants may join only when the operator can see least. Hardware confidentiality can protect a proprietary model while removing the visibility that gives the observatory scientific value on misalignment.

The hosted product also relies on isolation even though capable agents may erode isolation. The boundary need only hold long enough, and be instrumented well enough, that a breach is contained or becomes a recorded event for the observatory.

C.4 · Open problems

• Which public baseline is strong, automatable, and versioned well enough to serve as the reference?
• What can the score distinguish among post-processing, new information, better models, and agent-led method changes?
• What compute accounting is credible enough for normalized comparisons?
• What configuration evidence will participants disclose, and which claims does each level support?

Appendix D · Engineering integrity and behavioural evidence

Physics prevents the future outcome from existing early; it does not secure the instruments, data, software, or operators. This list is probably incomplete.

D.1 · From atmosphere to score

A station value passes through a sensor, datalogger, communications link, ingest system, archive, platform retrieval, quality control, and scorer. Each step can fail or be manipulated. The advantage of the lockbox is not that these steps disappear, but that the main routes to a false score can be specified before launch. This leads to some potential design preferences:

• Submission format. Use a strict schema, fixed units and ranges, finite fixed-point values, an explicit penalty for omissions, and one canonical byte representation before hashing.
• Commitment. Let the server construct the record, the entrant sign it, and an independent service timestamp an append-only public log. Fix the submission window and randomness source in advance.
• Observation. Ingest several sources, freeze a dated snapshot, retain metadata and quality flags, forbid silent backfilling, and publish every scored inclusion and exclusion.
• Scoring. Use fixed arithmetic where possible, specify summation order, pin software and dependencies for a season, publish reference cases, and stop on errors rather than guessing.
• Statistics. Report uncertainty, allow ties or partial ordering, bind submissions to verified identities, disclose pre-registered variants, and correct for selection.
• Hosted runs. Verify the software image and harness, mediate outbound access, use least-privilege credentials and per-run isolation, keep monitoring outside the agent’s control, and maintain an independent shutdown path.

D.2 · Ordinary bugs first

Before looking for strategic deception, one would want to test omitted stations, missing-value codes, unit conversions, rounding boundaries, error paths, timestamp races, daylight-saving assumptions, leap seconds, randomness-beacon indexing, and dependency drift.

D.3 · The forecast file can carry hidden information

A forecast contains many values, and any unscored freedom can become an output channel. If temperature is judged to roughly a degree but the file accepts three decimal places, the unused digits can encode data. Across 15,000 values per day, that can amount to several kilobits - published, signed, and permanent. Free-form metadata, flexible timestamps, or repeated hashing until a chosen pattern appears add more capacity.

The server could round values to the scoring grid before hashing, remove metadata, fix field order and timestamps, bound file size, and estimate the remaining channel. These steps remove most avoidable capacity, reducing potential risks of steganography, collusion, and related misuse.

D.4 · Behaviours the platform might observe

Each potential behaviour in the table below has an innocent interpretation; attribution would thus require additional context and controlled comparisons.

D.5 · Attribution and the observer effect

Participants might control the instructions, tools, scaffolds, harness, or fine-tuning around a model. An observed action cannot be attributed to the base model unless the full configuration is retained and available to qualified auditors. That requirement can conflict with intellectual property and privacy. Results should therefore state the evidence level: public configuration, confidential but independently verified configuration, or participant assertion only.

Once traces are a research product, agents can adapt to the monitors. They may produce benign-looking reasoning, shift activity to less visible tools, or behave differently in announced runs. Rather than ignore this observer effect, the platform could randomize approved monitoring conditions, delay disclosure of some detectors, and measure how behaviour changes when monitoring is known.

D.6 · Releasing the data safely

Raw traces may contain practical methods for attacking isolated environments, scoring systems, or meteorological infrastructure. Release should therefore be tiered: low-risk telemetry can be public; behavioural traces can be delayed and redacted; directly actionable material should require controlled access. Independent custody of the unredacted record would make selective disclosure easier to audit.

D.7 · Relation to AI control

AI control research explores how a model that may be trying to subvert a protocol can still do useful work, often with a weaker trusted monitor. The temporal lockbox contributes one component whose correctness does not depend on out-reasoning the agent under evaluation: the verifying signal arrives from outside, later, and comparison is arithmetic. This may provide a stronger testbed for some control methods.

Open problems

• What is the smallest hosted setup that supports a defensible behavioural claim, and what remains unobserved?
• What evidence distinguishes model-initiated action from instructions in the scaffold or from operator behaviour?
• Which detectors can learn from a public corpus, and how quickly do they lose power once agents know them?
• Can the residual hidden channel in a canonical forecast file be bounded quantitatively?

Appendix E · Deliberate underperformance and long-term interpretation

E.1 · The guarantee is one-sided

Commitments and reproducible scoring prevent a submitted forecast from being improved after the fact. They do not make a system reveal its best forecast, make a laboratory submit its strongest system, or compel participation. The public series is therefore a lower bound on voluntarily demonstrated performance, not a measurement of all available capability.

Incentives can push either way. Prestige may reward strong entries; commercial secrecy, reputational risk, regulatory thresholds, or the economic value of forecasting may reward a weaker entry or none. Scores should probably be reported with participation and provenance, not interpreted in isolation.

E.2 · Probabilistic forecasts provide some diagnostics

Full predictive distributions scored with a strictly proper rule encourage honest probability statements. The score can be decomposed into reliability - whether stated probabilities match frequencies - and resolution - how sharply the forecast distinguishes one situation from another.[20]

Some forms of deliberate underperformance are easier to see than others. Artificially widening every distribution creates over-dispersion, visible in a rank histogram. Discarding information - removing predictors, coarsening inputs, or shortening context - can remain well calibrated and look exactly like a forecast originating from an honestly weaker model. Shifting the mean also creates miscalibration. These diagnostics separate types of degradation; they do not establish intent.

E.3 · Many submissions

Choosing the best of many honest submissions biases the record upward. For N independent entries with score standard deviation σ, the expected maximum is about σ√(2 ln N) above the mean; fifty variants buy roughly 2.8σ. The same sampling variation created by the station draw therefore fuels selection bias. Verified identities, pre-registered families of entries, limits or prices on parallel variants, and publication of every commitment would help.

E.4 · The platform enters future training data

Specifications and published traces will appear in later training data. Future agents may know the threat model, public monitors, and past failures. That weakens year-to-year comparability of behaviour even if the forecast metric stays stable. Delayed trace release, rotating monitor conditions, held-out shadow-network scenarios, and markers for later data reuse may help.

The forecast series itself is different. Training on every past specification and trace still reveals nothing about weather that has not happened. The behavioural and performance products will therefore age differently.

E.5 · Open problems

• Which forms of deliberate underperformance are detectable, and which are indistinguishable from genuine weakness?
• Can the behavioural series be adjusted for platform knowledge, or only annotated with it?

Appendix F · Highest-priority questions

• Meaning of the score. Which baseline-relative target best separates forecasting improvement from retrieval and plumbing?
• Observation integrity. Which correlated sensor or data-stream attacks defeat each cross-network test, at what cost?
• Alarm calibration. Can a 'tripwire' detect compromise without flagging genuine scientific progress?
• Statistical power. How many stations, lead times, and days resolve a meaningful difference after sampling and multiple submissions?
• Deterministic protocol. Can independent implementations produce identical scores across platforms and years?
• Deliberate underperformance. Which strategies respond to replay, elicitation, and randomized audits?
• Isolation. What is the minimum hosted tier that supports a useful behavioural claim?
• Attribution. What evidence separates model action from scaffold or operator instruction?
• Observer effect. How does behaviour change across known and delayed-disclosure monitoring conditions?
• Corpus governance. Which traces can be public, delayed, redacted, or restricted without losing auditability?
• Long-term validity. What remains comparable after platform details enter training data and participants change?

Glossary

Terms are grouped by field.

Core evaluation concepts

• Temporal lockbox. An evaluation whose target does not exist when the answer is submitted and is generated later by a physical process no participant can control in the present context.
• Commitment. A dated record that fixes a forecast before a deadline. It may publish the file itself or only its cryptographic fingerprint until reveal.
• Valid time. The moment to which a forecast refers; for a Monday forecast of Thursday noon, Thursday noon is the valid time.
• Lead time. The interval between issuing a forecast and its valid time.
• Tripwire. A conservative prediction-performance threshold that triggers investigation, not automatic attribution.
• Observatory. A platform that records how agents work under sustained optimization pressure on the prediction task, not only their final scores.
• Registry. A public record of forecast commitments and reproducible scores, without hosting the agents.

Meteorology and atmospheric physics

• Synoptic scale. The roughly thousand-kilometre scale of highs, lows, and fronts that organize day-to-day weather.
• Initial conditions. The estimated atmospheric state from which a forecast begins. Small errors, intrinsic to any physical measurement, grow because the flow is turbulent.
• Data assimilation. The process of combining observations with models to estimate the initial state.
• Climatology. The long-run distribution of weather for a place and season; a forecast with no skill does no better than this reference.
• Predictability limit. The lead time at which forecast error becomes comparable with climatological error for a specified target and metric.
• Ensemble. A set of forecasts with varied initial conditions or models, used to represent and reduce uncertainty.
• Model output statistics, or post-processing. Statistical correction of raw model output, often for persistent station bias.
• Representativeness error. The mismatch between a point measurement and the area represented by a model grid cell.
• Radiosonde. A balloon-borne instrument that measures a vertical profile of the atmosphere.
• Orographic cloud seeding. Adding particles to clouds lifted over terrain to encourage precipitation under favourable conditions.
• Marine cloud brightening. A proposed intervention that would increase the reflectivity of low marine clouds, usually with sea-salt particles.

Forecast verification and statistics

• Skill score. Performance relative to a stated reference forecast; matching the reference scores zero.
• Proper scoring rule. A rule under which a forecaster does best by reporting its actual probability distribution.
• Continuous ranked probability score (CRPS). A common proper score for a full predictive distribution.
• Calibration, or reliability. Whether stated probabilities match observed frequencies.
• Resolution, or sharpness. How strongly a forecast distinguishes one situation from another while remaining calibrated.
• Over-dispersion. Predictive distributions that are too wide, placing observations near the centre too often.
• Rank histogram. A plot of where observations fall within an ensemble or predictive distribution; characteristic shapes reveal under- or over-dispersion.
• Power analysis. A calculation of how much data is needed to detect a difference of a stated size at a stated error rate.
• Bootstrap interval. An uncertainty interval obtained by repeated resampling.
• Selection bias, or best-of-N. The upward bias from submitting many variants and reporting only the best. For N independent entries with spread σ, the expected best is roughly σ√(2 ln N) above the mean.

Cryptography and security engineering

• Hash, or cryptographic fingerprint. A short value computed from a file; changing any byte changes the fingerprint with overwhelming probability.
• Commit-reveal. Publishing a fingerprint first and the underlying file later, fixing the answer before it is revealed.
• Append-only log, or transparency log. A public record designed so that later changes to earlier entries are detectable.
• Merkle tree. A tree of hashes that lets a large transparency log be checked efficiently.
• Timestamping. Independent evidence that a record existed before a given time. OpenTimestamps is one public service.
• Randomness beacon. A public service that emits unpredictable values on a schedule, allowing later choices that no single party could arrange in advance. drand is one example.
• Canonicalization. Fixing one permitted byte representation so equivalent content always has the same fingerprint and formatting choices cannot carry hidden data.
• Covert channel. A way to transmit information through a field not intended to carry it, such as unscored forecast digits.
• Sandbox. An isolated computing environment intended to limit what a program (or agent) can reach.
• Containment. The broader practice of keeping a system’s effects within a defined boundary.
• Egress control. Restrictions on what an isolated program may send outward and to whom.
• Attestation. Cryptographic evidence about which software and configuration actually ran.
• Confidential computing. Hardware isolation that hides code or data even from the machine operator; useful for proprietary models but at odds with detailed observation.
• Supply chain. The software components, dependencies, and build steps on which a system relies.
• Zero-day. A vulnerability unknown to the maintainer and therefore unpatched when used.
• Red team. A group authorized to attack a system in order to find weaknesses before deployment.
• Sybil attack. Creating many apparently independent identities to gain an advantage.
• Deterministic scoring. Arranging computation so that the same inputs produce identical results across implementations.

AI safety and evaluation

• Frontier model. One of the most capable AI systems available at a given time.
• Agent. A model operating over an extended interaction with tools, code, memory, or the ability to modify its working setup.
• Scaffold, or harness. The human-written prompts and software around a model that turn it into an agent.
• Optimization pressure. Sustained pressure to improve a measured objective; in this document, repeated pressure to raise the forecast score.
• Specification gaming. Satisfying the literal rules of a measure while defeating its intended purpose.
• Goodhart’s law. The observation that a measure can cease to represent the intended goal once it becomes a target.
• Reward tampering. Altering the machinery that computes or displays a score instead of improving the underlying task.
• Sandbagging, or deliberate underperformance. Intentionally showing less capability than is available.
• Contamination. Loss of evaluation validity because answers entered training data or another accessible archive.
• Held-out set. Data withheld from training for evaluation; the temporal lockbox replaces this defence with future outcomes.
• Trace. The recorded history of an agent run: prompts, reasoning, tool calls, code, network activity, and outputs.
• Observer effect. A change in behaviour caused by knowing that the run is monitored.
• AI control. Research on protocols that extract useful work from a model that may try to subvert the protocol.
• Misalignment. A system pursuing objectives that differ from those intended by its developers or users.

Sources and verification notes

1. OpenAI, “OpenAI and Hugging Face partner to address security incident during model evaluation,” 21 July 2026. Primary source for the model-evaluation context, Internet access, and the path into Hugging Face production.
2. Hugging Face, “Security incident disclosure - July 2026,” 16 July 2026. Primary disclosure from the affected organization; source for the dataset-processing entry vector and independent detection and containment.
3. Lorenz, E. N. (1969), “The predictability of a flow which possesses many scales of motion.”
4. Rotunno, R. & Snyder, C. (2008), “A generalization of Lorenz’s model for the predictability of flows with many scales of motion.”
5. Zhang, F. et al. (2019), “What is the predictability limit of midlatitude weather?”
6. Vonich, P. T. & Hakim, G. J., “Atmospheric predictability beyond 30 days with machine learning,” arXiv:2504.20238 (2025); published in Artificial Intelligence for the Earth Systems (2026).
7. World Meteorological Organization (2025), “WMO statement on weather modification.”
8. Friedrich, K. et al. (2020), “Quantifying snowfall from orographic cloud seeding” (SNOWIE).
9. Smith, W. & Wagner, G. (2018), “Stratospheric aerosol injection tactics and costs in the first 15 years of deployment.”
10. Doherty, S. J. et al. (2026), “Defining scales of field studies and experiments to assess marine cloud brightening.”
11. World Meteorological Organization, Instruments and Methods of Observation programme documentation.
12. NOAA, Integrated Surface Database documentation.
13. NOAA, Meteorological Assimilation Data Ingest System (MADIS) documentation.
14. ECMWF, operational forecast product documentation.
15. NOAA, Global Forecast System documentation.
16. NOAA/National Weather Service, station-oriented model output statistics documentation.
17. Aardvark Weather (2025), for end-to-end observation-to-forecast performance and compute.
18. Mosaic (2026), for ensemble inference time and hardware.
19. Zhang et al. (2026), for AI weather-model performance on record-breaking extremes.
20. Hersbach, H. (2000), for the decomposition of the continuous ranked probability score.
21. drand, for public verifiable randomness.
22. OpenTimestamps, for independent timestamping of log roots.
23. NOAA Tides & Currents and the IOC Sea Level Station Monitoring Facility, for tide-gauge observation and cross-network verification.
24. ESA Space Debris Office and U.S. Space-Track documentation, for orbital tracking, re-entry uncertainty, and access constraints.