SynthStats demonstrations

SynthStats is a research project on generating probabilistic programs with language models. These two demonstrations are on BoxingGym's dugong-growth and disease-spread environments, with simulated data from each environment's own generator. Candidate PyMC programs are checked, fitted and scored on held-out data.

What a demonstration is

One held-out dugong, aged 0.16 years, measured 0.64. The reference program's 5th to 95th percentile predictive range at that age was 0.69 to 1.55, so that observation counted as not covered, and the log density the program assigned to it is one of the twenty terms in its held-out log score.

One BoxingGym environment gives a natural-language task and observed data: 80 visible rows and 20 held-out rows, drawn from the same input range. Ten candidate PyMC programs come from four sources: the reference, a program we wrote ourselves knowing how the data is generated, which is the comparison point for the others; the floor model, which ignores the input and uses one predictive distribution for every input value; four single attempts by the Qwen3.6-35B-A3B language model, with no feedback; and four loop proposals from our own loop, which writes two programs, has a critic read their posterior predictive summaries, and writes two more. Each program is checked before fitting: parse, syntax-tree inspection, model build, a restricted PyMC subset, and a data-sensitivity check. A program that passes is fitted and scored on the held-out rows. Every scored program also predicts the 20 held-out rows, with the empirical coverage of its 90 percent predictive interval (5th to 95th percentile, bounds inclusive).

The held-out log score is the total log predictive density over the 20 held-out rows. Higher is better, and scores compare only within one task. Coverage is the number of the 20 held-out points inside that interval, against a nominal 18 of 20.

All twenty programs at a glance

Interactive comparison