PyPI version floorsNot observed here: OK — evidence expired
No further observations recorded.
Score
37/100Monitoring: REDRelease readiness: STALE
snapshot 2026-10-08T11:46:20.006777+00:00 ·
just now
cloud snapshot ·
dev box last observed 3d ago
Why this monitoring score: 37/100Start at 100. Each family deducts its worst unresolved status: red 10, yellow 5, missing or expired evidence 2; summary and individual rows are charged once per family, with a total floor of 0. 100 requires fresh green evidence for every applicable check. Explicit permanent exclusions are listed separately.
ci_status: −10 (9 unresolved observations; family cap 10)
ci_timing: −10 (7 unresolved observations; family cap 10)
manifest_drift: −2 (1 unresolved observations; family cap 10)
no_run_census: −5 (29 unresolved observations; family cap 10)
profiling_drift: −2 (1 unresolved observations; family cap 10)
release: −2 (1 unresolved observations; family cap 10)
repo_state: −5 (30 unresolved observations; family cap 10)
required_workflow_drift: −2 (1 unresolved observations; family cap 10)
script_timing: −2 (1 unresolved observations; family cap 10)
test_run: −2 (1 unresolved observations; family cap 10)
unit_test_timing: −5 (25 unresolved observations; family cap 10)
url_check: −2 (1 unresolved observations; family cap 10)
version_skew: −2 (8 unresolved observations; family cap 10)
version_skew_pypi: −2 (8 unresolved observations; family cap 10)
worktree_drift: −10 (31 unresolved observations; family cap 10)
Release readiness score: 85/100Start at 100; subtract these capped penalties, with a floor of 0. The verdict is determined by the reasons, not the score.
HowToFit/tutorial_5_expectation_propagation [yellow]: LinearRegressionAnalysis.log_likelihood_function returns a constant -1, ignoring `instance`, so every initial sample has an identical figure of merit and the per-factor search cannot initialise. The tutorial only "passes" today because CI runs at PYAUTO_TEST_MODE=2, which bypasses the sampler; at real sampling the linear_regression factor never completes a single update. Fix needs a real likelihood for the m/c regression over fwhm_list (currently computed and unused), which is tutorial authoring, not a mechanical repair. See PyAutoFit#1454. no_run_census:HowToFit/tutorial_5_expectation_propagation: · snapshot.no_run_census:no_run_census/rows/5 · 2026-10-08T11:46:11+00:00
autofit_workspace/features/expectation_propagation.py [yellow]: EP parked as not release-ready. EP message projection is unstable: a truncated per-factor search projects an ESS=1 posterior with zero weighted variance, which EP feeds back as a delta-function prior. See PyAutoFit #1332 F10 and autofit_workspace_test graphical/ep.py. no_run_census:autofit_workspace/features/expectation_propagation.py: · snapshot.no_run_census:no_run_census/rows/6 · 2026-10-08T11:46:11+00:00
autofit_workspace_test/graphical/analytic_gaussian_priors.py [yellow]: moments projection (PyAutoFit#1656) passes the gaussian and truncated scatter families (16/16 each) but the loggaussian family misses 11/16: log-space scatter comes back at log sigma 3.28 vs mode 1.53 / reference 1.83 - TransformedMessage moments path suspected; see PyAutoMind draft/bug/autofit/ep_moments_loggaussian_transformed_scatter.md no_run_census:autofit_workspace_test/graphical/analytic_gaussian_priors.py: · snapshot.no_run_census:no_run_census/rows/7 · 2026-10-08T11:46:11+00:00
autofit_workspace_test/graphical/ep.py [yellow]: EP not release-ready: the per-factor searches are truncated (maxcall/maxiter=1000), so each search projects an ESS=1 posterior whose weighted variance is exactly 0; EP feeds that back as a delta-function prior, and the next cycle's initializer raises InitializerException. Intermittent - broke the 2026-08-03 nightly integrate. See PyAutoFit #1332 F10. no_run_census:autofit_workspace_test/graphical/ep.py: · snapshot.no_run_census:no_run_census/rows/8 · 2026-10-08T11:46:11+00:00
autofit_workspace_test/graphical/ep_deterministic.py [yellow]: EP parked as not release-ready alongside graphical/ep.py. Passes today, but shares the EP machinery whose message projection is unstable. See PyAutoFit #1332 F10. no_run_census:autofit_workspace_test/graphical/ep_deterministic.py: · snapshot.no_run_census:no_run_census/rows/9 · 2026-10-08T11:46:11+00:00
autofit_workspace_test/graphical/ep_exact.py [yellow]: EP parked as not release-ready alongside graphical/ep.py. Passes today, but shares the EP machinery whose message projection is unstable. See PyAutoFit #1332 F10. no_run_census:autofit_workspace_test/graphical/ep_exact.py: · snapshot.no_run_census:no_run_census/rows/10 · 2026-10-08T11:46:11+00:00
autofit_workspace_test/graphical/ep_parity.py [yellow]: EP parked as not release-ready alongside graphical/ep.py. Passes today, but shares the EP machinery whose message projection is unstable. See PyAutoFit #1332 F10. no_run_census:autofit_workspace_test/graphical/ep_parity.py: · snapshot.no_run_census:no_run_census/rows/11 · 2026-10-08T11:46:11+00:00
autogalaxy_workspace/imaging/features/shapelets/modeling [yellow]: real-search JAX shapelet fit exceeds the 1800s mode=release cap (>30min); speedup tracked by the Profiling Agent (PyAutoHeart#72). Not a bug. no_run_census:autogalaxy_workspace/imaging/features/shapelets/modeling: · snapshot.no_run_census:no_run_census/rows/20 · 2026-10-08T11:46:11+00:00
autogalaxy_workspace_test/multi_dataset/jax_likelihood/delaunay.py [yellow]: quarantined: TIMEOUT 1805s at the release cap in TWO CONSECUTIVE release-integrate runs (34018429178, 34094964905) with XLA_FLAGS=--xla_cpu_multi_thread_eigen=false in force. Fifth family member (epic xla-cpu-eigen-pool-deadlock, XLA FftThunk/ducc0 Eigen-pool deadlock), and the direct twin of autolens_workspace_test's identically-named multi_dataset/jax_likelihood/delaunay.py. Both captured stacks park in jax _pjit_call_impl - EXECUTION, not compilation - reached via vmap_f -> flatten_fun_for_vmap -> cache_miss -> _pjit_batcher, i.e. the inner pjit that Fitness._vmap = jax.vmap(jax.jit(call)) eagerly executes while re-tracing, which it does on EVERY call. The two runs hung on DIFFERENT calls of the same graph, matching the two signatures already documented in the sister repo: 34094964905 at delaunay.py:201, the SECOND _vmap (jax_compile.py:306) after the first completed in 4.1s and printed its result - the autolens delaunay.py signature; 34018429178 at delaunay.py:196, the FIRST _vmap (jax_compile.py:320) with the heartbeat still logging 'still compiling' at 1770s - the autolens delaunay_mge.py / shared_preloads.py signature. A random per-execution deadlock with two exposures per script run; not a compile stall and not a profile split. NOT reproducible on the dev box, and the box cannot reproduce the family at all: 10/10 PASS under profile_release.yaml (10-25s, cold JAX cache each attempt), 2/2 under profile_smoke.yaml, and 4/4 with the eigen flag REMOVED - the configuration that gave 14/16 hangs in CI - so local evidence cannot discriminate here. RULED OUT by measurement: (a) the release profile, whose resolved env differs from smoke in only PYAUTO_TEST_MODE 2->0 and PYAUTO_FAST_PLOTS 1->0, neither of which this search-less, plot-less script reads (the two runs' logs are byte-identical apart from timestamps and durations, 716 masked pixels under both); (b) the mesh/mask ratio from eec09d6's pixel_scales 0.1->0.2, since sibling delaunay_mge.py has the identical pixels=500 / pixel_scales=0.2 / mask_radius=3.0 / 716 mask pixels (ratio 0.70) and passed 28.3s and 29.4s in the same two runs. CONTAINMENT, NOT A FIX - remove when the epic lands. Stays on the smoke gate (smoke_tests.txt), which deliberately does not apply this file. no_run_census:autogalaxy_workspace_test/multi_dataset/jax_likelihood/delaunay.py: · snapshot.no_run_census:no_run_census/rows/12 · 2026-10-08T11:46:11+00:00
autogalaxy_workspace_test/multi_dataset/jax_likelihood/rectangular.py [yellow]: re-quarantined: hung 1805s at the release cap in release-integrate run 33244320906 with XLA_FLAGS=--xla_cpu_multi_thread_eigen=false in force — same execution-hang signature as the family below (epic xla-cpu-eigen-pool-deadlock). Restored 2026-08-27 by PyAutoFit#1528 (20.0s/21.9s in the family re-time below); intermittent since — it passed integrate runs 33230028707 and 33255903385 the same day. Fourth family member re-quarantined, after autolens_workspace_test's delaunay.py, shared_preloads.py and delaunay_mge.py. Stays on the smoke gate (smoke_tests.txt), which deliberately does not apply this file. no_run_census:autogalaxy_workspace_test/multi_dataset/jax_likelihood/rectangular.py: · snapshot.no_run_census:no_run_census/rows/13 · 2026-10-08T11:46:11+00:00
autolens_workspace/cluster/start_here [yellow]: hits the full 1800s mode=release cap in workspace-validation (PyAutoHeart run 29912642195). Script-specific, not a shard-wide problem: every other cluster script passes, the next-slowest being lenstool/modeling.py at 137.8s. Consistent with cluster scripts being known un-smoke-able (>500s even in TEST_MODE). Not yet profiled — SLOW-skipped to unblock the release; remove once the cost is found and fixed (autolens_workspace#314). no_run_census:autolens_workspace/cluster/start_here: · snapshot.no_run_census:no_run_census/rows/21 · 2026-10-08T11:46:11+00:00
autolens_workspace/guides/modeling/advanced/expectation_propagation.py [yellow]: EP parked as not release-ready. EP message projection is unstable: a truncated per-factor search projects an ESS=1 posterior with zero weighted variance, which EP feeds back as a delta-function prior. See PyAutoFit #1332 F10 and autofit_workspace_test graphical/ep.py. no_run_census:autolens_workspace/guides/modeling/advanced/expectation_propagation.py: · snapshot.no_run_census:no_run_census/rows/14 · 2026-10-08T11:46:11+00:00
autolens_workspace/multi_galaxy/features/advanced/shapelets/modeling.py [yellow]: hit the 1800s mode=release cap in the 2026.10.7.1 release smoke job (PyAutoHands run 37653172766, run_smoke_tests 3.12 autolens_workspace: TIMEOUT at 1805s; the other 41 listed scripts passed). Real-search JAX shapelet fit, the same cost as its autogalaxy sibling imaging/features/shapelets/modeling (SLOW-skipped in autogalaxy_workspace#131). Passes in ~15s under profile_smoke (PYAUTO_TEST_MODE=2), so the cost is the release-profile sampler, not a bug. Not yet profiled - SLOW-skipped to unblock the release; remove once the cost is found and fixed (autolens_workspace#586). no_run_census:autolens_workspace/multi_galaxy/features/advanced/shapelets/modeling.py: · snapshot.no_run_census:no_run_census/rows/22 · 2026-10-08T11:46:11+00:00
autolens_workspace/multi_galaxy/start_here [yellow]: blocked two consecutive release runs after autolens_workspace#554 (2026-09-17) swapped it off the simulated `simple` dataset onto the real SDSS J1011+0143 ACS/WFC F814W frame. On the profile defaults it hit the 1800s mode=release cap (PyAutoHeart run 35319361459, "1 timeout"); with a per-script BUILD_SCRIPT_TIMEOUT of 3600s AND PYAUTO_SMALL_DATASETS lifted (#563, reverted here) the runner died at ~1478s on a shutdown signal (exit 143, PyAutoHeart run 35372831809) before either cap could fire — consistent with the uncapped real frame exhausting runner memory, though that is inferred, not measured. Not yet profiled - SLOW-skipped to unblock the release; remove once the cost is found and fixed (PyAutoMind draft/bug/autolens_workspace/multi_galaxy_start_here_release_cost.md). no_run_census:autolens_workspace/multi_galaxy/start_here: · snapshot.no_run_census:no_run_census/rows/23 · 2026-10-08T11:46:11+00:00
autolens_workspace/weak/features/strong_lensing/a2744 [yellow]: hits the full 1800s mode=release cap in workspace-validation (PyAutoHeart run 29912642195). Script-specific: its sibling weak/real_data/a2744.py passes in 8.9s and the next-slowest weak script is modeling.py at 23.6s, so the cost is in the strong_lensing feature path rather than the A2744 dataset itself. Not yet profiled — SLOW-skipped to unblock the release (autolens_workspace#314). no_run_census:autolens_workspace/weak/features/strong_lensing/a2744: · snapshot.no_run_census:no_run_census/rows/24 · 2026-10-08T11:46:11+00:00
autolens_workspace_test/cluster/visualization [yellow]: per-plane critical-curve + caustic computation on the required full-extent 250x250 viz_grid (multi-plane marching squares) totals ~580s, over the 300s cap; same perf family as the modeling_visualization_jit scripts. Curves DO recover (plane-1 7 CC, plane-2 1 CC at full data) and the per-plane physics assertion passes — this is NOT the mislabelled "#1280 zero_contour algorithmic regression", which does not reproduce. profile_smoke.yaml unsets FAST_PLOTS/SMALL_DATASETS so it runs green in manual/full runs. no_run_census:autolens_workspace_test/cluster/visualization: · snapshot.no_run_census:no_run_census/rows/25 · 2026-10-08T11:46:11+00:00
autolens_workspace_test/imaging/subhalo_recovery_evidence.py [yellow]: evidence-grid acceptance test of PyAutoLens#672 (156-point one-shot + 25-point iterative-to-convergence scan): ~1-2h by design; run by hand, results recorded on the issue. Deliberately a manual validation script, not a smoke/release entry - its 300s-scale sibling imaging/subhalo_recovery.py stays the automated gate. no_run_census:autolens_workspace_test/imaging/subhalo_recovery_evidence.py: · snapshot.no_run_census:no_run_census/rows/26 · 2026-10-08T11:46:11+00:00
autolens_workspace_test/imaging/visualization/modeling_visualization_delaunay_jit [yellow]: now FAILS its own JIT-cache assertion at 52s (cached 2.45s not < 0.5x compile 2.72s) - possible closure cache-busting; see bug prompt jit_cache_not_hit_modeling_visualization no_run_census:autolens_workspace_test/imaging/visualization/modeling_visualization_delaunay_jit: · snapshot.no_run_census:no_run_census/rows/15 · 2026-10-08T11:46:11+00:00
autolens_workspace_test/imaging/visualization/modeling_visualization_jit [yellow]: re-measured: times out at the 300s cap; JIT + full visualization pipeline no_run_census:autolens_workspace_test/imaging/visualization/modeling_visualization_jit: · snapshot.no_run_census:no_run_census/rows/0 · 2026-10-08T11:46:11+00:00
autolens_workspace_test/imaging/visualization/modeling_visualization_rectangular_jit [yellow]: now FAILS its own JIT-cache assertion at 36s (cached 2.21s not < 0.5x compile 2.28s) - same as delaunay variant no_run_census:autolens_workspace_test/imaging/visualization/modeling_visualization_rectangular_jit: · snapshot.no_run_census:no_run_census/rows/16 · 2026-10-08T11:46:11+00:00
autolens_workspace_test/interferometer/visualization/modeling_visualization_jit [yellow]: historically >300s cap; local re-measurement OOM-killed (SIGKILL) at 132s - memory-heavy interferometer JIT no_run_census:autolens_workspace_test/interferometer/visualization/modeling_visualization_jit: · snapshot.no_run_census:no_run_census/rows/27 · 2026-10-08T11:46:11+00:00
autolens_workspace_test/misc/database/scrape/multi_analysis_jax [yellow]: re-measured: times out at the real 300s cap (full search, no test mode) no_run_census:autolens_workspace_test/misc/database/scrape/multi_analysis_jax: · snapshot.no_run_census:no_run_census/rows/1 · 2026-10-08T11:46:11+00:00
autolens_workspace_test/misc/database/scrape/slam_general_jax [yellow]: re-measured: times out at the real 300s cap (full search, no test mode) no_run_census:autolens_workspace_test/misc/database/scrape/slam_general_jax: · snapshot.no_run_census:no_run_census/rows/2 · 2026-10-08T11:46:11+00:00
autolens_workspace_test/misc/database/scrape/slam_multi_one_by_one_jax [yellow]: re-measured: times out at the real 300s cap (full search, no test mode) no_run_census:autolens_workspace_test/misc/database/scrape/slam_multi_one_by_one_jax: · snapshot.no_run_census:no_run_census/rows/3 · 2026-10-08T11:46:11+00:00
autolens_workspace_test/misc/database/scrape/slam_pix_jax [yellow]: re-measured: times out at the real 300s cap (full search, no test mode) no_run_census:autolens_workspace_test/misc/database/scrape/slam_pix_jax: · snapshot.no_run_census:no_run_census/rows/4 · 2026-10-08T11:46:11+00:00
autolens_workspace_test/multi_dataset/jax_likelihood/delaunay.py [yellow]: hung 1805s in release-integrate run 33177898708 (second vmap call, post-compile) with XLA_FLAGS=--xla_cpu_multi_thread_eigen=false in force: XLA FftThunk/ducc0 Eigen-pool deadlock (epic xla-cpu-eigen-pool-deadlock); restored 2026-08-27 by PyAutoFit#1528 on 24/24, one hang since. Smoke run passed it in 18.9s the same day. no_run_census:autolens_workspace_test/multi_dataset/jax_likelihood/delaunay.py: · snapshot.no_run_census:no_run_census/rows/17 · 2026-10-08T11:46:11+00:00
autolens_workspace_test/multi_dataset/jax_likelihood/delaunay_mge.py [yellow]: hung twice in 30 min with XLA_FLAGS=--xla_cpu_multi_thread_eigen=false in force: 1805s at the release cap in release-integrate run 33230028707 (vmap batching trace at delaunay_mge.py:201; the inner pjit under _pjit_batcher never returned, heartbeat still logging "still compiling" at 1770s, then Fatal Python error: Aborted in pxla.py __call__), and 305s at the smoke cap in workspace-smoke run 33229145647. Same signature as the two entries above: XLA FftThunk/ducc0 Eigen-pool deadlock (epic xla-cpu-eigen-pool-deadlock), third family member re-quarantined. Intermittent like its siblings — it passed the 00:58 smoke run 33225081035 the same night. Also on the smoke gate (smoke_tests.txt), which this file does not apply, so smoke runs can still hit the hang until the epic lands. no_run_census:autolens_workspace_test/multi_dataset/jax_likelihood/delaunay_mge.py: · snapshot.no_run_census:no_run_census/rows/18 · 2026-10-08T11:46:11+00:00
autolens_workspace_test/multi_dataset/jax_likelihood/shared_preloads.py [yellow]: hung 1805s in release-integrate run 33220882817 (vmap trace at shared_preloads.py:211 _assert_two_band_shared_jit; the inner pjit executing under the batching trace never returned, heartbeat still logging "still compiling" at 1740s) with XLA_FLAGS=--xla_cpu_multi_thread_eigen=false in force: XLA FftThunk/ducc0 Eigen-pool deadlock (epic xla-cpu-eigen-pool-deadlock). Same family and same signature as the delaunay.py entry above; restored 2026-08-27 by PyAutoFit#1528 on 24/24 (43.9s/43.8s in the family re-time below), one hang since. Stays on the smoke gate (smoke_tests.txt), which deliberately does not apply this file. no_run_census:autolens_workspace_test/multi_dataset/jax_likelihood/shared_preloads.py: · snapshot.no_run_census:no_run_census/rows/19 · 2026-10-08T11:46:11+00:00
Local observation (private details) [yellow]: Inspect full evidence on the dev box; private paths omitted worktree_drift:private:08387b97bf20d079 · devbox.sections.worktree_drift · 2026-10-04T15:53:53.236071+00:00
Local observation (private details) [yellow]: Inspect full evidence on the dev box; private paths omitted worktree_drift:private:166bf26a34bfed3b · devbox.sections.worktree_drift · 2026-10-04T15:54:04.090281+00:00
Local observation (private details) [yellow]: Inspect full evidence on the dev box; private paths omitted worktree_drift:private:8352f69b089b8009 · devbox.sections.worktree_drift · 2026-10-04T15:53:53.236071+00:00
Local observation (private details) [yellow]: Inspect full evidence on the dev box; private paths omitted worktree_drift:private:897f368d0790f81d · devbox.sections.worktree_drift · 2026-10-04T15:53:53.236071+00:00
Local observation (private details) [yellow]: Inspect full evidence on the dev box; private paths omitted worktree_drift:private:9207cb1105f3294d · devbox.sections.worktree_drift · 2026-10-04T15:53:53.236071+00:00
Local observation (private details) [yellow]: Inspect full evidence on the dev box; private paths omitted worktree_drift:private:dd1d43986e1feb4d · devbox.sections.worktree_drift · 2026-10-04T15:54:04.090281+00:00
Local observation (private details) [yellow]: Inspect full evidence on the dev box; private paths omitted worktree_drift:private:df8459121a39227a · devbox.sections.worktree_drift · 2026-10-04T15:54:04.090281+00:00
Local observation (private details) [yellow]: Inspect full evidence on the dev box; private paths omitted worktree_drift:private:e4d9c1f001e84a64 · devbox.sections.worktree_drift · 2026-10-04T15:54:04.090281+00:00
Local observation (private details) [yellow]: Inspect full evidence on the dev box; private paths omitted worktree_drift:private:f81665099a921553 · devbox.sections.worktree_drift · 2026-10-04T15:54:04.090281+00:00
Local observation (private details) [yellow]: Inspect full evidence on the dev box; private paths omitted worktree_drift:private:fcedc7afa406c789 · devbox.sections.worktree_drift · 2026-10-04T15:53:53.236071+00:00
Use the health skill. clear the Heart's 1 evidence gap(s) — re-run the checks named below; never change code to clear one:
1. release validation stale: source moved since rehearsal (PyAutoFit, PyAutoGalaxy, PyAutoLens) → dispatch a release rehearsal with the release skill with `rehearse`, then `pyauto-heart validate --ingest <artifacts>` (main moved since the last rehearsal)
Then run `pyauto-heart tick && pyauto-heart readiness` and report the new verdict.
Evidence gaps (1)
1
These checks have not run, have expired, or do not cover the current source. Refreshing evidence may reveal failures.
release validation stale: source moved since rehearsal (PyAutoFit, PyAutoGalaxy, PyAutoLens)