Statistics: keeping every FEAT-024 family exact¶
Part of the temporary reversible deletion plan. Background: the
statistics reference and the
compatibility rules in .claude/rules/materialized-stats.md. The FEAT-024 owner should review this part
before slices S4a, S4b and S6 start.
1. What "exact" means here¶
A page either gets a stored count that equals the live calculation, or it falls back to the live
calculation. It is never served a stored count that has missed a change. Deletion keeps that true with
the mechanisms FEAT-024 already has: an operation fence before the source change, release as
Stale after it, a rebuild to make stored counts servable again, and the per-family parity audit
(GET api/admin/project-statistics/{id}/parity?family=<key>) as the proof.
Because nothing is ever erased, delete and restore are the only statistics events. A deleted search's Studies are recounted out when it is deleted and back in when it is restored. A deleted project's stored statistics are kept, not served, and served again when it is restored.
2. Why the recommended data model suits FEAT-024¶
- Delete and restore move a search's Studies out of or into the project. For the statistics that is
the same population change the old immediate deletion made (and that a search import makes in the other
direction), so the existing deletion declaration (
ProjectScreeningBulkOperation.DeletionNamespace) applies once it is fixed (§3). - Detached Studies stay outside every scope of the project for as long as they are deleted, because
they carry a holding
ProjectId. Nothing else ever needs to touch the project's statistics on their account. - A deleted project needs no Study writes.
ProjectStatisticsControl.IsProjectDeletedalready makes the serving gate, the bundle reader, the backfills, the repair, the daily observation recorder and seven query paths treat the project as unavailable, and makes the fold refuse moves for it (ProjectStatisticsFoldApplier.cs:138-146). Nothing writes the flag today; project deletion becomes its first writer.
3. Fixes the existing declaration needs¶
| # | Fix | Why |
|---|---|---|
| 1 | Add QuestionAnswers to the deletion and restore fence: per question scope when the project has at most 100 questions with answers, otherwise the family-wide stale (the same threshold the save path uses) | A deleted search's Studies carry annotations, so the answer tallies change. The current set omits it (#3849, QuestionAnswers "Known gaps") |
| 2 | Key the fence on the deletion operation id, not the search id: OperationId(DeletionNamespace, projectId, operationId), and a separate RestorationNamespace for restores |
A search can be deleted, restored and deleted again; a search-id key would collide with its earlier, released fence |
| 3 | Advance ClientInvalidationRevision in the release transaction |
The import-reveal rule: subscribed clients must not stay on the old population |
| 4 | Move only Studies with no pending fold entries; retry the rest after the fold drains them | Only the fold worker removes entries, and it finds them through the ProjectId-keyed pending index. A Study moved with entries would strand them under the holding id, and on restore they would come back into the overlay |
| 5 | On restore, re-persist derived screening fields of the returned Studies and remove answers to deleted questions inside the restore's own fence, before release | The agreement settings may have changed while the search was deleted. Families that read stored derived fields refuse to rebuild over drift (InclusionStatusDrift). Calling the admin re-persist route would try to admit a second fence and be refused FamilyFenced, so the restore runs the same logic under its own fence |
| 6 | Admit the fence only after every cancelled or awaited job is terminal and has released its own fence, and make sure no cancelled job can admit again afterwards (in-flight work §6) | Cancelled imports and bulk updates own fences over the same families. Since #4062 a late admission under a reused operation id re-activates a cleared fence, which would leave the project fenced with no owner after the deletion's release |
| 7 | Request the rebuilds after release (bounded, in the runner), where statistics writes are on for the project | Exactness does not depend on this (Stale rows are never served), but without it pages stay on the slower live calculation until the next repair run or a manual backfill |
Status of the fixes (as built)¶
Fixes 1, 2, 3 and 7 are settled by slice S4a-1 (fix 7 by relying on the scheduled repair rather than an inline
request; see below) in StagedSearchDeletionStatisticsFence
(ISearchDeletionStatisticsFence, Core Services/Deletion), a thin owner over the existing staged operation
fence (IProjectStatisticsStagedOperationFenceService). It is dark: only the runner's sweep (S4a-2) calls it, for
a search deletion created with deletionLifecycle on. With statistics off for the project its admission reads and
writes nothing, and its release writes nothing unless the deletion still holds a fence it admitted while statistics
were on, which it then releases (#4288). The flag-off legacy deletion keeps its own search-id-keyed fence and is unchanged.
- Fix 1 (QuestionAnswers). Decided from the questions the project defines
(
QuestionAnswerTallyRefresh.QuestionIds, system questions included) rather than from a Study scan. That set is a superset of the questions with answers, and the family's scopes are exactly the defined questions with a tally row, so it is still exact. At mostAnnotationStatisticsClassifier.MaximumQuestionScopeStales(100) questions: one fence per question scope. Above it: the family-wide stale (one family write-epoch advance, which makes every question row non-servable) plus one fence at the project scope, which owns the family guard for the whole deletion (the reader, the backfills, the point path and the fold all refuse on the guard's fence count). A project that defines no question has no QuestionAnswers row and fences none. A project with more than 100 defined questions but at most 100 answered ones takes the family-wide form: still exact, only coarser. - Fix 2 (key).
OperationId(DeletionNamespace, projectId, deletionOperationId). The operation kind is the existingSearchStudiesChanged, the kind the legacy search deletion records: the kind is persisted on the fence records and the guard but nothing branches on it, and the namespace already tells the deletion's fence apart.ProjectScreeningBulkOperation.RestorationNamespace(stats.source.screening.restoration) is reserved for S4b and unused. - Fix 3 (client invalidation). The release is
ReleaseStaleAsync(sourceChanged: true), whose transaction advancesSourceInvalidationRevisionandClientInvalidationRevisionwith the Stale lifecycles. It never throws and returns whether no fence of the operation is held afterwards (released, never admitted, or statistics off with nothing held). On a failure (already logged at Error and counted infence_release_failuresby the fence service) it returns false and the families stay fenced, the fail-closed direction; the runner keeps the operationDeletingand releases again on a later tick. Another owner holding a family answersBusy(the typedFamilyFencedrefusal); a repeat admission by the lease holder answersAdmitted. - Late admissions (#4060). The admission passes an ownership guard
(
IDeletionOperationStore.GuardAdmissionAsync) that reads the operation inside the admission's own transaction and refuses unless the caller's lease token holds the unexpired lease and the operation is stillRequested; a refusal answersSupersededand writes nothing. So a runner superseded while its admission was slow or retrying cannot re-activate theClearedfence records after the new holder's release. The guard is read-only: every admission and every release writes the project's statistics control document, so an admission whose snapshot predates the takeover write-conflicts and is re-run (and refused) rather than committing after the release. - Fix 7 (rebuilds). The release does not run backfills itself; the rebuild is left to the scheduled stale
repair (
ProjectStatisticsRepair, FEAT-024 C1, enabled hourly on staging, the only environment withdeletionLifecycleon), which republishes every Stale family of an allowlisted project through the same non-forced family backfill, with its backoff, settle check and budgets. A safe bounded inline call does not exist: each family backfill is an authoritative recalculation of every scope, so its duration grows with the project and can outlive the runner's operation lease (120 s by default; a stage renews it only between its own units of work), letting another pod take the operation over (and a time-boxed variant would have to cancel backfills part-way); and fence-triggered rebuilds are deliberately off for every other fence owner (ProjectStatisticsSourceFenceRegistry,rebuilds: null) pending their own leasing and contention review. Exactness does not depend on it: until the repair runs, pages use the live calculation. As for the legacy deletion, the repair cannot retire the deleted search's SearchPopulation row when it was the project's last search (the family then declares no scope and the repair reportsNotRepairable).
As built, fixes 4 and 6 (S4a-2). Fix 4: the sweep's update requires PendingStatistics to be absent (stricter
than PendingStatistics.OldestAtUtc absent, so even a malformed set is never moved) and leaves those Studies for a
later pass. A batch or pass that leaves Studies behind calls IProjectStatisticsFoldSignal for the project (a no-op in
the project-management host today, where the one-minute fold sweep drains them). After PendingStatisticsWaitMinutes
(default 15) without progress the operation shows PendingStatisticsNotDrained and keeps retrying; it completes only
when no Study of the search is left in the project, so no Study under the holding id ever carries a pending set
(DL-R1S-09). Fix 6: the fence is admitted in the same tick as the in-flight review that found nothing outstanding,
before the first Study write, with the tick's lease token for the admission guard (a Superseded answer writes
nothing), and released before the operation is marked Deleted. A release that fails keeps the operation Deleting
(blocked StatisticsFenceReleaseFailed, no ledger record) and is retried every tick until it succeeds, so a deletion
never completes while its fence may still be held (data model §5.7).
Fix 5 (restore re-persist under the restore's own fence) lands in S4b.
4. Fence set for a search¶
For a search delete or restore in project p with statistics enabled, one operation fence covers:
- every family in
ProjectScreeningInclusionRecalculation.ProjectWideFamilies(project): ProjectScreening (project scope), MembershipScreening and ReviewerScreening (every member), StageAnnotation and DomainReconciliation (every stage), MembershipStageAnnotation and ReviewerAnnotation (every member on every stage); - QuestionAnswers (fix 1);
- SearchPopulation, the deleted search's scope only (as today's declaration does).
While it is held, pages fall back to the live calculation, other fence owners (an import, a bulk update,
another deletion) are refused with the typed FamilyFenced 409, and the backfills of these families are
refused.
5. Per-family behaviour¶
"Fence" = covered by the operation fence; "Stale" = released Stale; "Rebuild" = backfill afterwards. In every case the parity reference is the family's own live calculation, as the statistics reference defines it.
| Family | What changes when a search's Studies leave or return | Search delete | Search restore | Project delete | Project restore |
|---|---|---|---|---|---|
| ProjectScreening | Study totals and screening-state profiles | Fence → Stale → Rebuild (recounted out) | Fence → re-persist → Stale → Rebuild (recounted in) | Rows kept, not served (IsProjectDeleted) |
Rows served again; any family that changed while deleted is already Stale and is rebuilt |
| MembershipScreening | Each member's decisions and agreement on the moved Studies | Same | Same | Same | Same |
| ReviewerScreening | What each reviewer has screened or can still screen | Same | Same | Same | Same |
| StageAnnotation | Session progress per stage | Same | Same | Same | Same |
| MembershipStageAnnotation | Each member's sessions per stage | Same | Same | Same | Same |
| ReviewerAnnotation | Annotation availability and capacity per reviewer and stage, including reserved-only Studies (reservations are released by the move) | Same | Same | Same | Same |
| QuestionAnswers | Studies and answers per question | Fence (fix 1) → Stale → Rebuild | Same, after deleted-question answers are removed | Same | Same |
| DomainReconciliation | Reconciliation progress per stage | Same as StageAnnotation | Same | Same | Same |
| SearchPopulation | The deleted search's scope disappears (it counts the imported file, so other searches' scopes do not change) | Fence (that search) → Stale → Rebuild; the scope's row becomes a tombstone | The scope returns with the same count | Same as above | Same |
| DerivedSummary | Not stored; computed from the families above at read time | Follows its inputs | Follows | Follows | Follows |
A search shared with another project: only this project's families change; the other project is never touched.
6. Sequences with the statistics steps¶
Search delete: request recorded → in-flight work handled (cancelled jobs release their own fences: the
import through its failure clean-up, a bulk update through its rollback and release) → admit fence
keyed by the operation id
(own transaction) → unlink → sweep (pending-free Studies only) → release Stale + client invalidation
(StagedSearchDeletionStatisticsFence.ReleaseAsync, which never throws; a failed release leaves the families fenced,
the fail-closed direction, and the operation stays Deleting until a later tick's release succeeds) → rebuilds by
the scheduled stale repair (§3 status, fix 7).
Search restore: checks and waits → operation to Restoring → admit fence → sweep back →
re-persist derived fields and remove deleted-question answers (version-guarded saves of the returned
Studies) → relink → release Stale + client invalidation → rebuilds (as for the delete).
Project delete: in the deletion transaction, IsProjectDeleted = true and ProjectDeletedAtUtc set.
Every stored row stays exactly as it is. Nothing is served while the flag is set.
Project restore: why the kept rows can be served again. While the project is deleted, the only writes that can reach its Studies are a save already authorised a moment before the deletion and the reservation timeouts. Each of them keeps the statistics exact by the existing rules:
- on the transactional path, the write path does not check
IsProjectDeleted, so a quick update moves the stored rows in the same transaction as the source, exactly as it would for a live project, or falls back to source-only and marks the rows Stale; - on the fold path, the fold refuses the project (
FamilyNotServable), discards the entries and stales their families.
Scheduled statistics maintenance is made to skip deleted projects (slice S2), so nothing else touches
the rows. Restore therefore only clears IsProjectDeleted: rows that are still Fresh at the current
write epoch are served again at once, and every family that changed while the project was deleted is
already Stale and is rebuilt by the backfill the runner requests (backfill rebuilds only scopes that are
not current). The restore-time release of old reservations goes through the ordinary release path, so it
keeps the rows exact too. If the FEAT-024 owner prefers, the restore can instead stale every family;
that is safe but makes every page use the live calculation until the backfills finish.
7. Checks that prove it¶
| Check | When | Pass |
|---|---|---|
| Per-family parity audit, all nine families (#4048) | After every search delete and restore, once the rebuilds have run | InParity for every family; no Mismatch; Inconclusive only while a rebuild is still running |
| Parity straight after a project restore, before any backfill, for a project with no activity while deleted | After every project restore in tests and drills | InParity for every family (the kept rows were served again) |
| Parity after a project restore with activity while deleted (a reservation timeout, a late save) | Same | The changed families read Inconclusive (Stale) before the backfill and InParity after it; the others InParity at once |
Inclusion drift check (…/reviewer-screening/inclusion-status-drift) |
After every search restore | driftedStudies: 0 |
| Fence diagnostics | After every operation | No fence left owned by a deletion or restoration operation |
| Pending entries under a holding id | After every search delete (test and soak query) | Zero Studies with a holding id and a pending set |
| Backfill as oracle | After rebuilds | A second backfill answers AlreadyCurrent for every scope |
| Staging preflight (six-hourly) and the weekly drift check | Continuously on staging | No mismatch for the pilot project across deletion activity |
8. When statistics are off¶
When statistics writes are off, or the project is not on the allowlist, the fence admission is a no-op
(IsFenceRequested is false) and there are no stored counts to keep exact. A release still clears a fence the
operation admitted while statistics were on (#4288). IsProjectDeleted is written
only when the project has a statistics control row. Every deletion path must behave identically, apart
from these steps, whether statistics are on or off; tests run both ways.
9. History and checkpoints¶
Retained checkpoints are never rewritten. A history chart after a search deletion shows the drop at the next checkpoint, and earlier points still include the search; restoring shows the rise. Daily observations skip a deleted project, so its history has a gap for the deleted period.