verify-analysis

Audit and reconcile an existing tool analysis produced by codebase-map or docs-map, fixing it against (1) the actual cited evidence and (2) its own prior versions in git.

Why this skill exists

codebase-map and docs-map are non-deterministic: re-running them on the same tool produces materially different output, and coverage-table verdicts (Yes / No / Unclear) flip between runs. This skill is not another generator — it does not resample a fresh analysis. It is a checker/reconciler anchored to two stable references: the evidence on disk/URL, and the set of prior versions in git. Disagreements are resolved by re-reading the cited evidence, which is deterministic: a given file:line either supports a verdict or it does not. The goal is that repeated runs converge instead of diverge.

Determinism guardrails (apply throughout)

• Do not generate new verdicts from scratch. Only confirm or correct existing ones against cited evidence.
• A verdict may only be changed when a specific file:line/URL is read and shown to contradict it. Record that evidence in the Reconciliation Log.
• When evidence is genuinely ambiguous, prefer Unclear (leaf colour Magenta) and say so — never flip a coin. Ambiguous must stay ambiguous across reruns.
• The profile is a pure function of the final coverage table. An identical table must yield an identical profile.

Inputs

• A tool name, e.g. verify analysis CARGO. The set of analysed tools lives in evaluation/analyses/.
• Or verify all analyses — iterate the procedure over every directory under evaluation/analyses/ that contains an analysis.md.

Resolve evaluation/analyses/<tool>/analysis.md. Detect the source type by reading its header line:

• Codebase path: → codebase-sourced (ground truth = local files under ../extracted-codebases/<tool>).
• Documentation: → docs-sourced (ground truth = the cited URLs).

If the analysis does not exist on disk, stop and tell the user to run codebase-map / docs-map first — this skill reconciles existing analyses, it does not create them.

Output files

• evaluation/analyses/<tool>/analysis.md — reconciled in place (table + prose + a new Reconciliation Log).
• feature_model/representation/.profiles/<tool>.profile — re-derived from the corrected table.
• evaluation/analyses/<tool>/README.md — left untouched unless a link is broken (see Step 6).

Never modify my_paper/text.md, feature_model/representation/feature_model.xml, or any other tool’s analysis/README/profile.

Procedure

Step 1 — Gather all versions

List every historical version of the analysis (search all refs, not just current branch):

git log --oneline --all -- evaluation/analyses/<tool>/analysis.md

For each commit, read that version:

git show <sha>:evaluation/analyses/<tool>/analysis.md

Always include the current on-disk version too. If there is only one version in history, the audit reduces to checking that version’s claims against evidence (Step 3 still applies to every feature, not just disagreements).

Step 2 — Build a per-feature verdict matrix across versions

Read .claude/skills/codebase-map/references/feature-model-dimensions.md for the full leaf list. For every leaf feature, collect each version’s verdict (Yes / No / Unclear) and the evidence it cited. Tabulate them side by side.

Flag every feature where versions disagree — these are the non-determinism hotspots that need adjudication. Features where all versions agree are presumed stable, but spot-check a representative sample (and any feature whose evidence looks weak or generic) against ground truth anyway.

Step 3 — Adjudicate against ground truth (the deterministic core)

For each feature being adjudicated, work in this order:

1. Read the analysis prose / justification first, across all versions that disagree. The justification tells you what claim each version is making and what concept to go look for. Without it you risk misreading a cited file:line because you don’t know what it was meant to support. The prose frames the question.

2. Open the cited evidence with that understanding:
   • Codebase-sourced: re-open each cited file:line in ../extracted-codebases/<tool>, grep/Read around it, and check whether it actually supports the claim the prose describes. If a version cited no usable reference, search the codebase for the concept before concluding absence.
   • Docs-sourced: re-fetch the cited URL with WebFetch. If WebFetch returns an HTTP error (403/401/429/5xx) or an empty/blocked response, fall back to curl | strip_html (the same pattern as docs-map Step 1) and do not retry WebFetch for that domain. Confirm the quote exists and supports the claim.
3. Decide. The evidence answers the question the prose posed. The file/URL only overrides the prose when the two genuinely conflict — the analysis cannot assert something the code/docs contradict. This does not replace reading the prose.

Resolve each feature to a single correct verdict plus the strongest evidence reference, applying the evidence-precedence and confidence rules the existing skills define:

• Code reference > documentation reference; cite the artifact (CSV/AST node/solver object) over a README quote when both exist.
• Unclear (→ Magenta) for indirect/vague evidence; No (→ Yellow/Red) only when you are confident the feature is absent.
• Also re-check cross-tree constraint violations (see the dimensions reference; the common one is Community detection → NOT Expert knowledge).

Step 4 — Rewrite analysis.md so the whole document is consistent

Update the file so prose and table agree on the reconciled verdicts:

• Coverage Table — corrected verdicts + strongest evidence per leaf.
• Filled Analysis Template — prose adjusted to match (template structure: .claude/skills/codebase-map/references/analysis-template.md).
• Observations — Model Gap Candidates, Other features outside the model, Model Vocabulary Fit, Cross-tree constraint violations, Noteworthy design decisions — adjusted to match.

Keep section structure intact so diffs stay reviewable. Update the Date: header to today.

Append a new section at the end:

## Reconciliation Log

Date: <today's date>
Versions compared: <list of commit SHAs + "working tree">

| Feature | From → To | Adjudicating evidence | Which version(s) were wrong |
|---------|-----------|-----------------------|-----------------------------|
| K-means++ | Unclear → Yes | `src/cluster/kmeans.py:31` | run cc53285 marked Unclear |
| ... | ... | `file:line` or `<url> — "quote"` | ... |

Features still genuinely Unclear after adjudication: <list, with why>.

If no verdict changed, still write the log with a row-free table and the note “No verdicts changed; all claims confirmed against evidence.” and use the project’s no-change date convention.

Step 5 — Re-derive the .profile from the corrected table

Rewrite feature_model/representation/.profiles/<tool>.profile as a pure function of the final coverage table. Use the exact colour scheme and group-propagation rules from codebase-map/docs-map Step 8.

Leaves:

Dark_Green is valid for group nodes only, never leaves.

Groups (propagate up, priority Red > Orange > Magenta > Light_Green > Yellow):

1. Evaluate each child first.
2. Mandatory group with any mandatory child Red → Red.
3. Else any child Orange → Orange.
4. Else if any mandatory child is Magenta — or, for an OR group, the only present child(ren) are Magenta (every confirmed branch is absent/Red) → Magenta. Uncertainty in a required child propagates up; confirmed sibling leaves do not rescue the group.
5. Else ≥1 mandatory child confirmed present (Light_Green/Dark_Green) — or, for an OR group, ≥1 child confirmed present — and no mandatory child missing or Magenta → Light_Green (OR) or keep evaluating (AND).
6. AND group with all mandatory children fully satisfied and no violations → Dark_Green. A child is fully satisfied when it is a leaf that is present (Light_Green — a leaf’s max, never Dark_Green) or a group that is itself Dark_Green. A mandatory child group that is only Light_Green (partial) caps the AND parent at Light_Green. (Optional children that are absent or Magenta do not block Dark_Green.)
7. Optional group with zero present children → Yellow.

Format: flat key=value text; first line is false; feature names must match exactly the leaf and group names in feature_model/representation/feature_model.xml. Include every leaf and group. End with the legend:

featureColorMeaning0=Leaf/Group: absent or broken (mandatory)
featureColorMeaning1=Constraint violation
featureColorMeaning2=Leaf/Group: absent (optional)
featureColorMeaning3=Group: fully satisfied
featureColorMeaning4=Leaf: exists / Group: partially satisfied
featureColorMeaning5=Extra Feature
featureColorMeaning6=InvalidColor (unused in FeatureIDE)
featureColorMeaning7=InvalidColor (unused in FeatureIDE)
featureColorMeaning8=Leaf: unclear / indirect evidence
featureColorMeaning9=InvalidColor (unused in FeatureIDE)

Step 6 — README.md (touch only if a link breaks)

Leave evaluation/analyses/<tool>/README.md untouched. Verify its Relevant-Files / source links resolve; if one is now broken, fix only that link.

Step 7 — Report to the user

Summarise:

• How many versions were compared and how many features disagreed.
• Which features’ verdicts changed (from → to) with the adjudicating evidence.
• Features still genuinely Unclear after adjudication.
• Any cross-tree constraint violations.
• Paths to the modified files.

For verify all analyses, report a per-tool summary line plus the totals.

Reference files

• .claude/skills/codebase-map/references/feature-model-dimensions.md — full feature model schema with cross-tree constraints (reused, not duplicated).
• .claude/skills/codebase-map/references/analysis-template.md — the analysis template structure (reused).
• feature_model/representation/feature_model.xml — canonical feature model (exact leaf and group names for the profile).
• feature_model/representation/.profiles/ — existing profiles for format reference.