Daily Report — 2026-06-12
Daily Overview
- What was done: Implemented sampled eval set and Fisher exact testing for ErrorRecoveryBenchmark and applied a 58-bug remediation plan to the Gadget research module.
- How it was done: Used Claude Code with Opus 4.8, test-first development, multi-agent workflow orchestration, and manual verification of critical fixes.
- Impact: Established statistically valid evaluation for the core recovery hypothesis and eliminated root causes of crash/resilience failures in the research pipeline.
Aligned ErrorRecoveryBenchmark evaluation protocol with authoritative spec (M16) and remediated 58 bugs in the Gadget research module.
Tasks
Architecture & Strategy
- ✅ ErrorRecoveryBenchmark Eval Protocol Alignment — Built eval_set_sampler.py and eval_metrics.py to implement M16: stratified sampling from 135k scenes, Fischer exact testing for recovery vs normal SR, and per-RBG aggregation. Integrated with 4 existing eval scripts without regression.
- ✅ Gadget Research Module Bug Remediation — Executed a plan to fix 58 bugs (1 HIGH, 23 MEDIUM, 34 LOW), focusing on exception handling in common/llm.py, cache poisoning, and configuration mismatches.
- 🔄 ErrorRecoveryBenchmark Tianhe2 Deployment — Committed changes, pushed to GitHub, identified the ~135k validation scene pool on Tianhe2, and initiated background sampler execution.
Implementation & Fixes
- ✅ Gadget Entry Point Fix (Item 18) — Fixed the broken ‘python -m research.scout’ entry point by updating main.py sys.path handling.
- ✅ Documentation Sync — Updated docs/ecl/gadget-features.yaml to reflect removal of forced English system prompt and corrected DEFAULT_LANGUAGE notes.
Problems & Solutions
Critical Issues
1. HIGH bug in research module: call_llm_raw allowed TimeoutExpired and other exceptions to escape, crashing the entire pipeline.
Solution: Wrapped all backend calls in a try/except block that converts them to RuntimeError as per the documented contract.
Key Insight: Shared utility functions must strictly adhere to their error contracts; partial exception handling is risky.
2. Cache poisoning: Failed LLM results were saved to permanent caches with no TTL, causing persistent failures on re-runs.
Solution: Added quality gates (_screening_is_usable/_deep_eval_is_usable) before cache writes and invalidated bad entries.
Key Insight: Caches must validate data integrity before persisting; failure states should never be cached without expiation.
General Issues
3. Entry point ‘python -m research.scout’ failed due to sys.path resolution issues.
Solution: Updated main.py to explicitly add both research/ and repo root to sys.path.
Key Insight: Module entry points in nested packages often require explicit path manipulation for import resolution.
4. Model selection failure ‘claude-fable-5’ not found in Gadget project logs.
Solution: User switch to Opus 4.8 resolved the issue, confirming the model name was invalid or inaccessible.
Key Insight: Verify model names via /model command before heavy usage; some preview/internal model names may be deprecated.
5. Workflow missed assigning Item 18 (entry point fix) to a file-group agent.
Solution: Manual override and direct edit completed the gap in the automated workflow.
Key Insight: Automated multi-agent workflows require manual auditing for scope completeness; edge cases can be overlooked.
Human vs AI Approaches
Strategic Level
Evaluation Protocol Misalignment
| Role | Approach |
|---|---|
| Human | User clarified that the authoritative protocol (M16) requires SR + Fischer testing on a sampled set, rejecting the AI’s initial focus on paper metrics like gap/CI. |
| AI | AI was initially focused on statistical methods from papers (Bootstrap CI, gap) and missed the operational requirement of the specific sampling strategy. |
Difference Analysis: Human provided strategic direction based on project documentation (Panorama Spec), correcting AI’s tendency to over-rely on academic literature rather than internal specs.
Implementation Level
Default Language Intent
| Role | Approach |
|---|---|
| Human | User specified that Chinese is the intended default language, despite code defaults to English. |
| AI | AI initially noted the discrepancy but proceeded to fix the code mismatch as a separate item, potentially delaying the core intent resolution. |
Difference Analysis: Human directly established business rule; AI treated it as a configuration consistency issue rather than a primary directive.
AI Limitations
Critical Limitations
- AI initially misinterpreted the evaluation protocol requirements by focusing on academic paper metrics instead of the internal M16 spec.
General Limitations
- AI missed Item 18 (entry point fix) in the multi-agent workflow assignment, requiring manual intervention to complete the bug list.
- Model name ‘claude-fable-5’ was invalid/unknown, causing immediate failure in initial Gadget project logs.
Learnings
Key Learnings
- Cache poisoning is a critical resilience issue; failure results should never be persisted in permanent caches without validation.
- Shared library functions (like common/llm.py) must enforce strict error contracts; partial exception handling causes cascading failures.
Practical Learnings
- Multi-agent workflows for bug remediation are efficient but require manual scope auditing to catch unassigned items.
Conversation Summaries
ErrorRecoveryBenchmark
✅ Eval Protocol Implementation 03:13:54.085 | claude_code User instructed AI to align evaluation metrics with the M16 protocol using Fischer exact testing and stratified sampling from 135k validation scenes. AI implemented eval_set_sampler.py and eval_metrics.py, updated 4 existing scripts, and verified all tests passed green (55/55).
• Tianhe2 Deployment and Sampling 03:20:00.994 | claude_code AI committed changes, pushed to GitHub, identified the ~135k validation pool on Tianhe2, and launched background sampling process. The session ended while monitoring the long-running sampler.
Gadget
✅ Research Module Bug Remediation 03:09:06.315 | claude_code AI executed a 58-bug remediation plan for the research module, including fixing high-severity exception handling in common/llm.py, resolving cache poisoning issues, and correcting configuration mismatches. A multi-agent workflow was used, followed by manual verification and a missing entry point fix.