Daily Report — 2026-05-07
Daily Overview
- What was done: Prepared ErrorRecoveryBench codebase and 30GB dataset for NeurIPS 2026 Evaluations & Datasets track submission, corrected critical paper data errors, and generated all required figures.
- How it was done: Generated Croissant JSON-LD metadata, downloaded data via SSH/scp from tianhe server, cross-verified paper tables against fact-check JSON files, and iteratively refined matplotlib figures to meet publication standards.
- Impact: Ensured submission compliance with NeurIPS 2026 mandatory requirements, fixed severe data inaccuracies that would have invalidated results (BC-RNN Normal SR off by 167%), and completed all camera-ready figures.
Prepared ErrorRecoveryBench for NeurIPS 2026 double-blind submission: generated Croissant metadata, downloaded 30GB dataset, fixed major data errors in paper tables (BC-RNN Normal SR 21%→56%), and created publication-quality figures.
Tasks
Architecture & Strategy
- ✅ Correct critical BC-RNN data errors in paper Table 6 — Fixed BC-RNN Normal SR from 21% to 56%/42% (V1/V2), split single BC-RNN row into two (Nominal FT + Recovery), and corrected Pi 0.5 Recovery Normal SR from 56% to 60% based on fact-check verification
- ✅ Generate Croissant metadata for NeurIPS 2026 submission — Created croissant.json with dataset description, 4 FileObjects (error_scenes/recovery_demos_human/augmented/mimicgen_prepared), 2 RecordSets, and RAI fields to meet NeurIPS E&D track mandatory requirement
- ✅ Download 30GB release_data from tianhe server — Used scp to download complete dataset (20958 files, 30GB) from /XYFS01/…/release_data to local machine for Hugging Face hosting
- ✅ Generate publication-quality figures (Fig 4/5/6/7) — Created Fig 4 (stack per-subtype heatmap), Fig 5+6 combined (RBG bar + Normal vs Recovery scatter), Fig 7 (D0 vs D1 comparison) with scientific blue color scheme and Arial font
- 🔄 Diagnose MimicGen segmentation and filter failures — Investigate why step logic mismatch causes low success rates, filter reduces training data imbalance (pick 160 / stack 175 / coffee 85 / serving 32), and RPG cross-step transfer <5%
Implementation & Fixes
- ✅ Fix README.md Environment Setup section — Completely rewrote README sections polluted by terminal spinner garbage, added proper bash code blocks and installation commands
Problems & Solutions
Critical Issues
1. BC-RNN Normal SR severely wrong in paper (21% vs actual 56%)
Solution: Cross-verified all table values against fact-check JSON from validation runs, corrected to V1=56% / V2=42%, split into two policy rows
Key Insight: Fact-checking against raw experiment outputs is critical; this error would have invalidated paper results (167% discrepancy)
2. Kirk questioned whether to use per-task cherry-picked checkpoints or unified checkpoint
Solution: User confirmed per-task best RSR approach: pick=step 19000 (31.7%), stack=19999 (76.7%), coffee=16000 (54.5%), etc.
Key Insight: Checkpoint selection strategy must be explicitly documented to handle reviewer questions about ‘which checkpoint was used’
3. Initial matplotlib figures had unprofessional colors and overlapping labels
Solution: Iteratively refined to scientific blue gradient colormap, unified Arial font, adjusted layout to eliminate overlap (Fig 4 horizontal, Fig 5+6 merged)
Key Insight: Publication-quality figures require multiple iterations and domain-specific aesthetic standards
General Issues
4. README.md Environment Setup polluted by terminal spinner output
Solution: Completely rewrote sections with clean markdown, proper code blocks, and actual installation commands
Key Insight: Automated log capture can corrupt documentation files; always validate generated content
5. SSH host key changed preventing server access
Solution: Removed old key with ssh-keygen -R [121.46.19.6]:6688 and reconnected with StrictHostKeyChecking=accept-new
Key Insight: Server infrastructure changes require proactive known_hosts maintenance
Human vs AI Approaches
Strategic Level
Experimental data accuracy requirements
| Role | Approach |
|---|---|
| Human | Kirk insisted ‘must use measured data not estimates’, repeatedly checked data source consistency (80 vs 168 samples), demanded fact-check on all tables |
| AI | Claude initially used outdated BC-RNN values from LaTeX without verification, needed user correction to cross-check against JSON fact-check files |
Difference Analysis: Human enforced rigorous data provenance standards; AI relied on existing document content without independent validation
MimicGen RPG design validity
| Role | Approach |
|---|---|
| Human | Kirk questioned core design goal and pointed out that if transfer fails, entire Table 3 design becomes meaningless; demanded concrete success cases |
| AI | Thomas defended theoretical feasibility but couldn’t provide cross-step success evidence (<5% rate), focused on implementation details rather than fundamental assumptions |
Difference Analysis: Human (Kirk) applied systems-level critique exposing architectural risks; student struggled with defensive reasoning lacking empirical validation
Detection of step logic mismatch in MimicGen
| Role | Approach |
|---|---|
| Human | Thomas identified that old contact-detection segmentation no longer applies, causing per-segment success rate collapse |
| AI | AI did not proactively notice this algorithm iteration compatibility issue |
Difference Analysis: Human detected subtle logic incompatibility from experimental failure patterns; AI lacks domain intuition for robotic manipulation pipeline debugging
AI Limitations
Critical Limitations
- Failed to independently validate paper table values against experiment outputs, trusted existing LaTeX content until user pointed out 167% error in BC-RNN Normal SR
- Could not anticipate architectural flaws in RPG design that Kirk immediately identified as fundamental risks
- Initial figure aesthetics (garish colors, overlapping labels) did not meet scientific publication standards, requiring multiple user-guided iterations
General Limitations
- Cannot directly access remote server data files, requiring user to manually SSH and provide file listings
Learnings
Key Learnings
- Fact-check validation against raw experiment JSONs is critical; paper had BC-RNN Normal SR error of 167% (21% vs 56%) that would invalidate results
- NeurIPS 2026 E&D track mandates Croissant JSON-LD metadata + small sample subset for datasets >4GB
- Per-task checkpoint cherry-picking maximizes RSR but requires unified explanation for reviewers asking ‘which checkpoint’
- Double-blind submission requires strict identity isolation (cannot push from personal account TzJ2006 to anonymous ErrorRecoveryBenchmark org)
- Publication figures require scientific color palettes (blue gradients), unified typography (Arial), and zero-overlap layout
Conversation Summaries
ErrorRecoveryBench
✅ NeurIPS 2026 double-blind submission prep 00:48:05.361 | claude_code Generated Croissant metadata file (mandatory for NeurIPS E&D track), investigated 30GB dataset structure on tianhe server, fixed README.md Environment Setup section polluted by terminal garbage. Discussed 4GB threshold rule requiring small sample subset for reviewers.
🔍 Kirk meeting notes archival 03:52:30.961 | claude_code Saved meeting summary documenting MimicGen issues: step segmentation logic mismatch, filter causing data imbalance (pick 160/stack 175/coffee 85/serving 32), RPG cross-step transfer <5% success rate, pmplace augmentation performance drop, BCRN GPU interruption. Listed action items: check filter logic, fix pmplace, monitor stackthree training, add paper figures.
ErrorRecoveryBenchData
✅ Dataset download from tianhe server 00:56:20.711 | claude_code Downloaded 30GB (20958 files) release_data from tianhe:/XYFS01/…/tangzijia/release_data via scp. Handled SSH host key change. Confirmed dataset exceeds 4GB threshold requiring small sample for NeurIPS reviewers.
NIPS 2026 Error Recovery
✅ Paper table data correction 08:00:56.210 | claude_code Corrected critical errors in main results table based on fact-check verification: BC-RNN Normal SR 21%→56% (V1)/42% (V2), Pi 0.5 Recovery Normal SR 56%→60%, split BC-RNN into two rows. User confirmed per-task cherry-pick strategy using best RSR checkpoint per task.
✅ Publication figure generation 05:55:14.684 | claude_code Generated Fig 4 (stack per-subtype RSR heatmap, horizontal layout), Fig 5+6 combined (RBG bar + Normal vs Recovery scatter), Fig 7 (D0 vs D1 comparison). Iteratively refined color scheme to scientific blue palette, unified Arial font, adjusted layout to eliminate label overlap. Evaluated 4 Fig 4 layout variants (grouped bar, radar, merged heatmap, stacked).