Daily Report — 2026-09-19

Daily Overview

• What was done: Addressed critical quantization errors in the $\pi$0.5 VLA model and established a structured ‘Oracle visual prompt’ training framework for 16 RoboMME tasks to evaluate visual memory augmentation.
• How it was done: Utilized ablation studies and video analysis to fix RMSNorm quantization ranges via AIMET TF-Enhanced, and developed an Oracle-driven sidecar system to generate high-fidelity training pairs using scene-anchored and robustly placed visual markers.
• Impact: Restored $\pi$0.5 performance from 0% to near-perfect success rates for sensitive tasks and provided a scalable, training-ready pipeline for unified visual memory representation.

MacOS

• What was done: Minimal exploratory testing and environment setup.
• How it was done: Standard terminal usage and command investigation.
• Impact: Provided initial task context.

TzJsDesktop

• What was done: Core research, robot policy training, and visual prompt design.
• How it was done: Executed large-scale GPU training, performed side-by-side video audits, and implemented the ‘oracle_v2’ drawing rule system.
• Impact: Successfully launched the ‘all16_oracle’ training run and stabilized the visual prompt generation logic.

lighthouse

• What was done: Large-scale evaluation and potential GPU rendering support.
• How it was done: Running heavy simulation/evaluation suites on GPUs.
• Impact: Provided a high-performance environment for critical validation.

Successfully resolved W4A4 quantization failures in the $\pi$0.5 model through RMSNorm optimization and transitioned the RoboMemory project to a robust Oracle-based visual prompt training pipeline for 16 tasks.

Tasks

Architecture & Strategy

• ✅ Quantization Debugging & Root Cause Analysis — Identify why W4A4 quantization causes 0% success in libero_goal/middle-drawer tasks by analyzing RMSNorm output noise.
• ✅ TF-Enhanced Quantization Pipeline Integration — Use AIMET TF-Enhanced to rebuild the backbone checkpoint with optimized norm ranges to fix quantization failures.
• ✅ Uniform-A4 Candidate Verification — Run 50-episode retest of the hybrid A4 candidate to verify the fix.
• ✅ Oracle Visual Prompt Specification (16 Tasks) — Define concise, trainable visual prompt rules (icons, rings, badges) for all 16 RoboMME tasks.
• 🔄 Oracle Sidecar Data Generation — Generating 1600+ training pairs using the new oracle_v2 drawing rules across multiple GPU shards.
• 🔄 all16_oracle Policy Training — Training the $\pi$0.5 policy on the new oracle-provided visual prompts using GPUs 4-7.
• ✅ RoboMME/pi0.5 Technical Audit — Investigate RoboMME evaluation protocols and the exact pi0.5 training recipe (frozen SigLIP, full LLM fine-tuning).
• • Gemini_V2_Full_Run — 16 tasks * ep0, Memory mode, 470 calls, ~15 mins via API.

Implementation & Fixes

• 🔄 Final Deployment Validation — Complete 50-episode testing for the official TF-Enhanced checkpoint and verify latency on AI Hub.
• • MoveCube/InsertPeg Variant Rendering — Render typical mp4s for MoveCube (hook/pick/push) and InsertPeg (4 combinations).
• 🔄 TokenMonitor SSH Sync Bug Fix — Investigating why SSH usage data fails to synchronize across devices due to archive boundary issues.

Problems & Solutions

Critical Issues

1. W4A4 quantization leads to 0% success in ‘middle-drawer’ tasks (misidentifying targets).

Solution: Isolated 35 RMSNorm output tensors and applied AIMET’s TF-Enhanced range setting to mitigate noise from outliers.

Key Insight: The failure was caused by min-max quantization ranges being stretched by outliers in uniformly distributed activation tensors, injecting ~10% noise per layer.

2. NVML/GPU invisibility error and Training/Test mismatch.

Solution: Cleared orphaned processes to recover GPU access; restarted training from scratch to ensure policy learns new bottom-right/scene-anchored distributions.

3. Gemini’s visual prompts/scoring were inconsistent (early counting, incorrect identity tracking).

Solution: Shifted priority from fixing Gemini’s errors to using a ground-truth ‘Oracle’ system to generate perfect visual prompts for training the robot.

Key Insight: The ‘Target Hit Rate’ is a dangerous metric for memory tasks as it conflates object detection with actual memory of history.

4. Ambiguity regarding $\pi$0.5 training methodology (LoRA vs Full Fine-tuning).

Solution: Inspected code to confirm full-parameter fine-tuning of the LLM and action expert while keeping the SigLIP visual tower frozen.

General Issues

5. Visual elements (icons/badges) were being cut off by robot random cropping/rotation.

Solution: Moved coordinates from (3,3) to (16,16)/(26,26) to ensure visibility under 95% random crop scenarios.

Key Insight: Visual prompts must be ’training-robust’ to survive augmentation.

6. Evaluation script left orphan server processes consuming ~230GB VRAM.

Solution: Modified cleanup logic to terminate the entire process tree/PIDs to ensure GPU recovery.

Key Insight: Standard process termination often fails to clean up child processes in complex environments.

Human vs AI Approaches

Strategic Level

Research Direction & Methodology

Role	Approach
Human	Rejected the idea of ‘improving Gemini first’; insisted on defining a minimal, trainable visual prompt set and using video analysis to prove the divergence point.
AI	Initially suggested low tolerance/calibration issues and focused on perfecting the scoring/evaluation system or fixing Gemini’s errors.

Difference Analysis: The human prioritized the training/effectiveness loop (success rate) over the evaluation perfection loop.

Task Sensitivity & Scoring

Role	Approach
Human	Pushed for ‘Role-Aware Scoring’ to distinguish between mere object detection and actual memory of target history.
AI	Initially relied on ‘Target Hit Ratio’ which masked errors via visual overlap.

Difference Analysis: The human’s scrutiny led to the discovery that high hit rates often hid critical memory failures.

Implementation Level

Command Execution

Role	Approach
Human	N/A
AI	Used exact PIDs to kill processes to avoid affecting other users in the shared pod.

Difference Analysis: None

AI Limitations

Critical Limitations

• Prematurely concluded that backbone 4-bit activation was unreachable and that the model ‘doesn’t look at language’ based on flawed experimental design.
• Gemini’s performance on V2 is unknown; current video output is V1. Full training loop with Oracle V2 is required to observe efficacy.
• Cannot restart the pod/cgroup issues; requires administrator intervention if hardware visibility issues persist.

General Limitations

• Initial broad summaries lacked specific architectural nuances (like frozen SigLIP) without heavy human prompting.

Learnings

Key Learnings

• Task complexity does not correlate with quantization sensitivity; tasks requiring fine-grained selection between similar targets (middle vs. top drawer) are most vulnerable to noise.
• The $\pi$0.5 baseline for RoboMME freezes the SigLIP visual tower and performs full-parameter updates on the LLM and action expert.
• Visual prompts must prioritize ’training-robustness’ (surviving crop/rotate) and ‘policy-utility’ (scene-anchored vs. corner glyphs) over aesthetics.
• The ‘Target Hit Rate’ is a dangerous metric for memory tasks: it conflates ‘knowing what to look at’ with ‘remembering what happened’.

Conversation Summaries

Qualcomm Project / $\pi$0.5 Quantization

✅ Solving W4A4 failure via TF-Enhanced quantization claude_code Identified that 4-bit quantization of 35 RMSNorm outputs caused ‘middle-drawer’ task failure. Implemented a re-quantization pipeline using AIMET’s TF-Enhanced ranges, successfully restoring success rates.

RoboMemory

✅ Transitioning to Oracle-based training for 16 tasks codex After auditing Gemini’s visual rendering errors and the flaws in ‘Target Hit Rate’ scoring, the team pivoted to an Oracle-driven ‘Visual Prompt’ system. This involves generating perfect training pairs for 16 tasks to verify if correct visual memory improves robot success rate via scene-anchored marks and robustly placed badges.

✅ RoboMME and pi0.5 Technical Audit Clarified RoboMME’s evaluation protocols and the exact training regime of the pi0.5 baseline, confirming it uses full-parameter fine-tuning of the backbone while freezing the SigLIP vision tower.

TokenMonitor

🔍 Debugging SSH synchronization issues cursor Investigated why SSH usage totals diverge across devices, identifying issues with archive boundaries during synchronization.

Token Usage