Daily Report — 2026-07-27
Daily Overview
- What was done: Resumed and monitored OpenVLA-LoRA training for four robotic manipulation tasks (Coffee/Stack Nominal and Recovery) on GPUs 4-7, successfully merging LoRA checkpoints and launching parallel evaluation servers. Investigated critical evaluation failures in the Coffee Nominal task, diagnosing zero-success metrics as a result of covariate shift rather than model collapse or data errors.
- How it was done: utilized a Codex agent to manage SSH sessions for process monitoring, checkpoint verification (SHA256 hashing), and environment troubleshooting. Implemented try-except logic for disk space fallback to /tmp during checkpoints, resolved dependency conflicts by probing existing conda environments, and executed teacher-forced diagnostics comparing action prediction errors between nominal and recovery models.
- Impact: Maintained uninterrupted training convergence despite significant storage constraints on Tianhe3. Clarified that nominal task failures were due to distributional mismatch in training data (covariate shift) rather than infrastructure bugs, establishing a strategic direction to increase recovery data ratios for improved closed-loop robustness.
Executed and monitored four parallel OpenVLA-LoRA training runs on Tianhe3 up to 20k steps, implementing disk-fault-tolerant checkpointing, while conducting forensic diagnostics that identified covariate shift as the primary cause of Coffee Nominal evaluation failure.
Tasks
Architecture & Strategy
- 🔄 OpenVLA-LoRA Training and Checkpointing — Monitored four parallel training runs (Coffee/Stack Nominal/Recovery) from 10k to 20k steps. Implemented robust checkpoint saving with disk-quota fallback logic to prevent training interruption.
- ✅ Forensic Diagnosis of Coffee Nominal Failure — Investigated zero-success evaluation metrics by verifying checkpoint integrity via SHA256 hashing, comparing dataset fingerprints using PyArrow, and inspecting inference post-processing. Confirmed model was active but failing due to covariate shift.
- ✅ LoRA Checkpoint Merging and Evaluation Deployment — Merged LoRA weights for all four task checkpoints on GPUs 4-7, resolving Python path dependencies. Launched parallel low-concurrency VLA evaluation servers and rollout workers to validate performance.
- ✅ Teacher-Forced Diagnostic Comparison — Executed diagnostic scripts on 96 paired samples to compare L1 action errors between nominal and recovery models, confirming error accumulation issues in closed-loop settings.
Implementation & Fixes
- ✅ Disk Space and Environment Management — Cleaned 700GB of disk space manually. Resolved complex dependency issues by probing existing conda environments (e.g., ‘rlds_env’ for PyArrow) and correcting PYTHONPATH hierarchies.
Problems & Solutions
Critical Issues
1. Coffee Nominal Clean evaluation reported exactly zero successes, contradicting domain expectations.
Solution: Performed forensic analysis: verified checkpoint hashes matched merge logs (ruling out bad models) and confirmed dataset pairing integrity. Concluded the failure was due to strict termination criteria in nominal tasks (boundary conditions) exacerbated by covariate shift, not data errors or model collapse.
Key Insight: Zero successes in nominal tasks often indicate trajectory deviations hitting obstacles rather than model incompetence; verifying ‘success’ criteria and distributional coverage is essential before blaming the model.
2. Primary disk space quota exceeded during checkpoint saving, risking training interruption.
Solution: Implemented try-except logic to fallback to /tmp if primary storage fails, ensuring training continuity. User later cleaned disk space to restore default behavior.
Key Insight: Automated training systems must prioritize process persistence over storage permanence in constrained HPC environments; graceful degradation is critical for long runs.
General Issues
3. PowerShell SSH command failures due to complex regex/escaping inconsistencies between Windows host and Linux remote.
Solution: Replaced inline bash commands with base64-encoded Python scripts sent via stdin to bypass shell escaping layers, ensuring code integrity across the tunnel.
Key Insight: Complex cross-platform shell escaping is error-prone; using base64-encoded payloads for remote execution provides a robust and reliable alternative.
4. LoRA merge failed with ModuleNotFoundError for ‘prismatic’; diagnostic scripts failed due to missing PyArrow in default environment.
Solution: Explicitly set PYTHONPATH for LoRA merge. Probed multiple conda environments on the remote host to find one containing required libraries (PyArrow, PIL) rather than installing new packages.
Key Insight: Remote execution environments rarely include implicit module paths; manual path injection is mandatory. Scanning pre-existing environments is faster and safer than installing dependencies in isolated training contexts.
Human vs AI Approaches
Strategic Level
Root Cause Analysis of Model Failure
| Role | Approach |
|---|---|
| Human | Human intuitively suspected covariate shift and domain mismatch, guiding the investigation toward data composition rather than infrastructure bugs. |
| AI | AI initially focused on literal data verification (hashes, file diffs) and dependency checks, only aligning with the human’s theoretical insight after diagnostic confirmation. |
Difference Analysis: Human provided strategic domain intuition and heuristics; AI performed the rigorous, tedious low-level verification required to confirm or refute hypotheses.
Handling Disk Full Errors in Training Loops
| Role | Approach |
|---|---|
| Human | User devised the strategic decision to never stop training on disk errors, opting instead for notification and fallback to /tmp. |
| AI | AI implemented the technical try-except wrappers and heartbeat monitors to execute this policy. |
Difference Analysis: Human defined the high-level operational constraint (persistence); AI provided the robust instrumentation to enforce it without manual intervention.
Parallel Evaluation Scaling Decision
| Role | Approach |
|---|---|
| Human | User explicitly approved running four parallel evaluation servers despite known GPU memory contention risks (70-95GB usage), prioritizing experimental throughput. |
| AI | AI assessed the resource constraints, warned of OOM risks, but executed the directive after validation. |
Difference Analysis: Human drove the strategic trade-off between stability and speed; AI acted as a risk manager ensuring feasibility.
AI Limitations
Critical Limitations
- AI failed to automatically detect missing module dependencies (‘prismatic’, ‘pyarrow’) during initial attempts, requiring human-in-the-loop debugging or manual environment probing.
General Limitations
- AI struggled with generating correct PowerShell-to-Bash SSH commands for complex regex patterns and multi-layered shell escaping, frequently resulting in syntax errors.
Learnings
Key Learnings
- In robotic VLA models, low offline training loss on nominal data does not guarantee closed-loop performance; recovery/correction scenarios must be explicitly weighted in training data to mitigate covariate shift.
- For long-running HPC jobs, fault-tolerant coding patterns (try-except with fallback/notify) are superior to standard crash-on-error behavior.
- Checkpoint provenance verification via SHA256 hashing against merge logs is the definitive method to confirm model state integrity, preventing wasted debugging time on ‘wrong model’ issues.
Conversation Summaries
• OpenVLA-LoRA Training Continuation, Checkpoint Handling, and Diagnostic Analysis 10:33:53.461 | codex The user directed the continuation of OpenVLA-OFT LoRA training for four tasks (Coffee/Stack Nominal/Recovery) on Tianhe3 GPUs 4-7. The AI implemented disk-quota fallback logic to prevent training interruption during checkpoint saves. Concurrently, diagnostic efforts focused on explaining zero-success metrics in Coffee Nominal evaluations; forensic analysis ruled out data mismatches or model corruption, identifying covariate shift as the root cause. Strategic discussions concluded that increasing recovery data ratios is necessary for robust closed-loop performance. The system maintained stable parallel execution across multiple stages of verification and training.