Daily Report — 2026-09-14
Daily Overview
- What was done: The day encompassed deep-dive investigations into model quantization effects (pi0.5 and NPU bitwidths), infrastructure maintenance for AI Hub and development environments, and the high-level synthesis of RoboMemory research.
- How it was done: Utilized technical archaeology (git diffs/logs), hardware-level profiling (WPR traces and ONNX probes), and structured documentation workflows (ccfix and iterative PPT generation) to resolve technical bottlenecks and communicate research progress.
- Impact: Successfully isolated the pi0.5 success rate issue to quantization, provided empirical proof that 4-bit activation is a performance trap on Dragonwing NPU, restored CI/CD stability via runtime updates, and produced a comprehensive, professional research reporting suite.
TzJsDesktop
- What was done: Performed high-level project auditing, code integration, and system performance diagnosis.
- How it was done: Analyzed upstream commits for conflicts, leveraged Windows Performance Recorder (.etl) for stuttering diagnosis, and managed project state through engine-driven documentation workflows.
- Impact: Resolved fundamental blocking issues in the development pipeline and established a coherent logical narrative for the RoboMemory research projects.
lighthouse
- What was done: Executed large-scale quantization benchmarking, automated server management, and controlled model evaluation.
- How it was done: Deployed custom Python probes and Bash drivers for NPU bitwidth profiling, managed GPU resources for pi0.5 Libero tests, and implemented systemd-based automation for Minecraft Forge servers.
- Impact: Generated accurate latency and memory telemetry for final reports and provided the empirical evidence needed to narrow down the pi0.5 failure causes.
The day focused on resolving critical performance discrepancies in the pi0.5 model on Libero goal tasks, conducting high-precision bitwidth benchmarking on the Qualcomm NPU to identify quantization pitfalls, fixing AI Hub runtime deprecations, and synthesizing summer research on RoboMemory into professional presentation materials.
Tasks
Architecture & Strategy
- ✅ Verify pi0.5 float baseline on Libero Goal — Run unquantized pi0.5 on the Libero goal suite to establish a performance baseline and verify evaluation pipeline integrity.
- ✅ Bitwidth Speed Microbenchmark (I-047) — Created a minimal MatMul stack probe to isolate the cost of different bitwidths (W4A8, W4A4, W8A8, W8A16, W16A16, FP16) on Dragonwing NPU.
- ✅ Update AI Hub submission runtime — Changed default target_runtime from ‘qnn_context_binary’ to ‘qnn_dlc’ in aihub_submit.py and updated all related documentation/graphs.
- 🔄 Investigate pi0.5 quantization on goal suite — Execute W8A8 and W4A4 quantized benchmarks to determine if the ‘goal suite cliff’ (0% success) is a quantization effect.
- ✅ Research Narrative Synthesis & PPT Construction — Synthesized RoboMemory research into a video-enhanced, bilingual presentation suite including multi-level summaries and a 21-page PPT.
- ✅ Memory Metadata Backfill (I-046) — Recovered 12 missing memory telemetry fields for 51 historical AI Hub jobs to ensure data completeness.
- 🔄 Merge collaborator’s commits and resolve conflicts — Integrate 6 new commits from origin/main, handling conflicts in precision control, rotation implementation, and bitwidth tables.
- 🔄 Limit test concurrency — Implement ‘vitest.config.ts’ to cap maxWorkers and prevent system exhaustion during multi-agent testing.
Implementation & Fixes
- ✅ Implement video recording in evaluation scripts — Enable the passthrough of the –save-video flag from the evaluation wrapper to the Libero client for visual debugging.
- ✅ Diagnose PC stuttering — Analyze WPR traces to find the cause of hardware-level lag during development.
Problems & Solutions
Critical Issues
1. pi0.5 Libero goal suite reported 0% success rate (contradicting reported benchmarks).
Solution: Ran unquantized (float) baseline locally, which achieved 100% success, proving the issue is related to quantization or environment configurations rather than the evaluation pipeline.
2. 4-bit activation (W4A4) performed significantly worse/slower than FP16 on NPU.
Solution: Used a minimal probe to isolate the cause: the NPU lacks a native INT4 activation kernel, causing high overhead from tensor upscaling/expansion.
Key Insight: 4-bit activation is a performance trap on this hardware; weight bitwidth saves bandwidth, but activation bitwidth without native kernels only adds unpacking overhead.
3. AI Hub deprecated the ‘qnn_context_binary’ target runtime, breaking automation.
Solution: Modified ‘aihub_submit.py’ to explicitly pass ‘–target_runtime qnn_dlc’ and updated all related documentation/graphs (I-029) and RESULTS.md.
4. AI Hub’s ‘peak_memory’ field reported discrepancies compared to expected model footprint.
Solution: Performed a comparison between context-binary (uses mmap, zero weight footprint reported) and DLC (requires explicit allocation), clarifying the measurement difference.
5. Lack of Out-of-Distribution (OOD) data for direct error recovery research.
Solution: Pivoted research focus to the ‘interface’ problem: how high-level judgments communicate to low-level policies via visual prompts.
Key Insight: The research bottleneck was the communication channel between high-level reasoning and low-level execution.
6. Evaluation script leaks ~25GB of GPU memory during pi0.5 runs.
Solution: Identified that ‘conda run’ fails to forward signals to the policy server; planned to adopt the collaborator’s PID tracking fix.
General Issues
7. System becomes unresponsive due to high Vitest concurrency and WeChat timer thrashing.
Solution: Diagnosed via WPR/xperf analysis and recommended limiting workers via VITEST_MAX_WORKERS or a config file.
8. Confusion between CUDA device indices and nvidia-smi indices.
Solution: Advised using CUDA_DEVICE_ORDER=PCI_BUS_ID to align indices.
Human vs AI Approaches
Strategic Level
Debugging Strategy for Zero Success Rate
| Role | Approach |
|---|---|
| Human | Suggested visual diagnostics (rendering cases) and provided initial hypotheses based on task-specific sensitivity. |
| AI | Executed systematic code-delta analysis and baseline testing to isolate the variable to quantization precision. |
Difference Analysis: Human focuses on visual/qualitative diagnostics, while AI focuses on structural and statistical verification.
Research Direction & Logic
| Role | Approach |
|---|---|
| Human | Defined the causal link between error recovery failures and the pivot toward visual prompting and memory systems. |
| AI | Initially focused on summarizing individual results without grasping the overarching strategic evolution. |
Difference Analysis: Human provides the strategic context that transforms experiments into a cohesive research story.
Bitwidth Benchmarking Scope & Data Integrity
| Role | Approach |
|---|---|
| Human | Vetoed 32-bit testing to maintain architectural purity (avoiding CPU vs NPU comparison) and demanded raw data audits over qualitative summaries. |
| AI | Initially proposed including 32-bit as a ‘CPU fallback’ and attempted to summarize outcomes descriptively. |
Difference Analysis: Human prioritizes scientific rigor and auditability, whereas AI tends towards completeness and descriptive convenience.
AI Limitations
General Limitations
- Failed to correctly predict GPU availability by ignoring the discrepancy between CUDA and PCI_BUS_ID indexing.
- Failed to recognize that modifying scripts containing ‘red’ phase tests would invalidate failure hashes without a ‘red-waiver’.
- Blocked from writing configuration files by the companion guard (D16/D21) because files were not explicitly ‘owned’ in the idea graph.
- Struggled with D21/D28 guard enforcement when attempting to chain commands through the safety engine.
Learnings
Key Learnings
- Weight bitwidth on this NPU is a linear bandwidth-saving mechanism, but activation bitwidth only provides benefits if high-bitwidth ‘islands’ (like softmax) are eliminated.
- The ‘goal’ suite in Libero (e.g., pulling a handle) is highly sensitive to quantization errors in the action head, unlike simple pick-and-place tasks.
- Visual prompts serve as an effective communication interface between high-level VLM reasoning and low-level VLA execution.
- Companion guards (D16/D17) can prevent productive work if the file structure is not properly registered in the idea graph.
Conversation Summaries
Qualcomm Project
🔄 pi0.5 Libero & NPU Bitwidth Investigation Investigated the 0% success rate of pi0.5 on Libero ‘goal’ tasks and conducted NPU bitwidth profiling. Confirmed that the float baseline works (100% success), isolating the issue to quantization. Developed a MatMul probe (I-047) proving W4A8 is optimal and W4A4 is a performance trap. Addressed AI Hub runtime deprecations by updating scripts to ‘qnn_dlc’.
RoboMemory
✅ Summer Research Synthesis & PPT Creation Synthesized summer research on RoboMemory—from error recovery to visual prompting and asynchronous memory systems—into a professional presentation suite, including a 21-page video-rich PPT and English versions.
D–GitHub
✅ System Performance & Concurrency Analyzed WPR traces to diagnose PC stuttering, identifying Vitest worker bursts and WeChat as the primary culprits. Initiated workflows to limit test concurrency.
AI-Companion
🔍 Infrastructure & Workflow Management Managed development environment stability, including resolving AI Hub runtime issues and navigating companion guard restrictions for configuration files.