Daily Report — 2026-09-30
Daily Overview
- What was done: Engineered and audited multiple high-stakes research projects, ranging from robotics benchmark usability and VLM evaluation frameworks to quantization diagnostics, WeChat bot personalization, and hardware-specific error investigations.
- How it was done: Employed a multi-pronged technical approach including automated pipeline audits, modular plugin development, programmatic PPT generation via python-pptx, strict environment variable enforcement for visual-only modes, and systematic error attribution sweeps via hardware-vs-simulation comparisons.
- Impact: Successfully transitioned research from documentation to operational readiness, achieved high-performance VLM evaluation results, established robust visual-only baselines, resolved critical development infrastructure bottlenecks, and provided a clear roadmap for handling quantization and hardware-specific anomalies.
The day’s work spanned diverse domains, including establishing a camera-ready robotics benchmark pipeline, extending OmniJev evaluation to 16 tasks, preparing a strategic research split presentation, enforcing visual-only integrity for Gemini-based memory, resolving development server port leaks, upgrading a WeChat bot’s persona, investigating hardware-specific NaN issues in C2S, and diagnosing W4A4 quantization failures.
Tasks
Architecture & Strategy
- ✅ OmniJev 16-task Extension & Evaluation — Implemented shim extensions and truth replay for 16 RoboMME tasks and conducted a full closed-loop evaluation using a 2B model, achieving a 76.3% success rate.
- • Gemini Visual-Only Integrity Enforcement (I-171) — Executing a pure-visual Gemini arm ablation (16 tasks) by stripping proprioception and implementing detection-based visual window tracking.
- ✅ W4A4 Quantization Accuracy Investigation — Diagnosed the 0% success rate in W4A4 simulation through device-vs-simulation comparisons and error attribution sweeps.
- ✅ Camera-ready Workspace & Pipeline Audit — Created a verified workspace on tianhe3, imported code/dependencies, and performed a stage-by-stage usability audit of the Coffee task pipeline.
- ✅ Strategic Research Split & PPT Generation — Developed a multi-version professional presentation to propose splitting research into two tracks (Jev visual reasoning vs. Gemini harness) between labs.
- ✅ Vite Dev Server Exit Synchronization (I-082) — Implemented a Rust-to-Vite POST signaling mechanism to ensure clean shutdowns and prevent port 1420 leaks on Windows.
- ✅ WeAgent Bot Advanced Feature Upgrade — Upgraded the bot with model-switching, a refined CS assistant persona, and a visual-similarity-based WeChat sticker sending system.
- 🔄 C2S NaN & Pilot Batch Management — Investigating hardware-specific NaNs in C2S and monitoring a batch of 17 pilot submissions and gene encoder notebook reruns.
Problems & Solutions
Critical Issues
1. Potential proprioception leaks and non-visual window advancement in Gemini-driven memory tasks.
Solution: Enabled ‘VISUAL_ONLY’ to strip state and ‘VISUAL_TRACK’ to advance windows based on VLM tip-detections rather than TCP/proprio data.
2. OmniJev PickHighlight reporting low recall and 2B model underperforming in temporal tasks (Swap/StopCube).
Solution: Corrected metric frame population and shifted architecture to use code for temporal logic/identity while VLM handles per-frame perception.
3. W4A4 simulation showing 0% success rate while actual hardware maintains higher accuracy.
Solution: Determined simulation is overly pessimistic due to aggressive 4-bit rounding on tensors that hardware actually processes in FP16.
4. Robotics pipeline errors including JSON serialization (np.bool_), uncropped image formats in BC-RNN, and human demo state mismatches.
Solution: Identified serialization needs in ‘core.py’, required HWC image fixes in ‘policy_adapter.py’, and corrected ‘replay’ settings to ensure error-state restoration.
5. Vite processes remaining active on Windows after exiting the Tauri app via the system tray.
Solution: Replaced a polling mechanism with an explicit POST-based notification sent from the Rust backend to a dedicated Vite plugin endpoint.
6. WeChat stickers being sent as images and false-positive database confirmation errors.
Solution: Implemented UIA-based panel browsing for sticker similarity matching and disabled strict verification to handle WeChat write latency.
7. C2S NaNs occurring on specific GPU nodes and invalid matrix command arguments.
Solution: Identified node-specific hardware issues (h36-6) and excluded them from templates; reconciled command arguments via subagent audit.
Human vs AI Approaches
Strategic Level
Architectural Strategy
| Role | Approach |
|---|---|
| Human | Insisted on separating ‘state/identity’ (code) from ‘perception’ (VLM) and preferred broadcast/signaling over polling for exit signals. |
| AI | Initially attempted to use the VLM for all decisions and utilized a standard polling loop for process management. |
Difference Analysis: Human insights optimized both model performance and system robustness.
Project Pivot & Scope
| Role | Approach |
|---|---|
| Human | Directed transition from passive documentation to practical usability audits and insisted on full-scale task extensions for fair comparisons. |
| AI | Focused on baseline documentation and initial successful tasks. |
Difference Analysis: Human intervention pivoted missions toward operational readiness and rigorous benchmarking.
Recovery & Communication
| Role | Approach |
|---|---|
| Human | Prioritized covering all potential recovery avenues (device-faithful simulation, etc.) and formulated diplomatic communication strategies for faculty. |
| AI | Proposed specific technical paths (attribution sweeps) and refined logic into a ‘respect-first’ communication framework. |
Difference Analysis: Human provided the strategic breadth and diplomatic nuance, while AI provided the structural execution.
AI Limitations
Critical Limitations
- Inherent limitations in simulating ‘pure visual’ modes (potential lag/drift) and the slow speed of UI-based expression browser (5-7s per sticker).
- Struggled with temporal reasoning (identity tracking), high-precision point generation, and complex custom data structure errors (AttributeError/KeyError).
General Limitations
- Encountered environment restrictions (writing to memory directories) and shell redirection/tool-call blocks requiring manual intervention.
Learnings
Key Learnings
- A lightweight 2B VLM is excellent for recognition but needs code-driven assistance for temporal reasoning and identity tracking.
- On specific hardware, 4-bit simulations are unreliable for predicting accuracy due to hardware automatically using FP16 for certain ops.
- Seeding all randomness (including env resets and Python’s hash()) is critical for robotics benchmark reproducibility.
- Visual prompts (drawing memory into observation) significantly outperform text-based memory in path-related tasks.
- In multi-process environments like Tauri/Vite, explicit exit signaling is more robust than ancestry/polling tracking on Windows.
Conversation Summaries
ErrorRecoveryBenchmark
✅ Workspace Setup & Pipeline Usability Established a patched workspace on tianhe3 and performed a comprehensive audit of the coffee task, identifying critical flaws in phase labeling and error-state restoration.
RoboMemory
✅ OmniJev Expansion & Strategic Research Split Extended OmniJev to 16 tasks with a 76.3% success rate. Simultaneously prepared a strategic presentation for splitting research into two independent tracks (Jev vs. Gemini) and enforced ‘pure-visual’ integrity for Gemini-based experiments.
TokenMonitor
✅ Vite Port Leak Resolution Resolved the issue of Vite processes holding port 1420 on Windows by implementing a Rust-to-Vite POST signaling mechanism.
WeAgent Bot
✅ Persona & Emoji System Development Upgraded the bot with a refined CS assistant persona and a visual-similarity-based mechanism for sending real WeChat stickers.
MihD-dev
✅ C2S NaN Investigation & Pilot Submission Investigated C2S NaNs, confirmed hardware-specific issues on certain nodes, and submitted a batch of pilot experiments after correcting command arguments.
Qualcomm Project
✅ W4A4 Quantization Failure Analysis Identified that W4A4 quantization simulation artifacts were driving a 0% success rate, while real hardware performed significantly better by using FP16 for certain operations.