Daily Report — 2026-10-03
Daily Overview
- What was done: Evaluated the OmniJev visual-only robotic memory arm across 16 RoboMME tasks and conducted a comparative experiment between GPT-6 Astra and Gemini 3.1 Pro regarding visual prompt generation quality for high-difficulty tasks.
- How it was done: Implemented a ‘chat shim’ and wrist-camera proprioception for OmniJev, and developed a paired offline replay and online closed-loop testing framework to compare Astra and Gemini using identical video sequences.
- Impact: Demonstrated that lightweight 2B models can match Gemini’s performance with structured state machines, and identified a significant performance gap where Astra outperforms Gemini in spatial/temporal reasoning and target identity tracking.
Evaluated the OmniJev visual-only arm achieving a 77.5% success rate and conducted comparative testing between GPT-6 Astra and Gemini 3.1 Pro, identifying Astra’s superiority in long-term object identity tracking and visual prompting.
Tasks
Architecture & Strategy
- ✅ OmniJev Protocol Implementation — Built a chat shim to translate Gemini protocol calls into restricted OmniJev question types (yes/no, multiple choice) and implemented a separate evaluation directory.
- ✅ Closed-loop Evaluation — Ran full closed-loop evaluations on 16 tasks using the visual-only arm on tianhe3, reaching a total success rate of 124/160.
- 🔄 Astra vs Gemini Visual Prompt Comparison — Compare Astra and Gemini 3.1 Pro by having them generate visual prompts on the same video replays for VideoRepick and VideoPlaceButton through paired side-by-side videos and closed-loop evaluations.
- ✅ Wrist-Camera Proprioception Integration (I-173) — Replaced high-level proprioceptive data with wrist-camera visual readings for BinFill, PickXtimes, and InsertPeg tasks, including adjusting the BOARD_R threshold to 60px to accommodate visual noise.
Implementation & Fixes
- ✅ Infrastructure Setup and API Debugging — Developed
astra_route_pilot.py, integrated it with tianhe3, and resolved API parameter mismatches (max_tokens vs max_completion_tokens) and LiteLLM proxy restrictions. - ✅ Optimization of Patch Embeddings — Replaced the slow bf16 conv3d patch embedding with an equivalent linear layer, reducing latency from 16s to 0.06ms per image.
Problems & Solutions
Critical Issues
1. OmniJev’s inability to perform long-term temporal reasoning (VideoPlace/Repick).
Solution: Shifted temporal sequencing and event retrieval logic from the VLM to the code-based state machine (the ‘Memory’ part).
General Issues
2. BinFill task failures in closed-loop due to wrist-camera pose error (visual noise).
Solution: Increased the BOARD_R threshold from 40px to 60px based on the maximum error observed in the training set.
3. API compatibility and proxy errors: mismatches in parameter names (max_tokens vs max_completion_tokens) and unsupported ‘reasoning_effort’ for the Astra model.
Solution: Patched compare_eval_entry.py to map parameters correctly and applied an ‘allowed_openai_params’ flag to permit reasoning parameters via the LiteLLM proxy.
4. Automated security block when attempting to send demonstration images to the Duke EGR LiteLLM proxy.
Solution: Requested explicit user authorization to proceed with the experiment via a secure prompt.
Human vs AI Approaches
Strategic Level
System Architecture Strategy
| Role | Approach |
|---|---|
| Human | Human proposed decomposing the memory task into ‘Code for sequence/state’ and ‘VLM for perception’. |
| AI | AI initially analyzed OmniJev as a potential direct replacement for Gemini for all memory tasks. |
Difference Analysis: The human realized that the VLM’s weakness in temporal reasoning is a bottleneck, leading to the design where the VLM only answers ‘is this state true’ while the code manages the ‘count/sequence’.
Task Selection for Model Comparison
| Role | Approach |
|---|---|
| Human | Suggested testing VideoRepick and VideoPlaceButton specifically to compare Astra against Gemini 3.1 Pro, prioritizing tasks that test semantic memory and identity tracking. |
| AI | Initially prioritized RouteStick and PatternLock based on historical success gaps in easier-to-measure trajectory tasks. |
Difference Analysis: The human insight correctly prioritized tasks where the core difficulty lies in cross-frame memory and object identity rather than just geometric pathing.
AI Limitations
General Limitations
- Failed to recognize that a side-by-side comparison of single snapshots was insufficient, missing the requirement for progressive (video-based) visual prompts.
- Initially failed to locate OmniJev work because it was stored in a previous session’s scratchpad rather than the main repository.
- Proposed evaluating RouteStick/PatternLock, which was less effective for testing deep memory compared to VideoRepick.
Learnings
Key Learnings
- A lightweight 2B model is sufficient for perception tasks if the ‘memory’ (sequence/counting) is handled by a structured state machine and visual prompts.
- GPT-6 Astra shows a significant advantage over Gemini 3.1 Pro in ‘remembering’ targets after a sequence of actions (e.g., VideoPlaceButton), whereas Gemini tends to lose track of the specific object identity.
- Rules optimized on expert demonstrations (training set) often fail during policy rollout due to distribution shift in visual appearance.
- Visual prompt quality (circles/badges) can be assessed independently of policy success via matched-replay videos, allowing for isolation of model ‘writing’ ability from ‘driving’ ability.
Conversation Summaries
RoboMemory
🔄 OmniJev Evaluation and Astra vs Gemini Comparison The team focused on two major fronts: validating the OmniJev 2B model as a visual-only writer and conducting a comparative study between GPT-6 Astra and Gemini 3.1 Pro. OmniJev demonstrated effectiveness (77.5% success) when paired with a code-based state machine. Simultaneously, a new testing pipeline for Astra revealed its superior ability in maintaining object identity during complex tasks like VideoPlaceButton compared to Gemini.
Claude Usage Inquiry
✅ Account Usage Investigation The user inquired why the Claude app still works despite 100% weekly usage. The AI confirmed that ’extra usage’ billing was disabled, implying a delay in quota enforcement or rounding.