Daily Report — 2026-10-14

Daily Overview

• What was done: The day focused on technical optimization across three major domains: fixing macOS audio capture deprecations, redesigning the RoboMME VLM harness into a unified protocol, and finalizing a robust WeChat media processing pipeline, alongside intensive strategic planning for PhD applications.
• How it was done: Technical execution involved debugging Swift/AVFoundation, designing a three-step unified VLM writer pipeline, implementing media decoding (SILK/animation frames) with task queue optimization, and using AI agents for academic research and document drafting.
• Impact: Successfully ensured software compatibility with macOS 27, established a rigorous framework for Sim2Real testing in robotics, moved the WeChat personal agent from text-only to full media capability, and created a high-precision PhD application roadmap.

MacOS

• What was done: Debugged Swift deprecation warnings in the LiveCaption project.
• How it was done: Replaced the deprecated installTap with installAudioTap in LiveSubtitle.swift to maintain compatibility with macOS 27.
• Impact: Resolved compiler warnings and ensured stable microphone capture for the ASR pipeline.

TzJsDesktop

• What was done: Executed high-level system architecture redesigns, academic strategic planning, and media pipeline development.
• How it was done: Utilized Bash, Codex, and AI agents to audit the drawn_memory library, implement WeChat media parsing logic (pillow/pilk/ffmpeg), and refine PhD SOPs and advisor rosters.
• Impact: Delivered a unified RoboMME design specification, a validated WeChat media retrieval system, and a data-driven PhD application plan covering 19 schools.

The day involved resolving macOS audio deprecations, redesigning the RoboMME VLM harness for task unification, optimizing the WeChat media processing pipeline, and executing strategic PhD application planning.

Tasks

Architecture & Strategy

• ✅ Draft PhD Statement of Purpose — Write and refine a ~850-word SOP emphasizing research logic (FLARE, RecoverBench, Visual Prompting Harness, and Latent Bridge).
• ✅ Harness Unification Design — Created a design document to unify 16 tasks by standardizing the VLM writer input (camera views), the questionnaire process, and the drawing rules.
• ✅ WeChat Media Pipeline Implementation — Implement decoding and retrieval for images, stickers, files, and SILK voice messages.
• ✅ Generate Advisor Shortlist Artifact — Create a web-based interactive roster of 41 advisors across 19 schools with program-specific recommendations.
• ✅ Queue Priority Optimization — Adjusted media attachment recognition priority to prevent it from being stuck behind long summary generation tasks.
• ✅ Sticker/Animation Frame Handling — Implemented a contact sheet approach to combine multiple animation frames into a single image for model compatibility.

Implementation & Fixes

• ✅ Experiment Result Consolidation — Re-aligned and summed up experiment results across different settings (empty, scramble, free, etc.) to identify the performance gap between constrained and free-form writing.
• ✅ Regression Testing — Ran full regression suite (147 tests) and specialized WeChat media tests (26 tests).
• ✅ Verify University Requirements — Research word limits, GRE policies, and department-specific application rules for target PhD programs.
• ✅ Onboarding Documentation — Authored a guide for new developers to set up the RoboHarness environment, including instructions for local offline testing and GPU cluster deployment.
• ✅ Fix LiveCaption Deprecation — Update macOS 27 deprecated audio tap API in LiveSubtitle.swift

Problems & Solutions

Critical Issues

1. The current harness uses task-specific configurations, preventing true generalizability claims and making comparisons unfair.

Solution: Proposed a ‘Three-Step Unified Writer’ that uses a standardized opening questionnaire to determine which memory types (count, ring, arrow, route) to activate.

2. SOP paragraph on FLARE felt illogical and disconnected.

Solution: Restructured the narrative to explain the ‘why’: trained on failure-free data $\rightarrow$ spurious pose-progress links $\rightarrow$ need for perturbation-bridging augmentation.

3. Potential performance drop if task-specific ’tricks’ are removed during unification.

Solution: Planned a phased verification: first a ‘ground truth’ injection test, then offline snippet testing, followed by small-scale closed-loop runs on Tianhe3.

4. Media attachment recognition was delayed by long-running summary generation tasks in the job queue.

Solution: Reprioritized media recognition to be at the same level as summary generation, ensuring sequential processing without extra model load.

5. AI models struggled with multi-frame animated stickers due to interface compatibility issues.

Solution: Created a ‘contact sheet’ by pasting all sampled animation frames into a single wide image for the model to process.

General Issues

6. SOP length exceeded common 1,000-word limits when adding school-specific paragraphs.

Solution: Aggressively trimmed the main body to ~830 words by removing redundant results and secondary research stories.

7. HTML artifact jump-to-anchor failed during page load due to browser scroll overriding.

Solution: Implemented a window.load event listener with a slight timeout to re-trigger the goHash() function.

8. Swift installTap warning on macOS 27.

Solution: Switched to installAudioTap and implemented a buffer copy to maintain compatibility with existing float conversion logic.

9. Incorrect assertion in existing test regarding web page wording after a UI update.

Solution: Updated the test case to assert the correct ‘publication refused’ status instead of outdated wording.

Human vs AI Approaches

Strategic Level

SOP Argumentation for FLARE

Role	Approach
Human	Argued that the distinction between retry and reset should be framed through ‘in-distribution’ vs ‘out-of-distribution’ or ’task change’ rather than just ‘policy failure’ to avoid looking like overfitting.
AI	Initially suggested using ‘out-of-distribution’ as a direct reason for reset, which the user flagged as potentially implying a weakness in generalization.

Difference Analysis: The human provided the critical nuance that the reset is a response to a change in the task (e.g., a toppled block), not a failure of the model’s ability to generalize.

System Architecture Unification

Role	Approach
Human	Demanded that all 16 tasks be treated as a single entity with identical operations and provided specific baseline requirements (GPT-6, Gemini, Qwen).
AI	Iteratively analyzed the code to identify why tasks were currently different and proposed a structural logic to bridge them.

Difference Analysis: The human set the strategic direction (unification and specific baseline comparison) while the AI translated these requirements into a technical implementation plan.

SOP Narrative Flow

Role	Approach
Human	Demanded more organic logical transitions (e.g., connecting ‘awareness’ to ‘memory’ via the failure to realize errors occurred).
AI	Attempted to bridge topics using standard academic phrasing like ‘Awareness requires memory’ or ‘What the benchmark revealed was sobering’.

Difference Analysis: The human pushed for a tighter, more intuitive connection: stating that policies fail because they lack awareness, and that awareness fundamentally requires memory.

Media Verification Logic

Role	Approach
Human	Human insisted on distinguishing between ‘parsing success’ and ‘content accuracy’ to prevent false positives when models add hallucinated details.
AI	AI initially attempted to group parsing and accuracy as a single ‘pass’ metric.

Difference Analysis: The human introduced a more rigorous validation standard for visual description accuracy.

AI Limitations

General Limitations

• AI occasionally hallucinated extra descriptive details in image recognition (e.g., identifying non-existent icons), requiring human-defined distinction between successful parsing and accurate description.
• AI used ‘ad-hoc’ in a way that might sound overly critical or negative in a professional Statement of Purpose context.
• AI initially included citations or specific names (Guanbin Li, Sun Yat-sen University) that the user wanted kept private or generalized.
• The AI initially struggled with the user’s exact requirements for baseline comparison and task uniformity until the user provided specific feedback on the ‘MoveCube’ experiment.

Learnings

Key Learnings

• Constrained writing (asking narrow, specific questions) significantly outperforms free-form writing in VLM-driven visual prompting (53/65 vs 39/65 successes).
• When describing research, focus on the ‘bottleneck’ and the ‘reasoning’ behind a solution rather than just listing high-performance metrics.
• In PhD applications, distinguish between a model’s failure to generalize and a fundamental change in the environment’s task state.
• Task queue priority is critical in agentic workflows to prevent high-latency tasks (like summarization) from blocking time-sensitive media processing.
• Using a ‘contact sheet’ approach for animated stickers is an effective way to support LLMs that only accept single-image inputs.

Conversation Summaries

LiveCaption

✅ Swift Deprecation Fix 10:49:00-04:00 | cursor Addressed macOS 27 deprecation warnings regarding audio taps. Replaced the legacy method with installAudioTap and verified the build process.

PhD Application Strategy

✅ SOP and Advisor Shortlist Refinement 03:10:29.004 | claude_code Conducted deep iterative editing of the Statement of Purpose to ensure logical flow and technical accuracy. Simultaneously built a complex, interactive advisor roster and program recommendation tool, accounting for multi-departmental application strategies and university constraints.

RoboMME Harness

✅ Unified VLM Writer Design and Documentation The user demanded a complete unification of the 16 RoboMME tasks. The AI analyzed the current discrepancies, proposed a standardized 3-step VLM reasoning/drawing protocol, and produced two major documents: a technical design specification and a developer onboarding guide.

LIBERO-PRO Analysis

🔍 Instruction Source and Environment Discrepancies A legacy/continued session discussing why certain settings in the LIBERO-PRO benchmark were inconsistent. Addressed issues regarding BDDL instruction sources and environment stability in the simulation.

WeChat Media Integration

✅ WeChat Media Pipeline Completion 04:13:29.332 | codex The agent analyzed existing test failures, implemented media decoding (SILK, image, file), and optimized the job queue. It verified the entire pipeline using synthetic fixtures for image, sticker, file, and voice media.

PersonalAgents Core

✅ System Regression and Deployment 05:00:00.000 | codex Performed a full deployment of the updated bridge to the VM and restarted the host server. Ran massive regression suites and verified the new media endpoints are reachable and stable.

Token Usage