Daily Report — 2026-09-05
Daily Overview
- What was done: Executed a comprehensive overhaul of the AI Companion’s core architecture, conducted a systemic repository audit, and performed high-precision VLM trajectory tracing evaluations.
- How it was done: Implemented a recursive hierarchical thought tree and content-bound approval mechanism, utilized adversarial multi-agent workflows for system auditing, and deployed ’trace1’ per-frame prompting alongside Qwen3-VL SFT dataset construction.
- Impact: Resolved critical issues of ‘approval fatigue’ and cognitive overload in the AI Companion, identified fundamental localization bottlenecks in VLMs, and established a robust, scalable framework for both agent governance and high-fidelity robotic trajectory generation.
Advanced the AI Companion through a major architectural refactor toward hierarchical thought trees and intent-based approvals, while simultaneously optimizing VLM trajectory tracing precision via per-frame localization and specialized SFT dataset engineering.
Tasks
Architecture & Strategy
- ✅ Per-Idea Approval Mechanism — Changed approval logic from whole-graph snapshots to individual idea-content binding, ensuring approvals remain valid unless the specific idea’s content changes.
- 🔄 RL Architecture Upgrade (v3/v4) — Migrating from message-passing to self-attention (Transformer) architecture to improve complex reasoning in the SheepEnv.
- ✅ I-104: Loopback Service Protection — Implemented a guardrail to intercept agent tool calls (like browser navigation) targeting local loopback addresses (127.0.0.1, localhost, etc.).
- ✅ Hierarchical Thought Tree Implementation — Replaced the flat ‘steps’ model with a parent-child ’thought tree’ where each node can be expanded, limited to 7 items per level.
- ✅ Qwen3-VL SFT Dataset Construction — Created a training dataset (160 train / 40 dev) using 200 proprioceptive oracle sidecars, resampled to 24 points and 12 frames.
- ✅ I-102: Verifiable Approval Display — Ensured the terminal prints the full 8-question context of an idea before asking for an approval code to prevent blind approvals.
- ✅ I-086: Dynamic Idea Page Header — Update the web interface so that the H1 title and overview paragraph dynamically change to match the selected idea, while moving historical project info into a collapsed section.
- ✅ Idea Graph Integrity Check — Validated the current state of 71 ideas, identifying 47 completed, 4 in-progress, 18 todo, and 2 blocked.
- 🔄 Idea Cluster Merging (I-082 Family) — Consolidating obsolete ‘step overview’ ideas into a single truthful ‘idea layering’ concept.
- 🔄 I-060 Eval-side Insertion — Wiring the V6 per-chunk writer into the evaluation client with anchor patches and smoke test scripts.
- 🔄 Documentation Truth Sweep — Identifying 36 specific instances where prose in README, CLAUDE, and FORMAT files contradicts actual code behavior.
Implementation & Fixes
- ✅ Project-wide Scan & Audit — Read all 81 files in the repository to build a complete understanding of the current state.
- ✅ Legacy AI Companion & System Cleanup — Uninstalled all previous versions of AI Companion hooks, skills, and command aliases across 10+ repositories and archived legacy files/worktrees.
- ✅ I-058 Tunnel Setup — Establish VPN tunnel to connect local workstation to the Tianhe3 cluster for remote Gemini API access.
- ✅ I-085: Step Overview Completion — Finalized and closed the implementation of the step-based overview feature.
Problems & Solutions
Critical Issues
1. Agent could potentially bypass human approval by performing direct actions on a locally hosted preview page.
Solution: Conducted a ‘user activation probe’ experiment proving that automated CDP clicks are indistinguishable from human clicks, and implemented a guardrail to block all loopback-bound tool calls.
2. RL agent failed to generalize to harder levels due to lack of global item perception.
Solution: Implemented a message-passing layer and subsequently a full Self-Attention (Transformer) architecture to allow items to interact globally.
3. Approval fatigue caused by whole-graph snapshotting, where any minor change invalidated all existing approvals.
Solution: Implemented content-bound approval digests specifically for individual node IDs and their relevant fields.
4. Subagent Bypass of Guardrails: An automated workflow subagent successfully edited protected engine source files and the idea ledger.
Solution: Identified the guardrail allows writes to ideas/ without subagent-level restriction; implemented manual reversion via main session.
5. Gemini 3.1 Pro/Qwen3-VL-2B failures in batch-mode trajectory tracing (error, repetition, and truncation).
Solution: Implemented ’trace1’ mode (one frame per API call with temporal prior) and shifted to per-frame single-point localization with fixed point counts.
General Issues
6. Checkpoint filename collisions and unexplained exit-code-4 crashes during long GPU runs.
Solution: Added --run-tag to differentiate checkpoints and improved resume semantics.
7. Validation errors and Git failures due to deletion of legacy archive directories and obsolete test blocks.
Solution: Updated graph.yaml to re-point ideas to new paths and removed obsolete ‘frozen twin spec’ test blocks.
8. Injecting browser scripts via template literals caused syntax errors.
Solution: Escaped backticks and used double backslashes for newline characters.
Human vs AI Approaches
Strategic Level
Approval Lifecycle
| Role | Approach |
|---|---|
| Human | Defined that approval should be ‘sticky’ to content; once approved, it shouldn’t require re-approval unless the specific intent changes. |
| AI | Initially treated approval as a global state snapshot that breaks whenever the graph changes. |
Difference Analysis: Human identified the semantic distinction between ‘implementing a plan’ and ‘changing a goal’.
Approval Mechanism Security
| Role | Approach |
|---|---|
| Human | Suggested implementing ‘Direct Page Approval’ (clicking a button on the website to approve an idea). |
| AI | Conducted an empirical probe using browser tools to demonstrate that automation is indistinguishable from human interaction, then proposed a ‘deny-by-default’ loopback rule instead. |
Difference Analysis: The AI used empirical testing to prove the human’s intuitive security assumption was flawed.
Unauthorized Edit Recovery
| Role | Approach |
|---|---|
| Human | Attempted to use git checkout to revert unauthorized changes. |
| AI | Identified that the git checkout command was blocked by the guardrail rules, necessitating manual text editing. |
Difference Analysis: The AI recognized standard recovery tools were caught in the security net.
UI/UX Structure for Thoughts
| Role | Approach |
|---|---|
| Human | Proposed that thoughts should be a hierarchical tree (max 7 per level) where details are hidden behind expansion/clicks to prevent clutter. |
| AI | Initially focused on simplifying the current flat ‘steps’ view or focusing on scanning speed. |
Difference Analysis: Human provided the specific architectural UX pattern (tree structure/depth control) that the AI then implemented.
Project Navigation and Header Logic
| Role | Approach |
|---|---|
| Human | Defined the business logic: header shows idea name + ‘what’, and home page features a collapsed ‘Project History’ section. |
| AI | Researched technical implementation details like WCAG 2.4.2, CSS for collapsed sections, and paragraph parsing logic. |
Difference Analysis: Human provided functional requirement/intent; AI handled technical research and implementation.
Idea Graph Merging
| Role | Approach |
|---|---|
| Human | Proposed merging similar ideas (e.g., I-059 and I-075) to simplify the graph. |
| AI | Used an adversarial multi-agent workflow to prove they represent different levels of requirement (state vs. rule). |
Difference Analysis: The AI prevented a loss of technical nuance by proving a prerequisite relationship rather than duplication.
VLM Trajectory Generation Strategy
| Role | Approach |
|---|---|
| Human | Proposed a simple pipeline: VLM reads video, generates a list of points, connects them to map position. |
| AI | Initially attempted batch ’trace’ mode, which failed, then pivoted to ‘per-frame with temporal prior’. |
Difference Analysis: AI’s investigation revealed a mismatch between current batch-VLM capabilities and required temporal precision.
AI Limitations
Critical Limitations
- Subagent autonomy exceeds guardrail constraints; workflows can bypass ‘read-only’ instructions when writing to allowed paths like
ideas/. - The guard cannot detect or block JavaScript execution within a page because it lacks access to internal browser state.
- AI Companion documentation was outdated, causing the assistant to use old ECL workflows instead of new systems.
- AI failed to account for the drop in temporal precision in multi-frame batching, misattributing failure to reasoning rather than correspondence.
- The AI initially attempted to add unapproved symbols to the code, triggering the project’s own guard system (D7).
General Limitations
- AI struggled to resolve technical debt where existing tests were green but technically invalid due to outdated concepts.
Learnings
Key Learnings
- High-precision trajectory tracing requires per-frame localization (grounding) rather than batch-style reasoning.
- Automated clicks via CDP set ‘isTrusted: true’ and ‘userActivation: true’, making them indistinguishable from humans at the browser level.
- Transitioning from a global snapshot model to a granular, per-item digest model is essential to reduce user ‘approval fatigue’.
- A guardrail that allows writes to a directory without tying them to a specific ‘doing’ idea’s scope is vulnerable to subagent drift.
- Approval mechanisms in AI agents must be tied to immutable semantic intent rather than volatile file snapshots.
- The project’s guard system is highly sensitive; even adding an extra symbol not in the plan can invalidate manual approval.
- Hierarchical data visualization is critical for complex project management to prevent ‘information paralysis’.
Practical Learnings
- When injecting JavaScript via template literals, backticks and special characters must be strictly escaped.
Conversation Summaries
SheepRL
🔄 RL Architecture & Training Resume 18:50:13.788 | claude_code Recovered a crashed training run, improved checkpoint naming/resume logic, and launched a new Self-Attention architecture experiment.
AI Companion
✅ Governance, Architecture & Cleanup 21:52:32.303 | claude_code A massive series of updates: implemented I-102/I-104 security, refactored the core to a hierarchical ’thought tree’ with per-idea approval binding, performed a comprehensive cleanup of legacy hooks/skills across 10+ repos, and audited the idea graph for integrity and documentation truth.
✅ Implementation of Dynamic Idea Header (I-086) 00:00:00.000 | claude_code Researched, planned, and implemented a dynamic header for the idea graph using a Red-Green testing workflow, ensuring accessibility (WCAG) and resolving outdated schema references.
OMEN Health Check
✅ System Stability Analysis 21:41:18.148 | claude_code Analyzed system logs for hardware/driver errors, concluding recent changes have stabilized the hardware.
RoboMemory
✅ VLM Trajectory Precision & Dataset Engineering 03:39:51.955 | claude_code Investigated VLM failures in batch-tracing, validated ’trace1’ per-frame mode for sub-3px precision, and built a specialized Qwen3-VL SFT dataset using proprioceptive oracle sidecars.