Daily Report — 2026-09-08

Daily Overview

• What was done: The day’s work involved a deep architectural overhaul of a tile-game RL framework to solve exploration and feature dependency issues, implementing a new quad-writer for VLM memory recording, and transitioning the MIHD project from experimental analysis to manuscript preparation.
• How it was done: Improvements were achieved through redesigning observation spaces using Deep Sets principles, applying logit shrinking to prevent BC-driven policy saturation, implementing a classification-then-tracking hierarchy for VLM tasks, and synthesizing Visium HD benchmarking data.
• Impact: These efforts significantly increased the reliability of game decision-making, established a robust protocol for diverse VLM memory tasks, and provided a rigorous foundation for the MIHD manuscript.

Completed a comprehensive refactor of the Phase-5 PPO v2 RL framework for a tile-matching game, implemented a quad-writer module for VLM memory tasks in RoboMME, and finalized benchmarking experiments for the MIHD project.

Tasks

Architecture & Strategy

• 🔄 MIHD Manuscript Preparation (I-033) — Synthesizing experimental data (DLPFC clustering, Visium HD scaling, and cross-section protocols) into a formal paper structure.
• 🔄 Implement Quad-Writer (I-141) — Developed a VLM writer that classifies 16 RoboMME tasks into a 2x2 matrix (Arm/Object × Before/During) before performing specific tracking and glyph drawing.
• ✅ HD k-sensitivity Sweep (I-038) — Submitted a SLURM array job to verify if Visium HD ’narrow band’ findings are artifacts of fixed k=4.
• • Architecture Redesign (v2) & RL Observation Reform — Reconstructed the Actor network using a ‘Deep Sets’ approach: replaced pattern ID one-hot encodings with numerical features and a symmetric relation matrix to achieve permutation invariance and prevent pattern collision. Corrected spatial encoding errors (grid_x/y) to ensure valid feature inputs.
• • Simulator & Solver Expansion/Fixes — Expanded the simulator to include complex rules (buffer zones, item counts, auto-remove) and fixed critical solver bugs including incorrect tray occupancy counting (treating unknown tiles as occupied) and inaccurate plan cost estimation.
• 🔄 Training & Exploration Optimization — Completed 3M steps of PPO training. To address policy saturation caused by Behavior Cloning (BC) weights, implementing ‘Logit Shrink’ (policy smoothing) and increased entropy coefficients to restore exploration capabilities.
• • Real-world Inference (capture.py) — Upgraded the real-world recognition pipeline with item detection, buffer state recognition, and session logging based on action outcomes.

Implementation & Fixes

• ✅ Administrative & Miscellaneous — Created AGENTS.md contributor guide, migrated the Idea Graph to a project-level structure, and assisted with LaTeX migration for ECE661 homework.

Problems & Solutions

Critical Issues

1. Decision instability and collisions due to pattern ID dependency and one-hot encoding limits.

Solution: Switched to a Deep Sets-inspired architecture using numerical features and a symmetric relation matrix to ensure permutation invariance.

2. VLM inaccuracy in distinguishing between arm motion and scene changes.

Solution: Introduced the ‘Quad-Writer’ hierarchy to classify task context (subject and timing) prior to visual grounding.

3. Solver inaccuracy regarding tray capacity and unknown tiles/occlusions.

Solution: Updated simulator logic to treat unidentified patterns as occupying a slot and refined occlusion unlocking rules.

4. RL policy saturation and lack of exploration caused by high-confidence BC pre-training.

Solution: Implemented Logit Shrink (policy smoothing) to increase distribution entropy without changing the argmax order, alongside increasing entropy coefficients.

General Issues

5. Quad-writer self-test failures due to multi-threaded non-determinism.

Solution: Pinned ‘workers=1’ in the self-test to ensure sequential data delivery.

6. Low precision in detecting patterns in narrow gaps.

Solution: Implemented a grid-aligned ‘gap matching’ algorithm using local pixel features and positional constraints.

Human vs AI Approaches

Strategic Level

VLM Writer Architecture

Role	Approach
Human	Defined a fundamental taxonomic shift: classifying tasks via a 2x2 matrix (Arm/Object x Before/During) before grounding.
AI	Suggested implementation paths by repurposing existing code components like ‘chunk_loop’.

Difference Analysis: The human provided the high-level architectural taxonomy that redefines the prompt generation process.

RL Improvement Strategy

Role	Approach
Human	Proposed a high-level hybrid architectural vision: Reliable state tracking + Limited search + RL scoring.
AI	Initially suggested deep search, but later pivoted to technical implementation details like fixing observability and input features.

Difference Analysis: The human provided the strategic framework, while the AI focused on identifying specific technical flaws (e.g., the coordinate modulo bug).

Planning Capability Verification

Role	Approach
Human	Qualitative observation of whether BC understands elimination logic.
AI	Quantitative ablation study: measuring win rate degradation when explicit policy hints are removed to see if the model learned implicit planning.

Difference Analysis: The AI proposed a scientific ablation methodology to prove autonomous feature utilization.

Sim-to-Real Alignment

Role	Approach
Human	Direct observation of real-world performance.
AI	Stress-testing the simulator by injecting artificial ‘recognition defects’ (misplaced cards/dropped items) to predict real performance.

Difference Analysis: The AI suggested proactive error injection to bridge the gap between simulation and reality.

AI Limitations

Critical Limitations

• The AI initially failed to detect a mathematical error where RL coordinates were being zeroed out by a modulo operator.

General Limitations

• Execution limitations: Blocked from direct ‘git’ or ‘python’ commands by security protocols, and struggled with precise browser interaction (Overleaf) due to coordinate bounds errors.
• Unable to directly prove the acquisition of spatial priors (e.g., ‘deep middle, shallow sides’) due to the lack of explicit spatial coordinates in input features.

Learnings

Key Learnings

• In unordered sets (like game tiles), symmetric aggregation (Deep Sets) is significantly superior to one-hot encoding as it removes arbitrary ID dependency.
• In high-resolution spatial tasks (Visium HD), the zero-shot advantages of foundation models observed in small crops tend to diminish when scaled to full-slide tiling.
• Cross-section niche retrieval requires consistent coordinate meanings across slices to avoid spurious results.
• Behavior Cloning (BC) is a powerful pre-training tool but can create ’exploration dead-ends’ if the policy becomes too peaked; temperature smoothing or entropy adjustment is essential.
• Feature engineering quality (removing ID dependence, adding action-effect states) is a more effective lever for discrete planning problems than simply increasing model parameter count.

Conversation Summaries

Sheep RL v2 / TestYang

🔄 Tile Game Solver & RL Optimization Conducted a massive refactor of the tile-matching game advisor. This included fixing critical solver logic (tray occupancy/occlusion), redesigning the RL observation space using Deep Sets to prevent pattern collisions, and implementing a logit-shrinking strategy to overcome BC-induced policy saturation. The update also integrated advanced simulator rules and improved real-world inference capabilities.

RoboMemory

🔄 Quad-Writer Implementation (I-141) Developed a new VLM strategy for handling 16 RoboMME tasks by categorizing them into four quadrants (Who/When). Implemented the quad_writer.py module and prepared the framework for batch evaluation across local and cloud environments.

MIHD

✅ Manuscript and Experiment Finalization Concluded the benchmarking phase for MIHD. Synthesized findings regarding DLPFC clustering, Visium HD scaling, and cross-section query protocols into a manuscript, while submitting k-sensitivity sweep jobs for further validation.

ECE661

🔍 Markdown to LaTeX Migration Assisted in converting homework Markdown files to LaTeX for Overleaf, exploring agentic automation despite encountering window coordinate issues.

Token Usage