Daily Report — 2026-10-16
Daily Overview
- What was done: The workflow spanned academic coursework completion, high-stakes conference submission finalization, diagnostic robotics testing, and strategic research repositioning.
- How it was done: Execution involved deep technical tasks like PyTorch/LoRA implementation, LaTeX/BibTeX management, multi-track VLM/API-based diagnostic experiments, and critical analysis of meeting transcripts to pivot research directions.
- Impact: Successfully validated model metrics for homework, secured readiness for NeurIPS release, established a structured empirical plan for robotic error correction, and defined a clear scientific contribution for the MIHD benchmark.
The day involved finalizing ECE 661 homework via LSTM and LoRA experiments, preparing the NeurIPS 2026 camera-ready submission and release, initiating a diagnostic phase for the JEV-six-task project to address robotic task failures, and repositioning the MIHD project toward a multimodal benchmarking scientific narrative.
Tasks
Architecture & Strategy
- • Advisor Consultation — Clarify funding, compute source, and COI for the final submission.
- • Conference Registration — Ensure at least one author has a main-conference registration before Oct 30.
- ✅ LabRNN.ipynb Completion — Implemented data loader, vocabulary builder, LSTM model, and training loop for IMDB sentiment analysis.
- ✅ LabLLM.ipynb Completion — Implemented GPT-2 fine-tuning and LoRA adaptation experiments.
- • Final Release Push — Synchronize HF upload and GitHub push after advisor feedback.
- 🔄 Draft SPEC for NCL+Fusion experiments — Modify pipeline/fusion.py to support NCL output as an input for downstream Concat and Q-Former fusion layers.
- • Overleaf Verification — Compile the provided .zip and ensure layout stays within 10 page limit.
- 🔄 Execute Image Perturbation Experiments (Plan B) — Perform shuffle, noise, and constant-value perturbations on embeddings to test model reliance on histology imagery.
- • VideoRepick — Testing multi-window identity tracking via single-call VLM prompts.
- • ButtonUnmaskSwap — Testing JEV non-binary swap detection (Is A-B swapped?) via short video clips.
- • BinFill/PickXtimes — Testing the 5 new JEV ‘release/lift’ wordings to fix the counting error.
- • PatternLock — Testing symbolic route generation via VLM to replace per-frame point detection.
- • RouteStick — Testing symbolic segments (target, stick, winding) to replace arm-TCP tracking.
Implementation & Fixes
- ✅ LaTeX Report Documentation — Populated main.tex with experimental results, loss tables, and analysis of LoRA vs Full Fine-tuning.
- ✅ Environment Troubleshooting — Resolved dependency conflicts between transformers, datasets, and huggingface-hub in the Conda ‘AI’ environment.
- ✅ Analyze existing fusion data — Reviewed ARI medians for DLPFC (151673) and HD (crop10large) across different gene inputs (HVG, PCA, scGPT, TEDDY).
- • MoveCube — Testing if ‘static icon + manner’ is sufficient without arm-tracking.
- ✅ Verify PDF for AI instructions (Homework TA task) — Checked Duke ECE661 HW3 PDF for hidden system prompts or instructions.
Problems & Solutions
Critical Issues
1. The current NCL method is not ’true fusion’ as it only modifies gene embeddings rather than combining modalities in the prediction stage.
Solution: Plan to implement ‘NCL + Concat’ and ‘NCL + Q-Former’ to provide a full-stack multimodal comparison.
2. Inconsistent clustering behavior in DLPFC where gene-only methods might fallback from mclust to KMeans, biasing results.
Solution: Use ‘Concat + constant’ (zeroed image info) as a gene-only baseline to ensure all groups use the same clustering backend.
3. JEV ‘clamping’ misjudgment (VideoRepick/BinFill)
Solution: Deploy Probe 2 wordings (’lifted’, ’empty’, ‘gap’) to bypass ‘clamping’ ambiguity.
4. Symbolic route accuracy (PatternLock/RouteStick)
Solution: Verify via API if one-shot symbol extraction hits >60% success rate.
General Issues
5. Dependency conflict: transformers 4.57 requires huggingface-hub < 1.0, but datasets 5.1 requires >= 1.31.
Solution: Downgraded datasets to version 4.8.5 to satisfy the huggingface-hub < 1.0 constraint.
6. Pandas 3.0 returns Arrow string arrays that are incompatible with scikit-learn indexing.
Solution: Converted dataframe columns to numpy arrays with object dtype using .to_numpy(dtype=object).
7. The provided helper function ‘predict_sentiment’ had a logic error (searching for word indices as strings).
Solution: Bypassed the helper and implemented direct prediction using the vocabulary mapping.
8. Type 3 font warnings in PDF
Solution: Re-rendered figures as 600dpi bitmaps to ensure embedding compliance.
9. Potential data leak (user paths in JSON)
Solution: Decided to keep existing HF training data as-is (names are public), but scrubbed all new/release documentation.
Human vs AI Approaches
Strategic Level
Author order/affiliations
| Role | Approach |
|---|---|
| Human | N/A |
| AI | Automation of LaTeX superscripts and unification across multiple documentation files (README, Datasheet, HF Card, Croissant). |
Difference Analysis: N/A
Project Positioning
| Role | Approach |
|---|---|
| Human | User provided human-centric advisor feedback to pivot the paper from ‘Method’ to ‘Benchmark’. |
| AI | AI initially suggested ‘Agent Harness’ as a key contribution, but corrected this following user’s correction of the meeting transcript analysis. |
Difference Analysis: The human correctly identified the hierarchy of contributions (Benchmark > Fusion Insights > Agent) based on spoken context.
ButtonUnmaskSwap
| Role | Approach |
|---|---|
| Human | Use continuous VLM monitoring of all possible swap pairs. |
| AI | One-shot VLM for initial setup + JEV ‘yes/no’ sequence for dynamic detection. |
Difference Analysis: N/A
Implementation Level
Code Refinement
| Role | Approach |
|---|---|
| Human | User refactored the AI’s initial implementation with explicit loops and custom logic (e.g., in build_vocab and getitem). |
| AI | AI provided a functional but more abstract implementation to solve the task. |
Difference Analysis: Human’s version was more verbose and specific to their coding preference, resulting in slightly higher accuracy (87.2% vs 84.3%) due to environment/implementation nuances.
BibTeX verification
| Role | Approach |
|---|---|
| Human | N/A |
| AI | Manual text-matching against known paper records and publisher databases. |
Difference Analysis: N/A
AI Limitations
Critical Limitations
- Cannot verify if the compute provided was ‘donated’ vs ‘paid’ without human confirmation from the advisor.
- AI misinterpreted the meeting transcript initially, misidentifying ‘STAIG’ as ‘STAGATE’ and misinterpreting the significance of ‘Agent’ in the paper structure.
- StopCube
- InsertPeg
General Limitations
- AI could not compile the LaTeX document due to lack of a TeX distribution in the environment.
- Cannot visually ‘see’ if the 10-page limit is visually pleasing, only can check the technical page count.
Learnings
Key Learnings
- The efficacy of histology images depends heavily on both the dataset (DLPFC vs HD) and the gene input method (e.g., PCA vs HVG).
- For small models like GPT-2 (124M), the speedup from LoRA is minimal because the forward/backward passes through frozen layers dominate the total computation time.
- NeurIPS camera-ready requirements for funding/COI are strict and must be placed precisely (post-conclusion, pre-references).
- Gemini 3.8 Flash success in ButtonUnmaskSwap is partially due to non-visual proprioception (manual oracle). For C, we must rely on JEV’s visual swap detection.
Conversation Summaries
ECE661 Homework 3
✅ RNN and LLM Implementation & Finalization The AI assisted in completing LSTM and GPT-2/LoRA experiments, managed environment dependency conflicts by downgrading ‘datasets’, and finalized the LaTeX report with validated metrics.
NeurIPS 2026 Camera-Ready & Release
✅ Final Cleanup & Release Prep The AI completed the bulk of technical and authorship updates for the camera-ready submission. A release checklist was provided to the user regarding advisor consultation on funding and registration requirements.
jev-six-task-phasea
✅ Detailed breakdown of the 7-task fix strategy Established a diagnostic phase (Phase A) employing parallel tracks (symbolic routes, multi-window tracking, JEV wording probes) to provide empirical data for solving the 7-task failure set.
MIHD Benchmark
🔄 Project Repositioning and Experimental Planning The project was pivoted from a method-focused paper to a benchmark-centric study investigating the conditions under which histology imagery benefits spatial transcriptomics, including a plan for NCL+Fusion experiments.
ECE661 Homework
✅ AI Instruction Audit As a TA, the user searched a Homework PDF for hidden AI instructions; the AI performed a thorough check of text, metadata, and invisible characters.