Can we patch recurrent models? TTT & read twice

Week 28 of Coding with Intelligence

📰 News

• LMSYS Chatbot Arena launches Math and Instruction Following leaderboard

• Anole: putting image generation back into Meta’s Chameleon

  It’s interesting because Meta probably removed the capability of image gen from Chameleon for legal reasons.

📦 Repos

• Builder.io launches micro-agent SWE agent

  Similar to e.g. Aider

• AiEleuther has released a set of top-k sparse autoencoders for every layer of Llama 3 8B

  Very interesting (open!) interpretability work. Golden Gate Llama when?

• PufferLib: Simplifying reinforcement learning for complex game environments

  TLDR a bunch of nice abstractions to make reinforcement learning on complex environments less painful. Nice video demonstration.

📄 Papers

• Learning to (Learn at Test Time): RNNs with Expressive Hidden States

• AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agents

  Dynamically accessing memory intelligently (LLM driven dynamic RAG) is becoming a crucial pattern for agentic LLM apps. Cool paper highlighting one such approach.

• Distilling System 2 into System 1

• FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation

  "demonstrating for the first time how to train a large-scale binary language model from scratch (not the partial binary or ternary LLM like BitNet b1.58) to match the performance of its full-precision counterparts (e.g., FP16 or BF16) in transformer-based LLMs"

• Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules

• Data curation via joint example selection further accelerates multimodal learning

  "13× fewer iterations and 10× less computation". By Google DeepMind

📱 Demos

• Parler TTS

  Not SoTA level but interesting study artifact!

• LivePortrait Demo on Hugging Face

🛠️ Products

• Poe launches Artifacts clone

  This is a solidification of the pattern of LLMs generating useful, self-contained programs that can directly be used by users.

📚 Resources

• Just read twice: closing the recall gap for recurrent language models

• Coding Stable Diffusion from scratch in PyTorch

• Deep dive into the architectural components of Gemma 2

• Extrinsic Hallucinations in LLMs

  Another banger by Lilian Weng from OpenAI

• torch.compile: the missing manual

  By Edward Z. Yang from the PyTorch team at Meta

• CapPa: Training vision models as captioners

  Very neat W&B report that is an "Open-source reproduction of "Image Captioners are Scalable Vision Learners Too"

• ColPali: Efficient Document Retrieval with Vision Language Models 👀

---

Want more? Follow me on X! @ricklamers