Strong Open LLMs ⇒ thriving open ecosystem

Week 30 of Coding with Intelligence

Dear audience, my newsletter comes to you on Saturday instead of the regular Wednesday. I blame the sunny weather in Spain ☀️🇪🇸 — I was on a short holiday to re-energize ⚡️ — but we’re back with an action packed newsletter, let’s dive in!

A note on the title, the overall theme of the updates of this week is that due to the ever increasing quality of open models (Llama 3.1, DeepSeek, Mi(s|x)tral) the ecosystem that builds on top of them continues to flourish, FLUTE as a novel quantization scheme is one of my favorite examples.

📰 News

• SGLang v0.2: beating vLLM and TensorRT-LLM in Llama 3 serving

• Mistral NeMo: 12B model with 128k context length

• AlphaProof and AlphaGeometry 2 achieve silver-medal standard solving International Mathematical Olympiad problems

  However, the competition problem translation into Lean format was still done manually.

• Athene-Llama3-70B

  Significantly improved scores in Arena-Hard compared to vanilla Llama 3 70B Instruct.

📦 Repos

• LMDeploy

  Inference engine with good feature support like quantization and tool use.

• E5-V: Universal Embeddings with Multimodal Large Language Models

• Parseltongue: Chrome extension for prompt jailbreaking

  Works on GPT-4o mini.

• FLUTE: Flexible Lookup Table Engine for LUT-quantized LLMs

  Supported directly in e.g. vLLM.

• dom-to-semantic-markdown

  Interesting project to optimize the HTML DOM representation in Markdown for LLMs to operate on. Will be interesting to see how "optimal Markdown" differs exactly from regular Markdown. Some example differences: Preserves HTML semantic tags (<header>, <footer>, <nav>, <aside>, etc.), Captures image metadata (alt text, dimensions, etc.), Maintains table structures and data relationships, Preserves link destinations while optimizing for token efficiency.

• Apple's truly open source Mistral 7B competitor

• Triton kernels in vLLM for AWQ quantization

  Fast quantized inference with AMD GPUs.

📄 Papers

• Reliable Reasoning Beyond Natural Language

  Use LLMs to translate problems into Prolog and then use fast, reliable and efficient symbolic evaluation code to get answers. I'm very bullish on this general approach (as long-time readers of this newsletter are surely aware by now).

• Llama 3.1 Technical Report

  The bootstrapping techniques of using the model itself to create further training data is really interesting.

• Advancing LLM Reasoning Generalists with Preference Trees

  Paper on producing Eurus-70B, a model with good reasoning performance. They use KTO and NCO for preference learning. They’ve also produced a more updated 8x22B model.

• SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers

🛠️ Products

• Reranking models by Mixedbread

  Models: mxbai-rerank-xsmall-v1, mxbai-rerank-base-v1, mxbai-rerank-large-v1

📚 Resources

• Llama 3.1 405B coding performance in Aider

  Better than Mistral Large 2, behind DeepSeek Coder V2 and Sonnet 3.5. Related note, some say Llama 3.1 coding ability might be hurt by (incorrect) DPO preference tuning.

• MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens

  It's all in the name. By Washington University, Stanford, University of Texas, Berkeley & Salesforce Research.

• Faith and Fate: Transformers as fuzzy pattern matchers

  “The paper's key idea is that while LLMs may sound like they are following a systematic procedure, they are mainly matching parts of a problem against items in their training data, and stitching together results.” @alexisgallagher

• Planning for Agents by Harrison Chase

  He also gave a short talk about the topic.

---

Want more? Follow me on X! @ricklamers