Discover more from Coding with Intelligence
OpenAI's embedding model dethroned in MTEB by BAAI general embedding model
Week 32 of Coding with Intelligence
bge stands for BAAI general embedding, with embeddings of just 384 dimensions versus the 1536 of OpenAI's text-embedding-ada-002 there is a new very interesting choice for embedding generation. Not just that, OpenAI's embeddings are costly API calls compared to self-hostable open source bge-family of models. Also check out their repo https://github.com/FlagOpen/FlagEmbedding
Will be interesting to see if more fabless chip companies enter the space of NVIDIA GPUs, AMD GPUs, and TPUs. Companies like Tenstorrent, SambaNova, Untether AI, Groq, Habana Labs, Graphcore and Cerebras are certainly trying!
Use this code to produce instruct-tuned ChatGPT style models from custom data (RLHF) from open source base models. A full tune only costs on the order of hundreds of dollars of compute time.
They even draw a full analogy to web design in their original blog post: https://www.cursor.so/blog/prompt-design Not so sure about whether prompt engineering (or prompt design as they prefer to call it) is really analogous to web design, but I do agree that rendering prompts for inspection and composability are key for efficient and high quality prompt authoring. One issue I see is for folks (yours truly) not in a JS context, can we port this to Python :)?
An open-source Python package to streamline and unify working with open LLMs. They promise swapping models is as easy as swapping the import line (not sure if that's the right abstraction level, but let's see!)
According to https://twitter.com/johnjnay/status/1688737571788967936 it scores the highest for agent tasks out of all evaluated open source models.
An alternative to RLHF, time to say goodbye to complex multi-stage fine-tuning processes?
Retraining, fine-tuning and context windows are no longer the only paradigms for updating language models. We now have LLM surgery as another option for including new knowledge for the model to use.
Docs > examples!
Great analysis! Kudos to Aman from the Cursor team.
The moment models shift and start to generalize instead of regurgitate can happen pretty abruptly during training. The phenomenon is called grokking and was first shown in 2021. A deeper exploration of this behavior is now called mechanistic interpretability and is getting a lot of attention from LLM researchers. Learn more by reading this excellent interactive exploration by a team of Google researchers.
A simple Python library to make chained attributes possible. You have to wonder, why isn't it already like this?
Simple implementation of a transformer model, with some borrowed efficiency improvements. The purpose is mainly pedagogical. Created by someone from the EleutherAI project.
Want more? Follow me on Twitter! @ricklamers