I agree! In between French News reporting that Maroilles Lesire (AOP Gros de 750 grammes) has won “meilleur fromage du monde 2024” in Lyon, and similarly for “la bière Vandale brune” (named world’s best beer), both from Picardie, we did get a tiny news item here on Mistral AI’s new Mixtral high quality Sparse Mixture-of-Experts (MoE) LLM ( https://mistral.ai/news/mixtral-of-experts/ ).

Apparently, the MoE approach used by Mistral makes it possible to run the equivalent of Llama-2 and GPT-3.5 locally (eg. WinoGrande BBQ), on last-generation Macs for example, with openly available weights (reminds me a bit of Google’s SparseCores splitting of a large ANN into problem-specific sub-components). Advances of this type have the potential to shift part of the AI/ML computations back towards CPUs, including those for “composition” of the underlying experts, in gastronomic fashion (eh-eh-eh!).

Yann LeCun (at Meta) recently suggested, for example, that improvements may come from replacing (or combining?) auto-regressive token prediction with planning (classical AI methods of tree- and graph-based searches, with heuristics and backtracking) as shown by Noam Brown’s (now at OpenAI) success at multiplayer Texas hold’em poker, and Diplomacy (Pluribus, and Cicero). Google’s DeepMind Gemini seems to also be moving in this direction if I understand well. This suggests (to me) that CPU oomph may become important again, beyond shoveling data into GPUs, and this could affect the architecture of future AI servers (I think).