Meta has launched the Llama 4 models, Scout and Maverick, which leverage mixture of experts (MoE) technology. Scout utilizes 17 billion active parameters with 16 experts, while Maverick also features 17 billion parameters but with 128 experts. This technology allows specific tasks to be divided among specialized neural network subsystems for more efficient results. The Llama 4 models are built on the as-yet unfinished Llama 4 Behemoth, which consists of 288 billion parameters and incorporates diverse datasets in over 200 languages. The pre-training process employs a new technique called MetaP to enhance model parameter settings, emphasizing increased multilingual support over previous iterations.
Meta has released its first models in the Llama 4 family using mixture of experts technology, improving efficiency and enabling advanced multilingual capabilities.
Collection
[
|
...
]