Chain-of-experts (CoE): A lower-cost LLM framework that increases efficiency and accuracy
1 min read
Summary
Chain-of-experts (CoE) uses less memory and is more efficient than large language models (LLMs), while increasing accuracy, according to researchers from the AI group at Meta.
The CoE framework splits the LLM into “experts” that specialise in different tasks and can communicate with each other – an approach that overcomes the limitations of mixture-of-experts models.
These have separate sets of parameters, or “experts”, that are employed depending on the demands of the task.
This creates a more resource-efficient model, but the disadvantage is that the separate experts don’t communicate, which hinders performance on reasoning tasks.
The new system uses an iterative process that passes information between different sets of experts, meaning the model can build on and communicate information from previous stages.