Researchers find you don’t need a ton of data to train LLMs for reasoning tasks
1 min read
Summary
Researchers at Shanghai Jiao Tong University have claimed that large language models (LLMs) can be trained on a small number of curated examples to complete reasoning tasks.
This is in contrast to commonly held views that tens of thousands of training examples were required, claiming that rich reasoning knowledge obtained in the pre-training phase, and the use of new training methods requiring fewer data and compute, can achieve the desired results.
The argument is built on the concept that high-quality demonstrations unlock complex reasoning capabilities rather than sheer volumes of data.
The researchers have published code and data and plan to expand the concept in the future.
This efficiency could enable more enterprises to create customised models without the need for large AI labs.