It has finally appeared. The long-awaited open source model TinyLlama is now available. Compact yet powerful.
TinyLlama project started last September, a group of developers are trying to train a tiny model on trillions of tokens. After a lot of work and some setbacks, the TinyLlama team released this model. It was 1 billion parameters in size and trained with about 1 trillion tokens over about 3 epochs, or cycles of training data.
by paper To give you an overview of the model, the finished TinyLlama outperforms existing open source language models of comparable size. Pythia-1.4B, OPT-1.3B and MPT-1.3B.
A potential use case for TinyLlama is that it could be deployed on edge devices since the model only consumes 637 MB.This model was developed by the team behind it. tutorial Written by Andrej Karpathy, former senior director of AI at Tesla, now at OpenAI.
Official Tiny Llama.Credit: TinyLlama Project Team
The model itself is designed as a compact version of Llama 2, MetaLlama’s open source language model means it can connect to and play with projects built on top of Llama, boasting the same architecture and tokenizer.
Despite its small size, TinyLlama can be used for downstream tasks, and the development team touts it as “an attractive platform for researchers and practitioners of language model research.”
For example, Awni Hannun, a machine learning research scientist at Apple, used LoRA to fine-tune TinyLlama locally on an 8GB Mac Mini. MLXApple’s suite of open source training tools.
“With its compact architecture and promising performance, TinyLlama serves as a lightweight platform for enabling end-user applications on mobile devices and testing a wide range of innovative ideas related to language models.”Development of Mini Models says the team.
More TinyLlamas are in development. The developers plan to develop an “improved version” that includes performance and versatility enhancements across a variety of tasks.
Visit Tiny Llama
TinyLlama can be downloaded for free below. GitHub. Checkpoints for all models are also available. TinyLlama is suitable for commercial use. Apache-2.0 license.
The team behind the model is Tweaked chat version TinyLlama’s current evaluation is high because the learning speed of the base model “hasn’t cooled down yet.”
Small models are also on the rise
Recently, a wave of smaller AI models has started to appear as companies look to reduce the running costs of their hardware.
microsoftFor example, in the Phi project, we are working on small models, billions of parameters in size, that can beat larger models. Phi-2was launched last December and outperforms models of its size by up to 25 times.
Scheduled to be released soon: gemini nanonewly announced smaller version Google The new flagship platform model will be approximately 3.2 billion parameters in size when released later this year.
These smaller models perform well as is, said Bradley Shimin, chief analyst of AI and data analytics at sister research firm Omdia. trained on synthetic data Generated by a larger model.
“Synthetic data is already driving a lot of innovation, and we’re seeing it coming from the generative AI space itself, where capabilities now rival frontier models like OpenAI. There are a lot of smaller models out there that are surprising people: GPT.”