Falcon-instruct (PEFT)

Falcon is a family of language models published by the Technology Innovation Institute (TII) in 2023. The Falcon family is composed of two base models: Falcon-40B and Falcon-7B. The 7B model is the best in its weight class. It was trained on 1.5 trillion tokens with the training data based on RefinedWeb, a novel massive web dataset based on CommonCrawl. The Falcon models also include curated sources in their training (such as conversational data from Reddit). This model version features an architecture optimized for inference with FlashAttention (Dao et al., 2022) and multiquery (Shazeer et al., 2019).

Falcon-7B-Instruct and Falcon-40B-Instruct models are 7B and 40B parameters causal decoder-only models built by TII based on Falcon-7B and Falcon-40B and fine tuned on a mixture of chat/instruct datasets, which are also made available under the Apache 2.0 license. These experimental variants have been fine tuned on instructions and conversational data; they thus lend better to popular assistant-style tasks.

Parameter-efficient fine-tuning (PEFT) is a library for efficiently adapting pretrained language models to downstream applications without fine-tuning all the model's parameters. PEFT methods only fine-tune a small number of (extra) model parameters, significantly decreasing computational and storage costs. The common methods for PEFT include LoRA, prefix-tuning, and P-tuning.

This model card demonstrates how to use PEFT to fine tune the Falcon-7B-Instruct and Falcon-40B-Instruct models on Vertex AI and deploy them as a serving endpoint.

• Text generation: the model takes a text prompt as input and generates a text as output.

Get started

This model can be used in a notebook. Click Open notebook to use the model in Colab.

Dataset and training

Falcon-7B-Instruct and Falcon-40B-Instruct have been fine tuned on a mixture of instruct and chat datasets. Falcon-40B-Instruct is a 40B parameters causal decoder-only model built by TII based on Falcon-40B and fine tuned on a mixture of Baize. You can fine-tune the model further using this model card, for example using the timdettmers/openassistant-guanaco dataset.

Model output

The model takes a text prompt as input and generates a text as output.

Best practices and limitations

• This is an instruct model, which may not be ideal for further finetuning. If you are interested in building your own instruct/chat model, we recommend starting from Falcon-7B.
• LoRA based finetune Falcon-7B A100 40G.
• LoRA based finetune Falcon-40B A100 80G.

Versions

Links

• falcon-7b-instruct on Hugging Face
• falcon-40b-instruct on Hugging Face
• PEFT on Hugging Face