Textbooks Are All You Need
We introduce phi-1, a new large language model for code, with significantly
smaller size than competing models: phi-1 is a Transformer-based model with
1.3B parameters, trained for 4 days on 8 A100s, using a selection of ``textbook
quality" data from the web (6B tokens) and synthetically generated textbooks
and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains
pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays
surprising emergent properties compared to phi-1-base, our model before our
finetuning stage on a dataset of coding exercises, and phi-1-small, a smaller
model with 350M parameters trained with the same pipeline as phi-1 that still
achieves 45% on HumanEval.
64926f44ef19676db9353f94
Suriya Gunasekar
admin_assigned
2023-06-21T09:18:37.285Z
63e1b4f77fbb6ae4d4f36aa4
/avatars/7febc332d393e0bb94255104527ffe2a.svg
Gunasekar
user
suriyagunasekar
64926f44ef19676db9353f95
Yi Zhang
64926f44ef19676db9353f96
Jyoti Aneja
admin_assigned
2023-06-21T09:19:58.085Z
623e17038972a8c030af23fa
https://cdn-avatars.huggingface.co/v1/production/uploads/623e17038972a8c030af23fa/2ERyhCNAdV2wt4uRy3Wq5.jpeg
Jyoti Aneja
user
jyotiA
64926f44ef19676db9353f97
Caio César Teodoro Mendes
64926f44ef19676db9353f98
Allie Del Giorno
admin_assigned
2023-06-21T09:20:21.299Z
63f5562471a5d395c721cd8e
/avatars/ac296b6017e86ea04c73803fe2c44433.svg
Allie Del Giorno
user
microallie
64926f44ef19676db9353f99
Sivakanth Gopi
admin_assigned
2023-06-21T09:20:47.585Z
644c401926de083beb98544e
/avatars/76dfef0ea98006cd2681b4de3aca9d22.svg
Sivakanth Gopi
user
Sivakanth
64926f44ef19676db9353f9a
Mojan Javaheripi
64926f44ef19676db9353f9b
Piero Kauffmann
64926f44ef19676db9353f9c
Gustavo de Rosa
admin_assigned
2023-06-21T09:23:05.446Z
6157454831624da88210e627
https://cdn-avatars.huggingface.co/v1/production/uploads/1666203761402-6157454831624da88210e627.jpeg
Gustavo de Rosa
user
gugarosa
64926f44ef19676db9353f9d
Olli Saarikivi
64926f44ef19676db9353f9e
Adil Salim
64926f44ef19676db9353f9f
Shital Shah
admin_assigned
2023-06-21T09:23:43.003Z
5fea740dbb3114fa14c82934
/avatars/5f9fe9aaf74a3b930b6fcebcf8a4e4cc.svg
Shital Shah
user
sytelus
64926f44ef19676db9353fa0
Harkirat Singh Behl
admin_assigned
2023-06-21T09:23:57.897Z
63c6aa9e656e7822e2359d9c
https://cdn-avatars.huggingface.co/v1/production/uploads/63c6aa9e656e7822e2359d9c/4rn_faJ23m_a6bevXEDG_.jpeg
Harkirat Behl
user
Harkirat
64926f44ef19676db9353fa1
Xin Wang
64926f44ef19676db9353fa2
Sébastien Bubeck
64926f44ef19676db9353fa3
Ronen Eldan
admin_assigned
2023-06-21T09:24:33.710Z
633723a80267ebcf0264c06b
/avatars/22bb971597e9f3abfa343280a9d0f65f.svg
Ronen Eldan
user
roneneldan
64926f44ef19676db9353fa4
Adam Tauman Kalai
64926f44ef19676db9353fa5
Yin Tat Lee
64926f44ef19676db9353fa6
Yuanzhi Li
admin_assigned
2023-06-21T09:25:09.590Z
641d5a313f04f9bf2da17e8c
/avatars/4035f73c69ddb045573ad96db4f13f04.svg
Yuanzhi Li
user
Uushizhu1234
Textbooks Are All You Need
We introduce phi-1, a new large language model for code, with significantly
smaller size than competing models: phi-1 is a Transformer-based model with
1.3B parameters, trained for 4 days on 8 A100s, using a selection of ``textbook
quality" data from the web (6B tokens) and synthetically generated textbooks
and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains
pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays
surprising emergent properties compared to phi-1-base, our model before our
finetuning stage on a dataset of coding exercises, and phi-1-small, a smaller
model with 350M parameters trained with the same pipeline as phi-1 that still
achieves 45% on HumanEval.
A new compact Transformer-based large language model for code, phi-1, achieves high accuracy on coding benchmarks despite having fewer parameters than competing models.
Transformer-based HumanEval MBPP parameter-efficient fine-tuning