Scaling Granite Code Models to 128K Context
This paper introduces long-context Granite code models that support effective
context windows of up to 128K tokens. Our solution for scaling context length
of Granite 3B/8B code models from 2K/4K to 128K consists of a light-weight
continual pretraining by gradually increasing its RoPE base frequency with
repository-level file packing and length-upsampled long-context data.
Additionally, we also release instruction-tuned models with long-context
support which are derived by further finetuning the long context base models on
a mix of permissively licensed short and long-context instruction-response
pairs. While comparing to the original short-context Granite code models, our
long-context models achieve significant improvements on long-context tasks
without any noticeable performance degradation on regular code completion
benchmarks (e.g., HumanEval). We release all our long-context Granite code
models under an Apache 2.0 license for both research and commercial use.
6699eca6c9111326dc2859ce
Matt Stallone
claimed_verified
2024-07-22T07:10:28.638Z
63b5dfd50d5913eee487b1b7
https://cdn-avatars.huggingface.co/v1/production/uploads/63b5dfd50d5913eee487b1b7/Av-QwMbfUScp0JpzoGGpK.jpeg
Matt
user
stallone
6699eca6c9111326dc2859cf
Vaibhav Saxena
claimed_verified
2024-07-22T07:10:26.166Z
64be2746ae436c88132a848f
/avatars/cd8b08c310b459189b1008119a778ed8.svg
Vaibhav Saxena
user
vaisaxen
6699eca6c9111326dc2859d0
Leonid Karlinsky
6699eca6c9111326dc2859d1
Bridget McGinn
claimed_verified
2024-07-24T07:28:52.137Z
641ddc37771e2336626dbb55
/avatars/b71be4ef8a21d10746bdec6d271f055d.svg
Bridget M
user
brmcg
6699eca6c9111326dc2859d2
Tim Bula
claimed_verified
2024-07-23T07:48:24.504Z
6408a0e2b6a334f53e20d4ab
https://cdn-avatars.huggingface.co/v1/production/uploads/6408a0e2b6a334f53e20d4ab/dIhkQsg-elWTAuqFRfYX-.jpeg
Tim Bula
user
timrbula
6699eca6c9111326dc2859d3
Mayank Mishra
claimed_verified
2024-07-19T11:23:02.972Z
62cd5057674cdb524450093d
https://cdn-avatars.huggingface.co/v1/production/uploads/62cd5057674cdb524450093d/f67rlrdsKPRTLdXCXoa_X.jpeg
Mayank Mishra
user
mayank-mishra
6699eca6c9111326dc2859d4
Adriana Meza Soria
claimed_verified
2024-09-04T07:25:39.046Z
639bcaa2445b133a4e942436
https://cdn-avatars.huggingface.co/v1/production/uploads/639bcaa2445b133a4e942436/h4Oa1ljMQvKXpGCHhTJY5.jpeg
Adriana Meza Soria
user
amezasor
6699eca6c9111326dc2859d5
Gaoyuan Zhang
claimed_verified
2024-08-28T08:01:09.758Z
652431dc0e7792fa3aa2adc7
/avatars/b865d99ff5267184b37ed11cd2bd5626.svg
Gaoyuan Zhang
user
gaoyuanz
6699eca6c9111326dc2859d6
Aditya Prasad
6699eca6c9111326dc2859d7
Yikang Shen
claimed_verified
2024-08-27T07:44:34.309Z
6463995939359568c63d740c
/avatars/11caacefc68c1259dbc8764cc4340481.svg
Yikang Shen
user
YikangS
6699eca6c9111326dc2859d8
Saptha Surendran
6699eca6c9111326dc2859d9
Shanmukha Guttula
6699eca6c9111326dc2859da
Hima Patel
6699eca6c9111326dc2859db
Parameswaran Selvam
6699eca6c9111326dc2859dc
Xuan-Hong Dang
6699eca6c9111326dc2859dd
Yan Koyfman
6699eca6c9111326dc2859de
Atin Sood
claimed_verified
2024-07-26T07:40:59.661Z
635b494062fb2bc1b52d795d
/avatars/26e2c80d3473670fa31342bccb75362a.svg
sood
user
atinsood
6699eca6c9111326dc2859df
Rogerio Feris
6699eca6c9111326dc2859e0
Nirmit Desai
6699eca6c9111326dc2859e1
David D. Cox
6699eca6c9111326dc2859e2
Ruchir Puri
6699eca6c9111326dc2859e3
Rameswar Panda
extracted_pending
2024-07-19T04:33:43.669Z
648a7542ac1fbb8a1a654645
https://cdn-avatars.huggingface.co/v1/production/uploads/648a7542ac1fbb8a1a654645/jLPnY11EOuOa6ZHE7dVEX.jpeg
Rameswar Panda
user
rpand002
Scaling Granite Code Models to 128K Context
This paper introduces long-context Granite code models that support effective
context windows of up to 128K tokens. Our solution for scaling context length
of Granite 3B/8B code models from 2K/4K to 128K consists of a light-weight
continual pretraining by gradually increasing its RoPE base frequency with
repository-level file packing and length-upsampled long-context data.
Additionally, we also release instruction-tuned models with long-context
support which are derived by further finetuning the long context base models on
a mix of permissively licensed short and long-context instruction-response
pairs. While comparing to the original short-context Granite code models, our
long-context models achieve significant improvements on long-context tasks
without any noticeable performance degradation on regular code completion
benchmarks (e.g., HumanEval). We release all our long-context Granite code
models under an Apache 2.0 license for both research and commercial use.
RoPE continual pretraining repository-level file packing length-upsampled instruction-tuned models long-context data long-context tasks HumanEval Apache 2.0 license