Scaling Granite Code Models to 128K Context
This paper introduces long-context Granite code models that support effective
context windows of up to 128K tokens. Our solution for scaling context length
of Granite 3B/8B code models from 2K/4K to 128K consists of a light-weight
continual pretraining by gradually increasing its RoPE base frequency with
repository-level file packing and length-upsampled long-context data.
Additionally, we also release instruction-tuned models with long-context
support which are derived by further finetuning the long context base models on
a mix of permissively licensed short and long-context instruction-response
pairs. While comparing to the original short-context Granite code models, our
long-context models achieve significant improvements on long-context tasks
without any noticeable performance degradation on regular code completion
benchmarks (e.g., HumanEval). We release all our long-context Granite code
models under an Apache 2.0 license for both research and commercial use.
6699eca6c9111326dc2859ce
Matt Stallone
claimed_verified
2024-07-22T07:10:28.638Z
63b5dfd50d5913eee487b1b7
https://cdn-avatars.huggingface.co/v1/production/uploads/63b5dfd50d5913eee487b1b7/Av-QwMbfUScp0JpzoGGpK.jpeg
Matt
user
stallone
6699eca6c9111326dc2859cf
Vaibhav Saxena
claimed_verified
2024-07-22T07:10:26.166Z
64be2746ae436c88132a848f
/avatars/cd8b08c310b459189b1008119a778ed8.svg
Vaibhav Saxena
user
vaisaxen
6699eca6c9111326dc2859d0
Leonid Karlinsky
6699eca6c9111326dc2859d1
Bridget McGinn
claimed_verified
2024-07-24T07:28:52.137Z
641ddc37771e2336626dbb55
/avatars/b71be4ef8a21d10746bdec6d271f055d.svg
Bridget M
user
brmcg
6699eca6c9111326dc2859d2
Tim Bula
claimed_verified
2024-07-23T07:48:24.504Z
6408a0e2b6a334f53e20d4ab
https://cdn-avatars.huggingface.co/v1/production/uploads/6408a0e2b6a334f53e20d4ab/dIhkQsg-elWTAuqFRfYX-.jpeg
Tim Bula
user
timrbula
6699eca6c9111326dc2859d3
Mayank Mishra
claimed_verified
2024-07-19T11:23:02.972Z
62cd5057674cdb524450093d
https://cdn-avatars.huggingface.co/v1/production/uploads/62cd5057674cdb524450093d/f67rlrdsKPRTLdXCXoa_X.jpeg
Mayank Mishra
user
mayank-mishra
6699eca6c9111326dc2859d4
Adriana Meza Soria
claimed_verified
2024-09-04T07:25:39.046Z
639bcaa2445b133a4e942436
https://cdn-avatars.huggingface.co/v1/production/uploads/639bcaa2445b133a4e942436/h4Oa1ljMQvKXpGCHhTJY5.jpeg
Adriana Meza Soria
user
amezasor
6699eca6c9111326dc2859d5
Gaoyuan Zhang
claimed_verified
2024-08-28T08:01:09.758Z
652431dc0e7792fa3aa2adc7
/avatars/b865d99ff5267184b37ed11cd2bd5626.svg
Gaoyuan Zhang
user
gaoyuanz
6699eca6c9111326dc2859d6
Aditya Prasad
6699eca6c9111326dc2859d7
Yikang Shen
claimed_verified
2024-08-27T07:44:34.309Z
6463995939359568c63d740c
/avatars/11caacefc68c1259dbc8764cc4340481.svg
Yikang Shen
user
YikangS
6699eca6c9111326dc2859d8
Saptha Surendran
6699eca6c9111326dc2859d9
Shanmukha Guttula
6699eca6c9111326dc2859da
Hima Patel
6699eca6c9111326dc2859db
Parameswaran Selvam
6699eca6c9111326dc2859dc
Xuan-Hong Dang
6699eca6c9111326dc2859dd
Yan Koyfman
6699eca6c9111326dc2859de
Atin Sood
claimed_verified
2024-07-26T07:40:59.661Z
635b494062fb2bc1b52d795d
/avatars/26e2c80d3473670fa31342bccb75362a.svg
sood
user
atinsood
6699eca6c9111326dc2859df
Rogerio Feris
6699eca6c9111326dc2859e0
Nirmit Desai
6699eca6c9111326dc2859e1
David D. Cox
6699eca6c9111326dc2859e2
Ruchir Puri
6699eca6c9111326dc2859e3
Rameswar Panda
extracted_pending
2024-07-19T04:33:43.669Z
648a7542ac1fbb8a1a654645
https://cdn-avatars.huggingface.co/v1/production/uploads/648a7542ac1fbb8a1a654645/jLPnY11EOuOa6ZHE7dVEX.jpeg
Rameswar Panda
user
rpand002
Scaling Granite Code Models to 128K Context
This paper introduces long-context Granite code models that support effective
context windows of up to 128K tokens. Our solution for scaling context length
of Granite 3B/8B code models from 2K/4K to 128K consists of a light-weight
continual pretraining by gradually increasing its RoPE base frequency with
repository-level file packing and length-upsampled long-context data.
Additionally, we also release instruction-tuned models with long-context
support which are derived by further finetuning the long context base models on
a mix of permissively licensed short and long-context instruction-response
pairs. While comparing to the original short-context Granite code models, our
long-context models achieve significant improvements on long-context tasks
without any noticeable performance degradation on regular code completion
benchmarks (e.g., HumanEval). We release all our long-context Granite code
models under an Apache 2.0 license for both research and commercial use.
Granite code models with extended context windows up to 128K tokens are achieved through lightweight continual pretraining and finetuning with increased RoPE frequencies and length-upsampled data, showing improvements in long-context tasks without degrading performance on standard benchmarks.
Granite code models RoPE repository-level file packing length-upsampled long-context data instruction-tuned models HumanEval long-context tasks code completion benchmarks