MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model
  Series
Large Language Models (LLMs) have made great strides in recent years to
achieve unprecedented performance across different tasks. However, due to
commercial interest, the most competitive models like GPT, Gemini, and Claude
have been gated behind proprietary interfaces without disclosing the training
details. Recently, many institutions have open-sourced several strong LLMs like
LLaMA-3, comparable to existing closed-source LLMs. However, only the model's
weights are provided with most details (e.g., intermediate checkpoints,
pre-training corpus, and training code, etc.) being undisclosed. To improve the
transparency of LLMs, the research community has formed to open-source truly
open LLMs (e.g., Pythia, Amber, OLMo), where more details (e.g., pre-training
corpus and training code) are being provided. These models have greatly
advanced the scientific study of these large models including their strengths,
weaknesses, biases and risks. However, we observe that the existing truly open
LLMs on reasoning, knowledge, and coding tasks are still inferior to existing
state-of-the-art LLMs with similar model sizes. To this end, we open-source
MAP-Neo, a highly capable and transparent bilingual language model with 7B
parameters trained from scratch on 4.5T high-quality tokens. Our MAP-Neo is the
first fully open-sourced bilingual LLM with comparable performance compared to
existing state-of-the-art LLMs. Moreover, we open-source all details to
reproduce our MAP-Neo, where the cleaned pre-training corpus, data cleaning
pipeline, checkpoints, and well-optimized training/evaluation framework are
provided. Finally, we hope our MAP-Neo will enhance and strengthen the open
research community and inspire more innovations and creativities to facilitate
the further improvements of LLMs.
6657e84f1436864bccbdcc40
Ge Zhang
claimed_verified
2024-05-31T06:25:40.803Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
6657e84f1436864bccbdcc41
Scott Qu
claimed_verified
2024-05-31T06:25:42.762Z
646b4c9fdf2609a541c0866e
/avatars/7ec6c709017dcf32d4ac49d1e3820328.svg
Qu
user
ScottQu
6657e84f1436864bccbdcc42
Jiaheng Liu
admin_assigned
2024-05-31T10:16:49.808Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
6657e84f1436864bccbdcc43
Chenchen Zhang
claimed_verified
2024-06-02T18:47:30.655Z
64b74b906ab5d14ca7f289cd
/avatars/b131b7c4ce5216708ca4a678f35ead0a.svg
xxzcc
user
xxzcc
6657e84f1436864bccbdcc44
Chenghua Lin
admin_assigned
2024-05-31T10:23:35.232Z
6442ba123610a28a4ad5382f
/avatars/97f2d3d2c10058ea41d7bf9087e1f619.svg
Chenghua Lin
user
chenghualin
6657e84f1436864bccbdcc45
Chou Leuang Yu
claimed_verified
2025-04-10T06:47:55.181Z
62a80fe3ac97233f1625235a
https://cdn-avatars.huggingface.co/v1/production/uploads/62a80fe3ac97233f1625235a/_rGtpqdY7OEBz3pyqb6fE.jpeg
Zhouliang Yu
user
zhouliang
6657e84f1436864bccbdcc46
Danny Pan
6657e84f1436864bccbdcc47
Esther Cheng
admin_assigned
2024-05-31T10:24:11.283Z
651fa34245d808ba81132cdb
/avatars/444d2cb7e4e4b3c32555886dc23fc512.svg
Esther
user
EstherCheng
6657e84f1436864bccbdcc48
Jie Liu
6657e84f1436864bccbdcc49
Qunshu Lin
6657e84f1436864bccbdcc4a
Raven Yuan
6657e84f1436864bccbdcc4b
Tuney Zheng
claimed_verified
2024-05-31T06:25:38.830Z
64ab99dcb76bfd863eba64c1
https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
TY.Zheng
user
aaabiao
6657e84f1436864bccbdcc4c
Wei Pang
claimed_verified
2025-05-28T09:02:51.659Z
65164444bc0631719873af81
/avatars/dab8b90db8bbd00806268fe276e3ea36.svg
Wei Pang
user
weipang142857
6657e84f1436864bccbdcc4d
Xinrun Du
claimed_verified
2024-05-31T06:25:45.759Z
654907a4a1faff97850c4eff
/avatars/458c90151614bc7f116943b6e67d6b8a.svg
du
user
dododododo
6657e84f1436864bccbdcc4e
Yiming Liang
admin_assigned
2024-05-31T10:25:32.380Z
654fed4d7490049d621d0cd3
/avatars/6bc1553d1571ab4e9ab80c58eeb9820c.svg
Yiming Liang
user
Yiming1234
6657e84f1436864bccbdcc4f
Yinghao Ma
admin_assigned
2024-05-31T10:25:40.489Z
6410665d5364a661bee22524
/avatars/f1cb0e07f36933187ceccbd5dcbeff79.svg
Yinghao Ma
user
nicolaus625
6657e84f1436864bccbdcc50
Yizhi Li
claimed_verified
2024-05-31T06:25:48.259Z
6382252f54421460665ec501
https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
Yizhi Li
user
yizhilll
6657e84f1436864bccbdcc51
Ziyang Ma
6657e84f1436864bccbdcc52
Bill Lin
6657e84f1436864bccbdcc53
Emmanouil Benetos
claimed_verified
2024-08-05T10:24:08.573Z
66b0a0839797b94f001ed874
https://cdn-avatars.huggingface.co/v1/production/uploads/66b0a0839797b94f001ed874/BBtlJdQNyzQX10chnU81s.jpeg
Emmanouil Benetos
user
emmanouilb
6657e84f1436864bccbdcc54
Huan Yang
6657e84f1436864bccbdcc55
Junting Zhou
admin_assigned
2024-05-31T10:26:53.213Z
65a374a59acab1998092a9bc
https://cdn-avatars.huggingface.co/v1/production/uploads/65a374a59acab1998092a9bc/M3s_7bSf9G-6b9nLg7N3Z.jpeg
Antonio
user
JuntingZhou
6657e84f1436864bccbdcc56
Kaijing Ma
admin_assigned
2024-05-31T10:27:00.457Z
65eb65722fbf6807134a636c
https://cdn-avatars.huggingface.co/v1/production/uploads/65eb65722fbf6807134a636c/8YgrynSM-WExVlu2mwRBY.jpeg
Kaijing Ma
user
mkj69
6657e84f1436864bccbdcc57
Minghao Liu
claimed_verified
2024-06-21T11:47:36.314Z
6417d9ea8f689506e7148417
https://cdn-avatars.huggingface.co/v1/production/uploads/6417d9ea8f689506e7148417/bAYcruWNw4WvmuQcGgcwC.jpeg
minghao
user
Liam-Liu
6657e84f1436864bccbdcc58
Morry Niu
admin_assigned
2024-05-31T10:27:37.870Z
65bb14f139c4e7087640a91c
/avatars/dbf75dd161d22b4511e9fccff6afc515.svg
Morry Niu
user
bl1ndbot
6657e84f1436864bccbdcc59
Noah Wang
6657e84f1436864bccbdcc5a
Quehry Que
6657e84f1436864bccbdcc5b
Ruibo Liu
admin_assigned
2024-05-31T10:28:16.504Z
5e91f1554957053f606489ff
https://cdn-avatars.huggingface.co/v1/production/uploads/5e91f1554957053f606489ff/eJty7TksRLwuta0deZTmt.jpeg
Ruibo Liu
user
agi-css
6657e84f1436864bccbdcc5c
Sine Liu
admin_assigned
2024-05-31T10:29:57.165Z
618e3adaa834a6636ab624c2
/avatars/107110d8384e5c2e2f5021bdf2c5a79c.svg
Siyao Liu
user
sine
6657e84f1436864bccbdcc5d
Shawn Guo
admin_assigned
2024-05-31T10:30:04.610Z
61b8402cae47fface62479ae
/avatars/deaef201a6e0844c6805bc8c124c5308.svg
Shawn Guo
user
shawnGuo
6657e84f1436864bccbdcc5e
Soren Gao
6657e84f1436864bccbdcc5f
Wangchunshu Zhou
admin_assigned
2024-05-31T10:30:22.948Z
628c8598ef14f971b698107f
/avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
Zhou
user
Wangchunshu
6657e84f1436864bccbdcc60
Xinyue Zhang
6657e84f1436864bccbdcc61
Yizhi Zhou
admin_assigned
2024-05-31T10:31:02.006Z
634182c506a304bdc43037c8
https://cdn-avatars.huggingface.co/v1/production/uploads/1665237671855-noauth.png
YIZHI ZHOU
user
IanZhou11
6657e84f1436864bccbdcc62
Yubo Wang
claimed_verified
2025-01-30T08:41:39.648Z
636a35eff8d9af4aea181608
https://cdn-avatars.huggingface.co/v1/production/uploads/636a35eff8d9af4aea181608/s9GFJYd_QXVbg0Lb4JpKj.jpeg
yubo
user
ubowang
6657e84f1436864bccbdcc63
Yuelin Bai
6657e84f1436864bccbdcc64
Yuhan Zhang
admin_assigned
2024-05-31T10:32:02.704Z
6405e13aa577649430bd3f47
/avatars/de63f5f5e6a0917d63a34395323063cd.svg
Yuhan Zhang
user
hemomo
6657e84f1436864bccbdcc65
Yuxiang Zhang
claimed_verified
2024-07-22T07:12:37.879Z
6239386535384c2bcccd2a4f
/avatars/c62cefc6235210ca12b46beb62751d8b.svg
Yuxiang Zhang
user
Joelzhang
6657e84f1436864bccbdcc66
Zenith Wang
6657e84f1436864bccbdcc67
Zhenzhu Yang
6657e84f1436864bccbdcc68
Zijian Zhao
6657e84f1436864bccbdcc69
Jiajun Zhang
admin_assigned
2024-05-31T10:34:09.599Z
63f108072f7c0152e86fa2ea
/avatars/992ec3f9642df1688c5b20b3f6c938dd.svg
Jiajun Zhang
user
jjzhang
6657e84f1436864bccbdcc6a
Wanli Ouyang
6657e84f1436864bccbdcc6b
Wenhao Huang
claimed_verified
2024-10-13T20:30:43.392Z
65bb11cb00a03997849e9e85
/avatars/17022b0254192a837f4fe00d84389cda.svg
Wenhao Huang
user
StephenHuang
6657e84f1436864bccbdcc6c
Wenhu Chen
admin_assigned
2024-05-31T10:32:41.678Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model
  Series
Large Language Models (LLMs) have made great strides in recent years to
achieve unprecedented performance across different tasks. However, due to
commercial interest, the most competitive models like GPT, Gemini, and Claude
have been gated behind proprietary interfaces without disclosing the training
details. Recently, many institutions have open-sourced several strong LLMs like
LLaMA-3, comparable to existing closed-source LLMs. However, only the model's
weights are provided with most details (e.g., intermediate checkpoints,
pre-training corpus, and training code, etc.) being undisclosed. To improve the
transparency of LLMs, the research community has formed to open-source truly
open LLMs (e.g., Pythia, Amber, OLMo), where more details (e.g., pre-training
corpus and training code) are being provided. These models have greatly
advanced the scientific study of these large models including their strengths,
weaknesses, biases and risks. However, we observe that the existing truly open
LLMs on reasoning, knowledge, and coding tasks are still inferior to existing
state-of-the-art LLMs with similar model sizes. To this end, we open-source
MAP-Neo, a highly capable and transparent bilingual language model with 7B
parameters trained from scratch on 4.5T high-quality tokens. Our MAP-Neo is the
first fully open-sourced bilingual LLM with comparable performance compared to
existing state-of-the-art LLMs. Moreover, we open-source all details to
reproduce our MAP-Neo, where the cleaned pre-training corpus, data cleaning
pipeline, checkpoints, and well-optimized training/evaluation framework are
provided. Finally, we hope our MAP-Neo will enhance and strengthen the open
research community and inspire more innovations and creativities to facilitate
the further improvements of LLMs.
MAP-Neo, a fully open-sourced bilingual LLM with 7B parameters, achieves performance comparable to existing state-of-the-art LLMs, enhancing transparency and fostering innovation in the open research community.
Large Language Models LLMs GPT Gemini Claude LLaMA-3 Pythia Amber OLMo MAP-Neo bilingual language model pre-training corpus data cleaning pipeline checkpoints training/evaluation framework