Hogwild! Inference: Parallel LLM Generation via Concurrent Attention
Large Language Models (LLMs) have demonstrated the ability to tackle
increasingly complex tasks through advanced reasoning, long-form content
generation, and tool use. Solving these tasks often involves long
inference-time computations. In human problem solving, a common strategy to
expedite work is collaboration: by dividing the problem into sub-tasks,
exploring different strategies concurrently, etc. Recent research has shown
that LLMs can also operate in parallel by implementing explicit cooperation
frameworks, such as voting mechanisms or the explicit creation of independent
sub-tasks that can be executed in parallel. However, each of these frameworks
may not be suitable for all types of tasks, which can hinder their
applicability. In this work, we propose a different design approach: we run LLM
"workers" in parallel , allowing them to synchronize via a concurrently-updated
attention cache and prompt these workers to decide how best to collaborate. Our
approach allows the instances to come up with their own collaboration strategy
for the problem at hand, all the while "seeing" each other's partial progress
in the concurrent cache. We implement this approach via Hogwild! Inference: a
parallel LLM inference engine where multiple instances of the same LLM run in
parallel with the same attention cache, with "instant" access to each other's
generated tokens. Hogwild! inference takes advantage of Rotary Position
Embeddings (RoPE) to avoid recomputation while improving parallel hardware
utilization. We find that modern reasoning-capable LLMs can perform inference
with shared Key-Value cache out of the box, without additional fine-tuning.
67f60df2d0df7eccaae93eb0
Gleb Rodionov
claimed_verified
2025-04-10T08:00:46.736Z
67f77a13c4266c8ae9f3a299
/avatars/37059bcfdc20bae69172354985ee57aa.svg
Gleb
user
eqimp
67f60df2d0df7eccaae93eb1
Roman Garipov
claimed_verified
2025-04-09T14:36:56.157Z
6261af8040e04009e813a43d
https://cdn-avatars.huggingface.co/v1/production/uploads/6261af8040e04009e813a43d/Cc4I98uaePHE-YglLodlZ.jpeg
Roman Garipov
user
garipovroma
67f60df2d0df7eccaae93eb2
Alina Shutova
admin_assigned
2025-04-09T16:31:48.423Z
6492b61da34fecfae1175ae6
/avatars/08ee7ffbba95f9363a7f1cd19c5a72db.svg
Alina Shutova
user
Alshutova
67f60df2d0df7eccaae93eb3
George Yakushev
claimed_verified
2025-04-09T16:50:42.359Z
655f8492e3dda95b340822ae
/avatars/69e2bed8045faf35105e15f228db0567.svg
George
user
MrDarkTesla
67f60df2d0df7eccaae93eb4
Vage Egiazarian
admin_assigned
2025-04-09T16:32:06.480Z
64b6c0862662b49f164bfdb1
/avatars/5a89a3eb7923d9579ad74ff77d288d37.svg
Vage Egiazarian
user
Vahe527887
67f60df2d0df7eccaae93eb5
Anton Sinitsin
admin_assigned
2025-04-09T16:32:12.347Z
67cd69a25221b49814460129
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/dCjdggjVowuuI-cIM8pLa.png
Anton Sinitsin
user
grievousxtinkt
67f60df2d0df7eccaae93eb6
Denis Kuznedelev
admin_assigned
2025-04-09T16:32:18.990Z
629cf0475a13ba8233dd18c9
https://cdn-avatars.huggingface.co/v1/production/uploads/1654452258405-noauth.jpeg
Denis Kuznedelev
user
SpiridonSunRotator
67f60df2d0df7eccaae93eb7
Dan Alistarh
admin_assigned
2025-04-09T16:32:25.593Z
64d100c5d8d0927372e3d4c0
/avatars/91d9e4f1dab25b70d901783cdfcd2fd1.svg
Dan Alistarh
user
dalistarh
Hogwild! Inference: Parallel LLM Generation via Concurrent Attention
Large Language Models (LLMs) have demonstrated the ability to tackle
increasingly complex tasks through advanced reasoning, long-form content
generation, and tool use. Solving these tasks often involves long
inference-time computations. In human problem solving, a common strategy to
expedite work is collaboration: by dividing the problem into sub-tasks,
exploring different strategies concurrently, etc. Recent research has shown
that LLMs can also operate in parallel by implementing explicit cooperation
frameworks, such as voting mechanisms or the explicit creation of independent
sub-tasks that can be executed in parallel. However, each of these frameworks
may not be suitable for all types of tasks, which can hinder their
applicability. In this work, we propose a different design approach: we run LLM
"workers" in parallel , allowing them to synchronize via a concurrently-updated
attention cache and prompt these workers to decide how best to collaborate. Our
approach allows the instances to come up with their own collaboration strategy
for the problem at hand, all the while "seeing" each other's partial progress
in the concurrent cache. We implement this approach via Hogwild! Inference: a
parallel LLM inference engine where multiple instances of the same LLM run in
parallel with the same attention cache, with "instant" access to each other's
generated tokens. Hogwild! inference takes advantage of Rotary Position
Embeddings (RoPE) to avoid recomputation while improving parallel hardware
utilization. We find that modern reasoning-capable LLMs can perform inference
with shared Key-Value cache out of the box, without additional fine-tuning.
A parallel inference engine for LLMs allows concurrent task collaboration through a shared attention cache, improving efficiency without additional fine-tuning.
Large Language Models (LLMs) parallel inference cooperation frameworks voting mechanisms independent sub-tasks Hogwild! Inference attention cache generated tokens Rotary Position Embeddings (RoPE) Key-Value cache