Generalized Parallel Scaling with Interdependent Generations
Parallel LLM inference scaling involves sampling a set of N>1 responses for
a single input prompt. However, these N parallel responses tend to be
generated independently from each other, partitioning compute resources and
leaving potentially useful information in one generation untapped by others.
This is in contrast to response length scaling where past computation is used
in all future steps. For higher quality responses and response sets, we propose
Bridge to generate interdependent responses in parallel by rethinking batched
LLM hidden states as holistic tensors rather than independent slices. With only
a small amount (2.8%-5.1%) of new parameters, Bridge improves the relative mean
accuracy gains from reinforcement learning with verifiable rewards by up to 50%
and boosts consistency of correct responses. Trained once, Bridge scales to any
generation width, all with greater performance than independent generations,
unlocking a more general mode of parallel scaling that effectively leverages
information between sequences, compatible with any post-generation aggregation
technique.
68df7fbadf49fb0df1e03e05
Harry Dong
68df7fbadf49fb0df1e03e06
David Brandfonbrener
68df7fbadf49fb0df1e03e07
Eryk Helenowski
68df7fbadf49fb0df1e03e08
Yun He
68df7fbadf49fb0df1e03e09
Mrinal Kumar
68df7fbadf49fb0df1e03e0a
Han Fang
68df7fbadf49fb0df1e03e0b
Yuejie Chi
68df7fbadf49fb0df1e03e0c
Karthik Abinav Sankararaman
Generalized Parallel Scaling with Interdependent Generations
Parallel LLM inference scaling involves sampling a set of N>1 responses for
a single input prompt. However, these N parallel responses tend to be
generated independently from each other, partitioning compute resources and
leaving potentially useful information in one generation untapped by others.
This is in contrast to response length scaling where past computation is used
in all future steps. For higher quality responses and response sets, we propose
Bridge to generate interdependent responses in parallel by rethinking batched
LLM hidden states as holistic tensors rather than independent slices. With only
a small amount (2.8%-5.1%) of new parameters, Bridge improves the relative mean
accuracy gains from reinforcement learning with verifiable rewards by up to 50%
and boosts consistency of correct responses. Trained once, Bridge scales to any
generation width, all with greater performance than independent generations,
unlocking a more general mode of parallel scaling that effectively leverages
information between sequences, compatible with any post-generation aggregation
technique.
Bridge enhances parallel LLM inference by generating interdependent responses, improving accuracy and consistency with minimal additional parameters.
LLM parallel inference response length scaling batched LLM hidden states holistic tensors reinforcement learning verifiable rewards generation width post-generation aggregation