Developer-LLM Conversations: An Empirical Study of Interactions and
  Generated Code Quality
Large Language Models (LLMs) are becoming integral to modern software
development workflows, assisting developers with code generation, API
explanation, and iterative problem-solving through natural language
conversations. Despite widespread adoption, there is limited understanding of
how developers interact with LLMs in practice and how these conversational
dynamics influence task outcomes, code quality, and software engineering
workflows. To address this, we leverage CodeChat, a large dataset comprising
82,845 real-world developer-LLM conversations, containing 368,506 code snippets
generated across over 20 programming languages, derived from the WildChat
dataset. We find that LLM responses are substantially longer than developer
prompts, with a median token-length ratio of 14:1. Multi-turn conversations
account for 68% of the dataset and often evolve due to shifting requirements,
incomplete prompts, or clarification requests. Topic analysis identifies web
design (9.6% of conversations) and neural network training (8.7% of
conversations) as the most frequent LLM-assisted tasks. Evaluation across five
languages (i.e., Python, JavaScript, C++, Java, and C#) reveals prevalent and
language-specific issues in LLM-generated code: generated Python and JavaScript
code often include undefined variables (83.4% and 75.3% of code snippets,
respectively); Java code lacks required comments (75.9%); C++ code frequently
omits headers (41.1%) and C# code shows unresolved namespaces (49.2%). During a
conversation, syntax and import errors persist across turns; however,
documentation quality in Java improves by up to 14.7%, and import handling in
Python improves by 3.7% over 5 turns. Prompts that point out mistakes in code
generated in prior turns and explicitly request a fix are most effective for
resolving errors.
68c81748f1d159e090f25e4d
Suzhen Zhong
claimed_verified
2025-09-19T06:50:30.362Z
65dd617096d71c27a19a28d4
/avatars/8f599654d0e84b95cfb20a43e75e4bc9.svg
Suzhen Zhong
user
Suzhen
68c81748f1d159e090f25e4e
Ying Zou
68c81748f1d159e090f25e4f
Bram Adams
Developer-LLM Conversations: An Empirical Study of Interactions and
  Generated Code Quality
Large Language Models (LLMs) are becoming integral to modern software
development workflows, assisting developers with code generation, API
explanation, and iterative problem-solving through natural language
conversations. Despite widespread adoption, there is limited understanding of
how developers interact with LLMs in practice and how these conversational
dynamics influence task outcomes, code quality, and software engineering
workflows. To address this, we leverage CodeChat, a large dataset comprising
82,845 real-world developer-LLM conversations, containing 368,506 code snippets
generated across over 20 programming languages, derived from the WildChat
dataset. We find that LLM responses are substantially longer than developer
prompts, with a median token-length ratio of 14:1. Multi-turn conversations
account for 68% of the dataset and often evolve due to shifting requirements,
incomplete prompts, or clarification requests. Topic analysis identifies web
design (9.6% of conversations) and neural network training (8.7% of
conversations) as the most frequent LLM-assisted tasks. Evaluation across five
languages (i.e., Python, JavaScript, C++, Java, and C#) reveals prevalent and
language-specific issues in LLM-generated code: generated Python and JavaScript
code often include undefined variables (83.4% and 75.3% of code snippets,
respectively); Java code lacks required comments (75.9%); C++ code frequently
omits headers (41.1%) and C# code shows unresolved namespaces (49.2%). During a
conversation, syntax and import errors persist across turns; however,
documentation quality in Java improves by up to 14.7%, and import handling in
Python improves by 3.7% over 5 turns. Prompts that point out mistakes in code
generated in prior turns and explicitly request a fix are most effective for
resolving errors.
Analysis of real-world developer-LLM conversations reveals patterns in task outcomes, code quality, and common issues across multiple programming languages.
CodeChat large dataset developer-LLM conversations code snippets WildChat token-length ratio multi-turn conversations topic analysis web design neural network training language-specific issues undefined variables required comments omitted headers unresolved namespaces syntax errors import errors documentation quality import handling