Skip to content

Support Streaming of Function Call Arguments #999

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

devtalker
Copy link
Contributor

Summary

This PR implements real-time streaming of function call arguments as requested in #834. Previously, function call arguments were only emitted after the entire function call was complete, causing poor user experience for large parameter generation.

Changes

  • Enhanced ChatCmplStreamHandler: Added real-time streaming of function call arguments during generation
  • New streaming logic: Function call arguments now stream incrementally as they are generated, similar to text content
  • Backward compatibility: Maintains existing behavior for completed function calls
  • Comprehensive testing: Added tests for both OpenAI and LiteLLM models
  • Example implementation: Created demonstration code showing the new streaming capability

Closes #834

@devtalker devtalker marked this pull request as ready for review July 3, 2025 11:01
@devtalker devtalker changed the title # Support Streaming of Function Call Arguments Support Streaming of Function Call Arguments Jul 3, 2025
@devtalker
Copy link
Contributor Author

Hey @seratch @rm-openai! 👋

Would you mind taking a look at this PR? I've implemented real-time streaming for function call arguments (fixes #834). Basically, instead of waiting for the entire function call to complete, users now see the arguments being built up incrementally as the LLM generates them. This should make the experience much smoother for things like code generation and API building.

Let me know what you think! 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support Streaming of Function Call Arguments
1 participant