Skip to content

Flatten messages by default, except if image found in messages #1002

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

aymeric-roucher
Copy link
Collaborator

@aymeric-roucher aymeric-roucher commented Mar 17, 2025

In our Model subclasses, flatten_messages_as_text actually should allow three values to adapt to the idiosyncracies of different APIs: 'never', 'auto' (flatten if possible, i.e. if messages don't contain any image), and 'always'.

@aymeric-roucher aymeric-roucher marked this pull request as draft March 20, 2025 12:08
@aymeric-roucher aymeric-roucher marked this pull request as ready for review April 14, 2025 13:03
@aymeric-roucher
Copy link
Collaborator Author

@albertvillanova would be interested to hear your thoughts on this change!

Copy link
Member

@albertvillanova albertvillanova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before going into the details, just a general thought: you introduced a breaking change. Previous code where users have explicitly set a boolean value for flatten_messages_as_text will fail after merging this PR.

Maybe introducing a new parameter for the transition/deprecation? Also note that I find the current variable name flatten_messages_as_text a little confusing... So this could be the opportunity to introduce another parameter.

@albertvillanova
Copy link
Member

The naming confusion I mentioned before is because we are flattening message content, not the messages themselves.

Some random ideas about naming:

  • Flag:
    • flatten_(message)_content: bool
    • (message)_content_format: "auto", "string", "structured"
    • (message)_content_mode: "string", "blocks"
  • Content:
    • (plain) text content
    • structured/rich/multimodal content

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants