feat(models): add support for Amazon Bedrock native API #1115

leandrodamascena · 2025-03-31T20:36:48Z

Summary

This pull request introduces the new AmazonBedrockServerModel class that provides direct access to Amazon Bedrock's foundation models via the Bedrock API. The implementation leverages the Bedrock converse API endpoint, allowing users to interact with all available Bedrock models through a consistent interface while maintaining the flexibility to configure model-specific parameters.

Implementation Details

Created AmazonBedrockServerModel class that extends ApiModel
Implemented support for the Bedrock Runtime converse API
Added flexible boto3 client configuration:
- Users can provide their own pre-configured boto3 client
- Default client creation with standard configurations when not provided
Added comprehensive parameter handling:
- Model-specific inference configurations (temperature, max_tokens, etc.)
- Guardrails configuration for content safety
- Custom role conversion mapping for model compatibility
- Support for flattened message formatting
Implemented proper error handling and response parsing
Added conversion logic for different message role formats across Bedrock models

Documentation Changes

Updated documentation with examples of how to use AmazonBedrockServerModel
Added parameter descriptions and usage patterns
Included model-specific configuration examples for popular Bedrock models

Testing

Added unit tests for the new class

Additional Notes

I've added detailed comments throughout the code to explain the implementation choices and help future contributors understand the design patterns. The implementation follows the project's established patterns while accommodating the requirements of the Bedrock API.

User experience

Basic Usage

# !pip install smolagents[aws_sdk]
from smolagents import CodeAgent, AmazonBedrockServerModel

model = AmazonBedrockServerModel(model_id="anthropic.claude-3-sonnet-20240229-v1:0")
agent = CodeAgent(tools=[], model=model, add_base_tools=True)

agent.run(
    "Could you give me the 118th number in the Fibonacci sequence?",
)

Advanced Configuration

import boto3
from smolagents import AmazonBedrockServerModel

# Create a custom Bedrock client
bedrock_client = boto3.client(
    'bedrock-runtime',
    region_name='us-east-1',
    aws_access_key_id='YOUR_ACCESS_KEY',
    aws_secret_access_key='YOUR_SECRET_KEY'
)

# Initialize with comprehensive configuration
model = AmazonBedrockServerModel(
    model_id="us.amazon.nova-pro-v1:0",
    boto3_client=bedrock_client,  # Use custom client
    inference_config={
        "temperature": 0.5,
        "top_p": 0.9,
        "max_tokens": 1000
    },
    guardrail_config={
        "guardrail_id": "my-custom-guardrail",
        "guardrail_version": "DRAFT"
    },
    bedrock_api_kwargs={
        "performanceConfig": {
            "latency": "standard"
        }
    }
)

agent = CodeAgent(tools=[], model=model, add_base_tools=True)

agent.run(
    "Could you give me the 118th number in the Fibonacci sequence?",
)

Let me know if you'd like any additional changes or have questions about the implementation.

albertvillanova

Awesome contribution! Really happy to see this new model supported!

I think you already tested it locally and the model calls work as expected: do you know if some additional test could be implemented?

Please find some comments/suggestions below to align your implementation better with the rest of the models.

pyproject.toml

src/smolagents/models.py

HuggingFaceDocBuilderDev · 2025-04-01T06:53:20Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

leandrodamascena · 2025-04-01T07:28:55Z

Hi @albertvillanova, thanks for the super quick review, I really appreciate it! I'm off work today until 4PM WEST, but I hope to respond to all comments today.

albertvillanova · 2025-04-01T08:27:54Z

No hurry at all, @leandrodamascena! Let me know if you need any help from my side.

leandrodamascena

Hey @albertvillanova! Thanks for all the feedback, I addressed all of them and it's probably read to another review.

I attached a quick video showing this working and I can create others with specific settings if you want to see this working.

Screen.Recording.2025-04-02.at.16.12.39.mov

pyproject.toml

src/smolagents/models.py

leandrodamascena · 2025-04-03T14:17:04Z

My bad here! I didn't saw that projects support python 3.10+ and I'm using typing.override from Python 3.12+. I'll remove it since you don't install typing-extensions for backward compatibility.

albertvillanova

Thanks a lot! Just some minor comments/suggestions below and we are ready to merge!

src/smolagents/models.py

albertvillanova · 2025-04-03T16:23:12Z

src/smolagents/models.py

+        # The Bedrock API does not support the `type` key in requests.
+        # This block of code modifies the object to meet Bedrock's requirements.
+        for message in messages.get("messages", []):
+            for content in message.get("content", []):


I think you should add an if condition here to apply this modification only if message["content"] is an instance of list. Note that it could also be an instance of str.

Can you help me here? In what situations can it be an instance of str? I tried many scenarios here and couldn't get an instance of str.
In Bedrock, it should always be an instance of list. And since I am just passing the array to bedrock, I expect it to always be a list. If it can be a str, then I need to do some logic to transform it.

client.converse( modelId='string', messages=[ { 'role': 'user'|'assistant', 'content': [ ... ] } }

We implemented the flatten_messages_as_text parameter to handle this behavior: depending on the provider, some models do require a str content, others require a list content and others support both formats.

Hmmm so in Bedrock I need to set the default to False always?

Yes, if no model can be passed a str content...

Yes! This is not so much about the model, but more about the bedrock-runtime API, and the API is the same for all models available in Bedrock. So I removed this field from the AmazonBedrockServerModel class and I am passing False to ApiModel.

I have already sent a commit.

src/smolagents/models.py

leandrodamascena · 2025-04-04T13:13:17Z

Hi @albertvillanova! Thanks for your super detailed review! I've addressed most of the comments and I think the one about messages is missing. I've also added a new docstring for the ApiModel class to make it documented, but I don't know if it makes sense, I might revert it.

albertvillanova

Great work! Thanks a lot for your effort and for submitting such a high-quality PR. I appreciate your contribution and really enjoyed reviewing it.

Just a few nits below and we are good to merge!

src/smolagents/models.py

leandrodamascena · 2025-04-04T15:59:55Z

Great work! Thanks a lot for your effort and for submitting such a high-quality PR. I appreciate your contribution and really enjoyed reviewing it.

Just a few nits below and we are good to merge!

Thank you so much for your patience and review @albertvillanova! Not knowing the codebase and all the features is a challenge, but you led me to the success. I just made one suggested change and commented on the other.

albertvillanova · 2025-04-04T16:18:04Z

Thanks again for your awesome contribution! 🤗

leandrodamascena · 2025-04-04T16:25:30Z

Thanks again for your awesome contribution! 🤗

Thanks for all the support and please fell free to reach me out if I can help with any bug/feature for this new integration.

aymeric-roucher · 2025-04-07T18:24:58Z

@leandrodamascena I'm reworking the cleaning of messages in https://github.com/huggingface/smolagents/pull/1002/files, and I wonder about one of your comments about bedrock chat template: we've got two possible templates

[{"role": "user", "content": [{"type": "text", "text": "Hello"}]}]
[{"role": "user", "content": "Hello"}]

The idea of the PR https://github.com/huggingface/smolagents/pull/1002/files is to flatten by default messages to format 2.
Would that work? If not, do you know if it's model specific or general?

leandrodamascena · 2025-04-07T18:35:55Z

Hey @aymeric-roucher let me see what you're doing and will reply this comment today.

leandrodamascena · 2025-04-07T19:53:28Z

@leandrodamascena I'm reworking the cleaning of messages in #1002 (files), and I wonder about one of your comments about bedrock chat template: we've got two possible templates

[{"role": "user", "content": [{"type": "text", "text": "Hello"}]}]

[{"role": "user", "content": "Hello"}]

The idea of the PR #1002 (files) is to flatten by default messages to format 2. Would that work? If not, do you know if it's model specific or general?

Hey @aymeric-roucher, neither of those payloads work with Bedrock because, when working with text, the Bedrock API only supports payloads in the following format:

messages = [
    {
        'role': 'user',
        'content': [
            {
                'text': "..."
            }
        ]
    }
]

You can see that the content does not include a type key. Instead, the API uses the key itself (e.g., text, document, or video) as the identifier. Currently, Bedrock supports "text" and, in some cases, "document" and "video" (though not all models support these types). Additionally, there is no way to pass flattened messages directly to the content key, due to the payload format.

The bedrock-runtime API is an abstraction that works with multiple models under the hood, but the exposed API is the same for all of them. You can check the supported features per model on this page.

Can I help with something in the PR #1002 ?

Adding support for Amazon Bedrock API

fd9954e

leandrodamascena mentioned this pull request Mar 31, 2025

Native Bedrock server model (willing to contribute) #1092

Closed

Bedrock runtime messages

d8a38ff

albertvillanova reviewed Apr 1, 2025

View reviewed changes

Addressing Albert's feedback

7def619

leandrodamascena commented Apr 2, 2025

View reviewed changes

leandrodamascena requested a review from albertvillanova April 2, 2025 15:29

leandrodamascena added 2 commits April 3, 2025 15:22

Removing override typing since it's not support in python 3.10/11

273a290

Making Ruff happy

e5b578c

albertvillanova reviewed Apr 3, 2025

View reviewed changes

leandrodamascena added 2 commits April 4, 2025 14:07

Addressing Albert's feedback

63d7355

Adding docstring

ec8fedb

leandrodamascena requested a review from albertvillanova April 4, 2025 13:12

Aliging with Bedrock API

008bc14

albertvillanova approved these changes Apr 4, 2025

View reviewed changes

src/smolagents/models.py Outdated Show resolved Hide resolved

src/smolagents/models.py Outdated Show resolved Hide resolved

src/smolagents/models.py Outdated Show resolved Hide resolved

src/smolagents/models.py Outdated Show resolved Hide resolved

Addressing Albert's feedback

b90bc20

leandrodamascena requested a review from albertvillanova April 4, 2025 16:00

Addressing Albert's feedback

cc4ee86

albertvillanova merged commit 3af67af into huggingface:main Apr 4, 2025
4 checks passed

feat(models): add support for Amazon Bedrock native API #1115

feat(models): add support for Amazon Bedrock native API #1115

Uh oh!

Conversation

leandrodamascena commented Mar 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Implementation Details

Documentation Changes

Testing

Additional Notes

User experience

Basic Usage

Advanced Configuration

Uh oh!

albertvillanova left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Apr 1, 2025

Uh oh!

leandrodamascena commented Apr 1, 2025

Uh oh!

albertvillanova commented Apr 1, 2025

Uh oh!

leandrodamascena left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

leandrodamascena commented Apr 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

albertvillanova left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

albertvillanova Apr 3, 2025

Choose a reason for hiding this comment

Uh oh!

leandrodamascena Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

albertvillanova Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

leandrodamascena Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

albertvillanova Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

leandrodamascena Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

leandrodamascena Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

leandrodamascena commented Apr 4, 2025

Uh oh!

leandrodamascena commented Mar 31, 2025 •

edited

Loading

leandrodamascena commented Apr 3, 2025 •

edited

Loading