Skip to content

Implement general awsEndpoint + lambda integration #428

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

Implement general awsEndpoint + lambda integration #428

wants to merge 2 commits into from

Conversation

EikeKohl
Copy link

@EikeKohl EikeKohl commented Sep 5, 2023

This small refactoring PR implements a general awsEndpoint, enabeling the user to not only connect sagemaker endpoints, but also lambda function URLs that require AWS_IAM authentication.
I had to implement this change for my personal setup, because upon requesting an LLM generation, the lambda performs additional scaling logic to provision / delete the sagemaker endpoint to save LLM hosting costs. I think that everyone could benefit from this functionality.

@nsarrazin
Copy link
Contributor

Hello and thanks for contributing!

On paper this looks good to me, I haven't tested yet but what do you think @philschmid ?

@philschmid
Copy link

Hello and thanks for contributing!

On paper this looks good to me, I haven't tested yet but what do you think @philschmid ?

Agree! Looks good, but we need to make sure both ways work and should have some checks that people will not provide, e.g. ecs or something as service.

@EikeKohl
Copy link
Author

EikeKohl commented Sep 8, 2023

Thank you for considering my contribution 🙏. I actually tested it with both, Lambda Function URL and Sagemaker invocation URL and it worked 🥳. @philschmid If ecs is specified, the build will fail with a validation error from Zod, because I defined the service parameter to be a Union of the Literals lambda and sagemaker.

P.S. I noticed the workflow failed with a linting error, so I pushed another commit after I ran npm run format to hopefully fix it.

@nsarrazin nsarrazin self-requested a review October 2, 2023 08:45
@nsarrazin nsarrazin self-assigned this Oct 2, 2023
@nsarrazin
Copy link
Contributor

Hi @EikeKohl, sorry for taking so long to take a look at this PR. I've added support for your changes in this PR with more modular backends for the future.

Can you take a look and let me know if this works with your lambda workflow for now?

@EikeKohl
Copy link
Author

EikeKohl commented Nov 2, 2023

Hi @EikeKohl, sorry for taking so long to take a look at this PR. I've added support for your changes in this PR with more modular backends for the future.

Can you take a look and let me know if this works with your lambda workflow for now?

Looks good to me and works for my workflow 🙂 Thank you!

@nsarrazin nsarrazin mentioned this pull request Nov 2, 2023
Merged
@nsarrazin
Copy link
Contributor

Cool, going to close this for now then, as we'll merge the other PR soon. I'll ping you when it's done!

@nsarrazin nsarrazin closed this Nov 2, 2023
@nsarrazin nsarrazin removed their assignment Nov 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nsarrazin nsarrazin Awaiting requested review from nsarrazin

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@EikeKohl @nsarrazin @philschmid