Modular backends & support for openAI & AWS endpoints #541

nsarrazin · 2023-11-01T15:39:54Z

This PR adds support for openAI endpoint with a refactor that make it easier to handle adding independent backends with library dependencies in the future. For example if you want to add gpt-3.5-turbo:

Add to your MODELS :

{
      "name": "gpt-3.5-turbo",
      "displayName": "ChatGPT",
      "preprompt": "",
      "parameters": {
        "temperature": 0.1,
        "top_p": 0.95,
        "repetition_penalty": 1.2,
        "top_k": 50,
        "truncate": 1000,
        "max_new_tokens": 2048
      },
      "endpoints" : [{
        "type": "openai"
      }]
}

and add OPENAI_API_KEY to your .env.local

Closes #452
Closes #428

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Co-authored-by: Henry Chen <1474479+chenhunghan@users.noreply.github.com>

nsarrazin · 2023-11-03T09:14:12Z

Tested this locally with openAI, the llama.cpp local server with openAI adapter and also plain TGI and everything works great.

I think this is good to go imo!

nsarrazin · 2023-11-03T09:15:41Z

cc @coyotte508 if you want to take a quick peek, especially in endpoints.ts and models.ts

README.md

Co-authored-by: Mishig <mishig.davaadorj@coloradocollege.edu>

src/lib/server/endpoints/endpoints.ts

src/lib/server/models.ts

src/routes/conversation/[id]/+server.ts

wdhorton · 2023-11-08T16:46:51Z

I've been testing this branch and from what I've seen, it seems to break the LLM Summarization for the conversation title

Maemol

I'm using mostly Azure Open AI so I tested this branch with an Azure Endpoint. With some minor tweaks it works well so thanks for this PR!

I also tested the websearch but I had some issues with the GenerateQuery that didn't understand the preprompt (system prompt) correctly.

Appart from that, it's working well for me.

src/lib/server/endpoints/openai/endpointOai.ts

src/lib/server/websearch/generateQuery.ts

src/lib/server/endpoints/openai/openAIChatToTextGenerationStream.ts

src/lib/server/endpoints/openai/openAICompletionToTextGenerationStream.ts

Fadude · 2023-11-12T15:36:46Z

Looking forward for this PR 👀

nsarrazin · 2023-11-12T16:11:26Z

I was on holiday last week, I'll be back Tuesday and implement the feedback/merge this then!

Co-authored-by: Eliott C. <coyotte508@gmail.com>

nsarrazin · 2023-11-15T08:25:33Z

I also just added support directly for the llama.cpp API server directly since the protocol is a bit different!

* Fix the response Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Should use /completions Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Use async generator Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Use openai npm Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Fix generateFromDefaultEndpoint Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Fix last char become undefined Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Better support for system prompt Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Updates Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Revert Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Update README Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Default system prompt Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * remove sk- Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Fixing types Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Fix lockfile Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Move .optional Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Add try...catch and controller.error(error) Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * baseURL Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Format Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Fix types Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Fix again Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Better error message Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Update README Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> * Refactor backend to add support for modular backends * readme fix * readme update * add support for lambda on aws endpoint * upsate doc for lambda support * fix typecheck * make imports really optional * readme fixes * make endpoint creator async * Update README.md Co-authored-by: Henry Chen <1474479+chenhunghan@users.noreply.github.com> * Update README.md Co-authored-by: Henry Chen <1474479+chenhunghan@users.noreply.github.com> * Update src/lib/server/endpoints/openai/endpointOai.ts Co-authored-by: Henry Chen <1474479+chenhunghan@users.noreply.github.com> * trailing comma * Update README.md Co-authored-by: Mishig <mishig.davaadorj@coloradocollege.edu> * change readme example name * Update src/lib/server/models.ts Co-authored-by: Eliott C. <coyotte508@gmail.com> * fixed preprompt to use conversation.preprompt * Make openAI endpoint compatible with Azure OpenAI * surface errors in generation * Added support for llamacpp endpoint * fix llamacpp endpoint so it properly stops * Add llamacpp example to readme * Add support for legacy configs --------- Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> Co-authored-by: Hung-Han (Henry) Chen <chenhungh@gmail.com> Co-authored-by: Henry Chen <1474479+chenhunghan@users.noreply.github.com> Co-authored-by: Mishig <mishig.davaadorj@coloradocollege.edu> Co-authored-by: Eliott C. <coyotte508@gmail.com>

chenhunghan and others added 28 commits September 19, 2023 15:28

Fix the response

ca59222

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Should use /completions

484aa65

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Use async generator

b26989b

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Use openai npm

cc5a5b3

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Fix generateFromDefaultEndpoint

5d7b6a1

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Fix last char become undefined

1574fd5

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Better support for system prompt

a5bb367

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Updates

abdcc2b

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Merge branch 'huggingface:main' into main

2fd74ad

Revert

7542584

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Update README

6c1cb50

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Default system prompt

7284ec6

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

remove sk-

d950966

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Merge branch 'main' into main

f4dabb8

Fixing types

bebcce1

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Fix lockfile

58d4f92

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Move .optional

9132568

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Add try...catch and controller.error(error)

fac7113

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

baseURL

d6225ad

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Merge branch 'main' into main

0ec5862

Merge branch 'main' into main

084ef9a

Format

2220033

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Fix types

5cfdab4

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Fix again

cac6de8

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Better error message

b29a10d

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Update README

c30233a

Signed-off-by: Hung-Han (Henry) Chen <chenhungh@gmail.com>

Refactor backend to add support for modular backends

c7518e8

readme fix

7a1c73d

nsarrazin added enhancement New feature or request back This issue is related to the Svelte backend or the DB labels Nov 1, 2023

nsarrazin and others added 2 commits November 3, 2023 08:54

Update src/lib/server/endpoints/openai/endpointOai.ts

6d5fb5e

Co-authored-by: Henry Chen <1474479+chenhunghan@users.noreply.github.com>

trailing comma

b77e91b

mishig25 reviewed Nov 3, 2023

View reviewed changes

README.md Outdated Show resolved Hide resolved

Update README.md

e242674

Co-authored-by: Mishig <mishig.davaadorj@coloradocollege.edu>

coyotte508 approved these changes Nov 3, 2023

View reviewed changes

src/lib/server/endpoints/endpoints.ts Show resolved Hide resolved

src/lib/server/models.ts Outdated Show resolved Hide resolved

chenhunghan reviewed Nov 3, 2023

View reviewed changes

src/routes/conversation/[id]/+server.ts Show resolved Hide resolved

Maemol reviewed Nov 9, 2023

View reviewed changes

nsarrazin and others added 6 commits November 14, 2023 08:42

change readme example name

29c6155

Update src/lib/server/models.ts

697301b

Co-authored-by: Eliott C. <coyotte508@gmail.com>

fixed preprompt to use conversation.preprompt

4d5cde8

Make openAI endpoint compatible with Azure OpenAI

cca104c

surface errors in generation

552eb89

Added support for llamacpp endpoint

fbfcbb8

nsarrazin added 3 commits November 15, 2023 09:46

fix llamacpp endpoint so it properly stops

318cee1

Add llamacpp example to readme

8d0cce1

Add support for legacy configs

98c28cc

nsarrazin merged commit 9db8ced into main Nov 15, 2023

nsarrazin deleted the refacto/backend_openai branch November 15, 2023 09:11

mishig25 mentioned this pull request Jan 8, 2024

Add embedding models configurable, from both transformers.js and TEI #646

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Modular backends & support for openAI & AWS endpoints #541

Modular backends & support for openAI & AWS endpoints #541

Uh oh!

nsarrazin commented Nov 1, 2023 •

edited

Loading

Uh oh!

nsarrazin commented Nov 3, 2023

Uh oh!

nsarrazin commented Nov 3, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wdhorton commented Nov 8, 2023

Uh oh!

Maemol left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Fadude commented Nov 12, 2023

Uh oh!

nsarrazin commented Nov 12, 2023

Uh oh!

nsarrazin commented Nov 15, 2023

Uh oh!

Uh oh!

Modular backends & support for openAI & AWS endpoints #541

Modular backends & support for openAI & AWS endpoints #541

Uh oh!

Conversation

nsarrazin commented Nov 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nsarrazin commented Nov 3, 2023

Uh oh!

nsarrazin commented Nov 3, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wdhorton commented Nov 8, 2023

Uh oh!

Maemol left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Fadude commented Nov 12, 2023

Uh oh!

nsarrazin commented Nov 12, 2023

Uh oh!

nsarrazin commented Nov 15, 2023

Uh oh!

Uh oh!

nsarrazin commented Nov 1, 2023 •

edited

Loading