Skip to content

chore: update Vercel AI SDK implementation to support latest version #42

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

chore: update Vercel AI SDK implementation to support latest version #42

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 16 additions & 36 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ In this example, we'll build a full-stack application that uses Retrieval Augmen

RAG is a powerful tool that combines the benefits of retrieval-based models and generative models. Unlike traditional chatbots that can struggle with maintaining up-to-date information or accessing domain-specific knowledge, a RAG-based chatbot uses a knowledge base created from crawled URLs to provide contextually relevant responses.

Incorporating Vercel's AI SDK into our application will allow us easily set up the chatbot workflow and utilize streaming more efficiently, particularly in edge environments, enhancing the responsiveness and performance of our chatbot.
Incorporating Vercel's AI SDK into our application will allow us easily set up the chatbot workflow and utilize streaming more efficiently, enhancing the responsiveness and performance of our chatbot.

By the end of this tutorial, you'll have a context-aware chatbot that provides accurate responses without hallucination, ensuring a more effective and engaging user experience. Let's get started on building this powerful tool ([Full code listing](https://github.com/pinecone-io/pinecone-vercel-example/blob/main/package.json)).

Expand All @@ -18,10 +18,10 @@ To create a new Next.js app, run the following command:
npx create-next-app chatbot
```

Next, we'll add the `ai` package:
Next, we'll add the `ai` package and the OpenAI AI SDK Provider:

```bash
npm install ai
npm install ai @ai-sdk/openai
```

You can use the [full list](https://github.com/pinecone-io/pinecone-vercel-example/blob/main/package.json) of dependencies if you'd like to build along with the tutorial.
Expand Down Expand Up @@ -144,26 +144,9 @@ The useful `useChat` hook will manage the state for the messages displayed in th
Next, we'll set up the Chatbot API endpoint. This is the server-side component that will handle requests and responses for our chatbot. We'll create a new file called `api/chat/route.ts` and add the following dependencies:

```ts
import { Configuration, OpenAIApi } from "openai-edge";
import { Message, OpenAIStream, StreamingTextResponse } from "ai";
```

The first dependency is the `openai-edge` package which makes it easier to interact with OpenAI's APIs in an edge environment. The second dependency is the `ai` package which we'll use to define the `Message` and `OpenAIStream` types, which we'll use to stream back the response from OpenAI back to the client.

Next initialize the OpenAI client:

```ts
// Create an OpenAI API client (that's edge friendly!)
const config = new Configuration({
apiKey: process.env.OPENAI_API_KEY,
});
const openai = new OpenAIApi(config);
```

To define this endpoint as an edge function, we'll define and export the `runtime` variable

```ts
export const runtime = "edge";
import { Message, streamText } from "ai";
import { openai } from "@ai-sdk/openai";
import { getContext } from "@/utils/context";
```

Next, we'll define the endpoint handler:
Expand All @@ -187,25 +170,22 @@ export async function POST(req: Request) {
];

// Ask OpenAI for a streaming chat completion given the prompt
const response = await openai.createChatCompletion({
model: "gpt-3.5-turbo",
stream: true,
const response = await streamText({
model: openai("gpt-4o-mini"),
messages: [
...prompt,
...messages.filter((message: Message) => message.role === "user"),
],
});
// Convert the response into a friendly text-stream
const stream = OpenAIStream(response);
// Respond with the stream
return new StreamingTextResponse(stream);
return response.toDataStreamResponse();
} catch (e) {
throw e;
}
}
```

Here we deconstruct the messages from the post, and create our initial prompt. We use the prompt and the messages as the input to the `createChatCompletion` method. We then convert the response into a stream and return it to the client. Note that in this example, we only send the user's messages to OpenAI (as opposed to including the bot's messages as well).
Here we deconstruct the messages from the post, get the context using the `getContext` function, and create our initial prompt. We use the prompt and the messages as the input to the `streamText` function from the AI SDK. We then return the response as a data stream. Note that in this example, we only send the user's messages to the AI model (as opposed to including the bot's messages as well).

<!-- Add snapshot of simple chat -->

Expand Down Expand Up @@ -553,22 +533,22 @@ useEffect(() => {
}, [messages, gotMessages]);
```

## Running tests
## Running tests

The pinecone-vercel-starter uses [Playwright](https://playwright.dev) for end to end testing.
The pinecone-vercel-starter uses [Playwright](https://playwright.dev) for end to end testing.

To run all the tests:
To run all the tests:

```
npm run test:e2e
```

By default, when running locally, if errors are encountered, Playwright will open an HTML report showing which
By default, when running locally, if errors are encountered, Playwright will open an HTML report showing which
tests failed and for which browser drivers.

## Displaying test reports locally
## Displaying test reports locally

To display the latest test report locally, run:
To display the latest test report locally, run:
```
npm run test:show
```
Expand Down
3 changes: 2 additions & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,13 @@
"test:show": "playwright show-report"
},
"dependencies": {
"@ai-sdk/openai": "^0.0.54",
"@pinecone-database/doc-splitter": "^0.0.1",
"@pinecone-database/pinecone": "^2.1.0",
"@types/node": "20.4.0",
"@types/react": "18.2.14",
"@types/react-dom": "18.2.6",
"ai": "^2.1.16",
"ai": "^3.3.19",
"cheerio": "^1.0.0-rc.12",
"eslint": "8.44.0",
"eslint-config-next": "13.4.9",
Expand Down
48 changes: 19 additions & 29 deletions src/app/api/chat/route.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,31 +1,21 @@
import { Configuration, OpenAIApi } from 'openai-edge'
import { Message, OpenAIStream, StreamingTextResponse } from 'ai'
import { getContext } from '@/utils/context'

// Create an OpenAI API client (that's edge friendly!)
const config = new Configuration({
apiKey: process.env.OPENAI_API_KEY
})
const openai = new OpenAIApi(config)

// IMPORTANT! Set the runtime to edge
export const runtime = 'edge'
import { Message, streamText } from "ai";
import { openai } from "@ai-sdk/openai";
import { getContext } from "@/utils/context";

export async function POST(req: Request) {
try {

const { messages } = await req.json()
const { messages } = await req.json();
console.log(messages)

// Get the last message
const lastMessage = messages[messages.length - 1]
const lastMessage = messages[messages.length - 1];

// Get the context from the last message
const context = await getContext(lastMessage.content, '')

const context = await getContext(lastMessage.content, "");

const prompt = [
{
role: 'system',
role: "system",
content: `AI assistant is a brand new, powerful, human-like artificial intelligence.
The traits of AI include expert knowledge, helpfulness, cleverness, and articulateness.
AI is a well-behaved and well-mannered individual.
Expand All @@ -41,19 +31,19 @@ export async function POST(req: Request) {
AI assistant will not invent anything that is not drawn directly from the context.
`,
},
]
];

// Ask OpenAI for a streaming chat completion given the prompt
const response = await openai.createChatCompletion({
model: 'gpt-3.5-turbo',
stream: true,
messages: [...prompt, ...messages.filter((message: Message) => message.role === 'user')]
})
const response = await streamText({
model: openai("gpt-4o-mini"),
messages: [
...prompt,
...messages.filter((message: Message) => message.role === "user"),
],
});
// Convert the response into a friendly text-stream
const stream = OpenAIStream(response)
// Respond with the stream
return new StreamingTextResponse(stream)
return response.toDataStreamResponse();
} catch (e) {
throw (e)
throw e;
}
}
}