Vercel AI SDK - Observability & Analytics

The Vercel AI SDK is the TypeScript toolkit designed to help developers build AI-powered applications with React, Next.js, Vue, Svelte, Node.js, and more.

The SDK supports tracing via OpenTelemetry. With the LangfuseExporter you can collect these traces in Langfuse.

Full Demo

Get Started

const result = await generateText({
  model: openai("gpt-4-turbo"),
  prompt: "Write a short story about a cat.",
  experimental_telemetry: { isEnabled: true },
});

LANGFUSE_SECRET_KEY="sk-lf-..."
LANGFUSE_PUBLIC_KEY="pk-lf-..."
LANGFUSE_BASEURL="https://cloud.langfuse.com" # 🇪🇺 EU region
# LANGFUSE_BASEURL="https://us.cloud.langfuse.com" # 🇺🇸 US region

import { LangfuseExporter } from "langfuse-vercel";
 
new LangfuseExporter({
  secretKey: "sk-lf-...",
  publicKey: "pk-lf-...",
  baseUrl: "https://cloud.langfuse.com", // 🇪🇺 EU region
  // baseUrl: "https://us.cloud.langfuse.com", // 🇺🇸 US region
});

npm install @vercel/otel langfuse-vercel @opentelemetry/api-logs @opentelemetry/instrumentation @opentelemetry/sdk-logs

/** @type {import('next').NextConfig} */
const nextConfig = {
  experimental: {
    instrumentationHook: true,
  },
};
 
module.exports = nextConfig;

import { registerOTel } from "@vercel/otel";
import { LangfuseExporter } from "langfuse-vercel";
 
export function register() {
  registerOTel({
    serviceName: "langfuse-vercel-ai-nextjs-example",
    traceExporter: new LangfuseExporter(),
  });
}

import { openai } from "@ai-sdk/openai";
import { generateText } from "ai";
import { NodeSDK } from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { LangfuseExporter } from "langfuse-vercel";
 
const sdk = new NodeSDK({
  traceExporter: new LangfuseExporter(),
  instrumentations: [getNodeAutoInstrumentations()],
});
 
sdk.start();
 
async function main() {
  const result = await generateText({
    model: openai("gpt-3.5-turbo"),
    maxTokens: 50,
    prompt: "Invent a new holiday and describe its traditions.",
    experimental_telemetry: {
      isEnabled: true,
      functionId: "my-awesome-function",
      metadata: {
        something: "custom",
        someOtherThing: "other-value",
      },
    },
  });
 
  console.log(result.text);
 
  await sdk.shutdown(); // Flushes the trace to Langfuse
}
 
main().catch(console.error);

Example Application

We created a sample repository (langfuse/langfuse-vercel-ai-nextjs-example) based on the next-openai template to showcase the integration of Langfuse with Next.js and Vercel AI SDK.

Customization

Group multiple executions in one trace

You can open a Langfuse trace and pass the trace ID to AI SDK calls to group multiple execution spans under one trace. The passed name in functionId will be the root span name of the respective execution.

import { randomUUID } from "crypto";
import { Langfuse } from "langfuse";
 
const langfuse = new Langfuse();
const parentTraceId = randomUUID();
 
langfuse.trace({
  id: parentTraceId,
  name: "holiday-traditions",
});
 
for (let i = 0; i < 3; i++) {
  const result = await generateText({
    model: openai("gpt-3.5-turbo"),
    maxTokens: 50,
    prompt: "Invent a new holiday and describe its traditions.",
    experimental_telemetry: {
      isEnabled: true,
      functionId: `holiday-tradition-${i}`,
      metadata: {
        langfuseTraceId: parentTraceId,
        langfuseUpdateParent: false, // Do not update the parent trace with execution results
      },
    },
  });
 
  console.log(result.text);
}
 
await langfuse.flushAsync();
await sdk.shutdown();

The resulting trace hierarchy will be:

Disable Tracking of Input/Output

By default, the exporter captures the input and output of each request. You can disable this behavior by setting the recordInputs and recordOutputs options to false.

Link Langfuse prompts to traces

You can link Langfuse prompts to Vercel AI SDK generations by setting the langfusePrompt property in the metadata field:

import { generateText } from "ai";
import { Langfuse } from "langfuse";
 
const langfuse = new Langfuse();
 
const fetchedPrompt = await langfuse.getPrompt("my-prompt");
 
const result = await generateText({
  model: openai("gpt-4o"),
  prompt: fetchedPrompt.prompt,
  experimental_telemetry: {
    isEnabled: true,
    metadata: {
      langfusePrompt: fetchedPrompt.toJSON(),
    },
  },
});

The resulting generation will have the prompt linked to the trace in Langfuse. Learn more about prompts in Langfuse here.

Pass Custom Attributes

All of the metadata fields are automatically captured by the exporter. You can also pass custom trace attributes to e.g. track users or sessions.

const result = await generateText({
  model: openai("gpt-4-turbo"),
  prompt: "Write a short story about a cat.",
  experimental_telemetry: {
    isEnabled: true,
    functionId: "my-awesome-function", // Trace name
    metadata: {
      langfuseTraceId: "trace-123", // Langfuse trace
      tags: ["story", "cat"], // Custom tags
      userId: "user-123", // Langfuse user
      sessionId: "session-456", // Langfuse session
      foo: "bar", // Any custom attribute recorded in metadata
    },
  },
});

Debugging

Enable the debug option to see the logs of the exporter.

new LangfuseExporter({ debug: true });

Troubleshooting

• If you deploy on Vercel, Vercel’s OpenTelemetry Collector is only available on Pro and Enterprise Plans (docs).
• You need to be on "ai": "^3.3.0" to use the telemetry feature as it was recently added. In case of any issues, please update to the latest version as this feature is under active development.
• On NextJS, make sure that you only have a single instrumentation file.
• If you use Sentry, make sure to either:
  • set skipOpenTelemetrySetup: true in Sentry.init
  • follow Sentry’s docs on how to manually set up Sentry with OTEL

Short-lived environments

In short-lived environments such as Vercel Cloud Functions, AWS Lambdas etc. you may force an export and flushing of spans after function execution and prior to environment freeze or shutdown by awaiting a call to the forceFlush method on the LangfuseExporter instance.

Learn more

See the telemetry documentation of the Vercel AI SDK for more information.

GitHub Discussions