Integrating Large Language Models with Effect AI: A Comprehensive Guide

Introduction

In today’s rapidly evolving tech landscape, integrating large language models (LLMs) into applications has become essential for developers. Whether you’re crafting content, analyzing data, or building user interfaces, AI-driven capabilities can significantly enhance your product’s functionality and user experience. However, the path to successful LLM integration is fraught with challenges, from network errors to provider limitations. This article explores how Effect AI’s integration packages can streamline this process, offering flexibility and provider independence.

Why Choose Effect for AI Integration?

Effect AI’s packages provide simple, compositional building blocks for modeling interactions with LLMs in a safe, declarative, and modular manner. Here’s what you can achieve:

  • Provider-Agnostic Business Logic: Define your LLM interactions once and easily switch between supported providers without altering your business logic.
  • Testing LLM Interactions: Use mock implementations to ensure your AI-dependent logic performs as expected.
  • Structured Concurrency: Manage parallel LLM calls, cancel outdated requests, and implement streaming or «racing» between providers safely.
  • Enhanced Observability: Utilize built-in tracing, logging, and metrics to identify performance bottlenecks or production failures.

Understanding the Effect AI Ecosystem

Effect AI’s ecosystem comprises specialized packages, each serving a distinct purpose:

  • @effect/ai: The core package defining provider-independent services and abstractions for LLM interactions.
  • @effect/ai-openai: Specific implementations based on OpenAI’s API.
  • @effect/ai-anthropic: Specific implementations based on Anthropic’s API.

This architecture allows you to describe LLM interactions using provider-independent services and plug in specific implementations at runtime.

Key Concepts

Provider-Agnostic Programming

Effect AI’s core philosophy is provider-agnostic programming. Instead of hardcoding API calls to a specific LLM provider, you describe interactions using universal services from the @effect/ai package.

import { Completions } from "@effect/ai"
import { Effect } from "effect"

const generateDadJoke = Effect.gen(function*() {
  const completions = yield* Completions.Completions
  const response = yield* completions.create("Generate a dad joke")
  return response
})

AiModel Abstraction

To bridge the gap between provider-independent business logic and specific LLM providers, Effect introduces the AiModel abstraction. This represents a specific LLM from a provider that can fulfill service requirements like Completions or Embeddings.

import { OpenAiCompletions } from "@effect/ai-openai"
import { Effect } from "effect"

const Gpt4o = OpenAiCompletions.model("gpt-4o")

const main = Effect.gen(function*() {
  const gpt4o = yield* Gpt4o
  const response = yield* gpt4o.provide(generateDadJoke)
  console.log(response.text)
})

End-to-End Example

Let’s walk through a complete example of setting up LLM interactions using Effect:

  1. Define Provider-Agnostic AI Interaction
  2. Create an AiModel for a Specific Provider and Model
  3. Develop a Program Using the Model
  4. Create a Layer Providing the OpenAI Client
  5. Provide an HTTP Client Implementation
  6. Run the Program with Dependencies
import { OpenAiClient, OpenAiCompletions } from "@effect/ai-openai"
import { Completions } from "@effect/ai"
import { NodeHttpClient } from "@effect/platform-node"
import { Config, Effect, Layer } from "effect"

const generateDadJoke = Effect.gen(function*() {
  const completions = yield* Completions.Completions
  const response = yield* completions.create("Generate a dad joke")
  return response
})

const Gpt4o = OpenAiCompletions.model("gpt-4o")

const main = Effect.gen(function*() {
  const gpt4o = yield* Gpt4o
  const response = yield* gpt4o.provide(generateDadJoke)

  console.log(response.text)
})

const OpenAi = OpenAiClient.layerConfig({
  apiKey: Config.redacted("OPENAI_API_KEY")
})

const OpenAiWithHttp = Layer.provide(OpenAi, NodeHttpClient.layerUndici)

main.pipe(
  Effect.provide(OpenAiWithHttp),
  Effect.runPromise
)

Advanced Features

Error Handling

Effect excels in robust error handling, crucial when dealing with LLMs where failure scenarios can be complex. Errors are typed and can be handled explicitly.

import { AiResponse, AiRole } from "@effect/ai"
import { Effect } from "effect"

class RateLimitError extends Data.TaggedError("RateLimitError") {}
class InvalidInputError extends Data.TaggedError("InvalidInputError") {}

const withErrorHandling = generateDadJoke.pipe(
  Effect.catchTags({
    RateLimitError: (error) =>
      Effect.logError("Rate limited, retrying in a moment").pipe(

        Effect.delay("1 seconds"),
        Effect.andThen(generateDadJoke)
      ),
    InvalidInputError: (error) =>
      Effect.succeed(AiResponse.AiResponse.fromText({
        role: AiRole.model,
        content: "I couldn't generate a joke right now."
      }))
  })
)

Structured Execution Plans

For complex scenarios requiring high reliability across multiple providers, Effect offers the powerful AiPlan abstraction.

import { AiPlan } from "@effect/ai"

import { OpenAiCompletions } from "@effect/ai-openai"
import { AnthropicCompletions } from "@effect/ai-anthropic"
import { Data, Effect, Schedule } from "effect"

const DadJokePlan = AiPlan.fromModel(OpenAiCompletions.model("gpt-4o"), {
  attempts: 3,
  schedule: Schedule.exponential("100 millis"),
  while: (error: NetworkError | ProviderOutage) =>
    error._tag === "NetworkError"
}).pipe(
  AiPlan.withFallback({
    model: AnthropicCompletions.model("claude-3-7-sonnet-latest"),
  })
)

Concurrency Control

Effect’s structured concurrency model simplifies managing parallel LLM requests:

import { Effect } from "effect"

const concurrentDadJokes = Effect.all([
  generateDadJoke,
  generateDadJoke,
  generateDadJoke
], { concurrency: 2 })

Streaming Responses

Effect AI integrations support streaming responses using the Stream type:

import { Completions } from "@effect/ai"
import { Effect, Stream } from "effect"

const streamingJoke = Effect.gen(function*() {
  const completions = yield* Completions.Completions
  const stream = completions.stream("Tell me a long dad joke")
  return yield* stream.pipe(
    Stream.runForEach(chunk =>
      Effect.sync(() => {
        process.stdout.write(chunk.text)
      })
    )
  )
})

Conclusion

Whether you’re building an intelligent agent, an interactive chat, or a system leveraging LLMs for background tasks, Effect AI’s packages provide all the necessary tools and more. Our provider-agnostic approach ensures your code remains adaptable as the AI landscape evolves.

Ready to try Effect for your next AI application? Check out the Getting Started Guide. While Effect AI integration packages are in experimental/alpha stages, we encourage you to try them and provide feedback to help us improve and expand their capabilities.

We look forward to seeing your projects! Dive into the full documentation for a deeper understanding and join our community to share experiences and get support.