Developing AI Applications with Effect

Integrating with large language models (LLMs) has become essential for developing modern applications. Whether you’re generating content, analyzing data, or building conversational interfaces, adding AI-powered features to your application has the potential to enhance your product’s capabilities and improve user experience.

However, successfully integrating LLM-powered interactions into an application can be quite challenging. Developers must navigate a complex landscape of potential failures: network errors, provider outages, rate limits, and more, all while keeping the underlying application stable and responsive for the end user. In addition, the differences between LLM provider APIs can force developers to write brittle “glue code” which can become a significant source of technical debt.

Today, we are going to discuss Effect’s AI integration packages — a set of libraries designed to make working with LLMs simple, flexible, and provider-agnostic.

Why Effect for AI?

Effect’s AI packages provide simple, composable building blocks to model LLM interactions in a safe, declarative, and composable manner. With Effect’s AI integrations, you can:

🔌 Write Provider-Agnostic Business Logic

Define your LLM interactions once and plug in the specific provider you need later. Switch between any supported provider without changing your business logic.

🧪 Test LLM Interactions

Test your LLM interactions by simply providing mock service implementations during testing to ensure your AI-dependent business logic is executed in the way that you expect.

🧵 Utilize Structured Concurrency

Run concurrent LLM calls, cancel stale requests, stream partial results, or race multiple providers — all safely managed by Effect’s structured concurrency model.

🔍 Gain Deep Observability

Instrument your LLM interactions with Effect’s built-in tracing, logging, and metrics to identify performance bottlenecks or failures in production.

Understanding the Package Ecosystem

Effect’s AI ecosystem consists of several focused packages, each with a specific purpose:

• @effect/ai: The core package that defines provider-agnostic services and abstractions for interacting with LLMs

• @effect/ai-openai: Concrete implementations of AI services backed by the OpenAI’s Responses API

• @effect/ai-anthropic: Concrete implementations of AI services backed by the Anthropic’s Messages API

• @effect/ai-amazon-bedrock: Concrete implementations of AI services backed by Amazon Bedrock’s Converse API

• @effect/ai-google: Concrete implementations of AI services backed by Google’s Gemini API

This architecture allows you to describe your LLM interactions with provider-agnostic services, and the provide a concrete implementation once you are ready to run your program.

Core Concepts

Provider-Agnostic Programming

The central philosophy behind Effect’s AI integrations is provider-agnostic programming.

Instead of hardcoding calls to a specific LLM provider’s API, you describe your interaction using generic services provided by the base @effect/ai package.

Let’s look at a simple example to understand this concept better:

1
import { LanguageModel } from "@effect/ai"
2
import { Effect } from "effect"
3

4
// Define a provider-agnostic AI interaction
5
const generateDadJoke = LanguageModel.generateText({
6
  prompt: "Generate a dad joke",
7
})

Notice that this code doesn’t specify which model provider to use - it simply describes what we want to do (generate a dad joke), not how or where to do it.

This separation of concerns is at the heart of Effect’s approach to describing interactions with large language models.

The Model Abstraction

To bridge the gap between provider-agnostic business logic and concrete LLM providers, Effect introduces the Model abstraction.

An Model represents a specific LLM from a provider that can be used to satisfy service requirements, such as LanguageModel or EmbeddingsModel.

Here is an example of how you can create and use an Model designed to satisfy the LanguageModel service using OpenAI:

1
import { LanguageModel } from "@effect/ai"
2
import { OpenAiLanguageModel } from "@effect/ai-openai"
3
import { Console, Effect } from "effect"
4

5
const generateDadJoke = LanguageModel.generateText({
6
  prompt: "Generate a dad joke",
7
})
8

9
// Create an AiModel for OpenAI's GPT-4o
10
const Gpt4o = OpenAiLanguageModel.model("gpt-4o")
11

12
const main = generateDadJoke.pipe(
13
  // Log out the generated text
14
  Effect.flatMap((response) => Console.log(response.text)),
15
  // Provide the concrete model to the program
16
  Effect.provide(Gpt4o),
17
)

This approach offers several key benefits:

1. Reusability: You can reuse the same model for multiple operations
2. Flexibility: Easily switch between providers or models based on your needs
3. Abstractability: Extract your AI logic into services that hide implementation details

End-to-End Example

Now let’s walk through a complete example of setting up an LLM interaction with Effect:

1
import { LanguageModel } from "@effect/ai"
2
import { OpenAiClient, OpenAiLanguageModel } from "@effect/ai-openai"
3
import { NodeHttpClient } from "@effect/platform-node"
4
import { Config, Console, Effect, Layer } from "effect"
5

6
// 1. Define our provider-agnostic AI interaction
7
const generateDadJoke = LanguageModel.generateText({
8
  prompt: "Generate a dad joke",
9
})
10

11
// 2. Create an AiModel for a specific provider and model
12
const Gpt4o = OpenAiLanguageModel.model("gpt-4o")
13

14
// 3. Create a program that uses the model
15
const main = generateDadJoke.pipe(
16
  Effect.flatMap((response) => Console.log(response.text)),
17
  Effect.provide(Gpt4o),
18
)
19

20
// 4. Create a Layer that provides the OpenAI client
21
const OpenAi = OpenAiClient.layerConfig({
22
  apiKey: Config.redacted("OPENAI_API_KEY"),
23
})
24

25
// 5. Provide an HTTP client implementation
26
const OpenAiWithHttp = Layer.provide(OpenAi, NodeHttpClient.layerUndici)
27

28
// 6. Run the program with the provided dependencies
29
main.pipe(Effect.provide(OpenAiWithHttp), Effect.runPromise)

Advanced Features

Error Handling

One of Effect’s greatest strengths is its robust error handling, which is particularly valuable for LLM interactions where failure scenarios can be complex and varied. With Effect, these errors are typed and can be handled explicitly.

For example, if our generateDadJoke program were re-written to possibly fail with a RateLimitError or an InvalidInputError, we could write logic to handle those errors:

1
import { Response } from "@effect/ai"
2
import { Effect } from "effect"
3

11 collapsed lines
4
import { LanguageModel } from "@effect/ai"
5
import { Data } from "effect"
6

7
class RateLimitError extends Data.TaggedError("RateLimitError") {}
8
class InvalidInputError extends Data.TaggedError("InvalidInputError") {}
9

10
declare const generateDadJoke: Effect.Effect<
11
  LanguageModel.GenerateTextResponse<{}>,
12
  RateLimitError | InvalidInputError,
13
  LanguageModel.LanguageModel
14
>
15

16
const withErrorHandling = generateDadJoke.pipe(
17
  Effect.catchTags({
18
    RateLimitError: (error) =>
19
      Effect.logError("Rate limited, retrying in a moment").pipe(
20
        Effect.delay("1 seconds"),
21
        Effect.andThen(generateDadJoke),
22
      ),
23
    InvalidInputError: (error) =>
24
      Effect.succeed(
25
        new LanguageModel.GenerateTextResponse([
26
          Response.makePart("text", {
27
            text: "I could not generate a dad joke right now!",
28
          }),
29
        ]),
30
      ),
31
  }),
32
)

Structured Execution Plans

For more complex scenarios where you need reliability across multiple providers, Effect offers the powerful ExecutionPlan abstraction.

ExecutionPlan lets you create a structured execution plan for your Effect programs with built-in retry logic, fallback strategies, and error handling:

1
import { AiError, LanguageModel } from "@effect/ai"
2
import { AnthropicLanguageModel } from "@effect/ai-anthropic"
3
import { OpenAiLanguageModel } from "@effect/ai-openai"
4
import { Console, Data, Effect, ExecutionPlan, Schedule } from "effect"
5

6
const generateDadJoke = LanguageModel.generateText({
7
  prompt: "Generate a dad joke",
8
})
9

10
// Define domain-specific error types
11
class NetworkError extends Data.TaggedError("NetworkError") {}
12
class ProviderOutage extends Data.TaggedError("ProviderOutage") {}
13

14
// Build a resilient plan that:
15
// - Provides an LanguageModel which uses OpenAi's Chat Completions API
16
// - Attempts to call OpenAI up to 3 times
17
// - Waits with an exponential backoff between attempts
18
// - Only attempts the call to OpenAI if the error is a `NetworkError`
19
// - Falls back to using Anthropic otherwise
20
const DadJokePlan = ExecutionPlan.make(
21
  {
22
    provide: OpenAiLanguageModel.model("gpt-4o"),
23
    attempts: 3,
24
    schedule: Schedule.exponential("100 millis"),
25
    while: (error: AiError.AiError | NetworkError | ProviderOutage) =>
26
      error._tag === "NetworkError",
27
  },
28
  {
29
    provide: AnthropicLanguageModel.model("claude-3-7-sonnet-latest"),
30
  },
31
)
32

33
const program = generateDadJoke.pipe(
34
  Effect.flatMap((response) => Console.log(response.text)),
35
  Effect.withExecutionPlan(DadJokePlan),
36
)

With ExecutionPlan, you can:

• Create sophisticated retry policies with configurable backoff strategies
• Define fallback chains across multiple providers
• Specify which error types should trigger retries vs. fallbacks

This is particularly valuable for production systems where reliability is critical, as it allows you to leverage multiple LLM providers as fallbacks for one other, all while keeping your business logic provider-agnostic.

Concurrency Control

Effect’s structured concurrency model also makes it easy to manage concurrent model interactions:

1
import { LanguageModel } from "@effect/ai"
2
import { Effect } from "effect"
3

4
const generateDadJoke = LanguageModel.generateText({
5
  prompt: "Generate a dad joke",
6
})
7

8
// Generate multiple jokes concurrently
9
const concurrentDadJokes = Effect.all([generateDadJoke, generateDadJoke, generateDadJoke], {
10
  concurrency: 2,
11
}) // Limit to 2 concurrent requests

Streaming Responses

Effect’s AI integrations support streaming responses via Effect’s Stream type:

1
import { LanguageModel } from "@effect/ai"
2
import { Effect, Stream } from "effect"
3

4
// Use `LanguageModel.streamText` to stream parts of a response from a large
5
// language model provider
6
const streamingJoke = LanguageModel.streamText({
7
  prompt: "Tell me a long dad joke",
8
}).pipe(
9
  Stream.runForEach((part) =>
10
    Effect.sync(() => {
11
      if (part.type === "text-delta") {
12
        process.stdout.write(part.delta)
13
      }
14
    }),
15
  ),
16
)

Conclusion

Whether you’re building an intelligent agent, an interactive chat application, or a system that leverages LLMs for background tasks, Effect’s AI packages provide all the tools you need and more. Our provider-agnostic approach will ensure your code remains adaptable as the AI landscape continues to evolve.

Ready to try out Effect for your next AI application? Take a look at our Getting Started guide.

The Effect AI integration packages are currently in the experimental/alpha stage, but we encourage you to give them a try and provide feedback to help us improve and expand their capabilities.

We’re excited to see what you build! Check out the full documentation to dive deeper, and join our community to share your experiences and get help along the way.