Tracing in Effect

Although logs and metrics are useful to understand the behavior of individual services, they are not enough to provide a complete overview of the lifetime of a request in a distributed system.

In a distributed system, a request can span multiple services and each service can make multiple requests to other services to fulfill the request. In such a scenario, we need to have a way to track the lifetime of a request across multiple services to diagnose what services are the bottlenecks and where the request is spending most of its time.

Spans

A span represents a single unit of work or operation within a request. It provides a detailed view of what happened during the execution of that specific operation.

Each span typically contains the following information:

Span Component	Description
Name	Describes the specific operation being tracked.
Timing Data	Timestamps indicating when the operation started and its duration.
Log Messages	Structured logs capturing important events during the operation.
Attributes	Metadata providing additional context about the operation.

Spans are key building blocks in tracing, helping you visualize and understand the flow of requests through various services.

Traces

A trace records the paths taken by requests (made by an application or end-user) as they propagate through multi-service architectures, like microservice and serverless applications.

Without tracing, it is challenging to pinpoint the cause of performance problems in a distributed system.

A trace is made of one or more spans. The first span represents the root span. Each root span represents a request from start to finish. The spans underneath the parent provide a more in-depth context of what occurs during a request (or what steps make up a request).

Many Observability back-ends visualize traces as waterfall diagrams that may look something like this:

Trace Waterfall Diagram

Waterfall diagrams show the parent-child relationship between a root span and its child spans. When a span encapsulates another span, this also represents a nested relationship.

Creating Spans

You can add tracing to an effect by creating a span using the Effect.withSpan API. This helps you track specific operations within the effect.

Example (Adding a Span to an Effect)

1
import { Effect } from "effect"
2

3
// Define an effect that delays for 100 milliseconds
4
const program = Effect.void.pipe(Effect.delay("100 millis"))
5

6
// Instrument the effect with a span for tracing
7
const instrumented = program.pipe(Effect.withSpan("myspan"))

Instrumenting an effect with a span does not change its type. If you start with an Effect<A, E, R>, the result remains an Effect<A, E, R>.

Printing Spans

To print spans for debugging or analysis, you’ll need to install the required tracing tools. Here’s how to set them up for your project.

Installing Dependencies

Choose your package manager and install the necessary libraries:

# Install the main library for integrating OpenTelemetry with Effect
npm install @effect/opentelemetry

# Install the required OpenTelemetry SDKs for tracing and metrics
npm install @opentelemetry/sdk-trace-base
npm install @opentelemetry/sdk-trace-node
npm install @opentelemetry/sdk-trace-web
npm install @opentelemetry/sdk-metrics

# Install the main library for integrating OpenTelemetry with Effect
pnpm add @effect/opentelemetry

# Install the required OpenTelemetry SDKs for tracing and metrics
pnpm add @opentelemetry/sdk-trace-base
pnpm add @opentelemetry/sdk-trace-node
pnpm add @opentelemetry/sdk-trace-web
pnpm add @opentelemetry/sdk-metrics

# Install the main library for integrating OpenTelemetry with Effect
yarn add @effect/opentelemetry

# Install the required OpenTelemetry SDKs for tracing and metrics
yarn add @opentelemetry/sdk-trace-base
yarn add @opentelemetry/sdk-trace-node
yarn add @opentelemetry/sdk-trace-web
yarn add @opentelemetry/sdk-metrics

# Install the main library for integrating OpenTelemetry with Effect
bun add @effect/opentelemetry

# Install the required OpenTelemetry SDKs for tracing and metrics
bun add @opentelemetry/sdk-trace-base
bun add @opentelemetry/sdk-trace-node
bun add @opentelemetry/sdk-trace-web
bun add @opentelemetry/sdk-metrics

Printing a Span to the Console

Once the dependencies are installed, you can set up span printing using OpenTelemetry. Here’s an example showing how to print a span for an effect.

Example (Setting Up and Printing a Span)

1
import { Effect } from "effect"
2
import { NodeSdk } from "@effect/opentelemetry"
3
import { ConsoleSpanExporter, BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"
4

5
// Define an effect that delays for 100 milliseconds
6
const program = Effect.void.pipe(Effect.delay("100 millis"))
7

8
// Instrument the effect with a span for tracing
9
const instrumented = program.pipe(Effect.withSpan("myspan"))
10

11
// Set up tracing with the OpenTelemetry SDK
12
const NodeSdkLive = NodeSdk.layer(() => ({
13
  resource: { serviceName: "example" },
14
  // Export span data to the console
15
  spanProcessor: new BatchSpanProcessor(new ConsoleSpanExporter()),
16
}))
17

18
// Run the effect, providing the tracing layer
19
Effect.runPromise(instrumented.pipe(Effect.provide(NodeSdkLive)))
20
/*
21
Example Output:
22
{
23
  resource: {
24
    attributes: {
25
      'service.name': 'example',
26
      'telemetry.sdk.language': 'nodejs',
27
      'telemetry.sdk.name': '@effect/opentelemetry',
28
      'telemetry.sdk.version': '1.28.0'
29
    }
30
  },
31
  instrumentationScope: { name: 'example', version: undefined, schemaUrl: undefined },
32
  traceId: '673c06608bd815f7a75bf897ef87e186',
33
  parentId: undefined,
34
  traceState: undefined,
35
  name: 'myspan',
36
  id: '401b2846170cd17b',
37
  kind: 0,
38
  timestamp: 1733220735529855.5,
39
  duration: 102079.958,
40
  attributes: {},
41
  status: { code: 1 },
42
  events: [],
43
  links: []
44
}
45
*/

Understanding the Span Output

The output provides detailed information about the span:

Field	Description
`traceId`	A unique identifier for the entire trace, helping trace requests or operations as they move through an application.
`parentId`	Identifies the parent span of the current span, marked as `undefined` in the output when there is no parent span, making it a root span.
`name`	Describes the name of the span, indicating the operation being tracked (e.g., “myspan”).
`id`	A unique identifier for the current span, distinguishing it from other spans within a trace.
`timestamp`	A timestamp representing when the span started, measured in microseconds since the Unix epoch.
`duration`	Specifies the duration of the span, representing the time taken to complete the operation (e.g., `2895.769` microseconds).
`attributes`	Spans may contain attributes, which are key-value pairs providing additional context or information about the operation. In this output, it’s an empty object, indicating no specific attributes in this span.
`status`	The status field provides information about the span’s status. In this case, it has a code of 1, which typically indicates an OK status (whereas a code of 2 signifies an ERROR status)
`events`	Spans can include events, which are records of specific moments during the span’s lifecycle. In this output, it’s an empty array, suggesting no specific events recorded.
`links`	Links can be used to associate this span with other spans in different traces. In the output, it’s an empty array, indicating no specific links for this span.

Span Capturing an Error

Here’s how a span looks when the effect encounters an error:

Example (Span for an Effect that Fails)

1
import { Effect } from "effect"
2
import { NodeSdk } from "@effect/opentelemetry"
3
import { ConsoleSpanExporter, BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"
4

5
const program = Effect.fail("Oh no!").pipe(Effect.delay("100 millis"), Effect.withSpan("myspan"))
6

7
const NodeSdkLive = NodeSdk.layer(() => ({
8
  resource: { serviceName: "example" },
9
  spanProcessor: new BatchSpanProcessor(new ConsoleSpanExporter()),
10
}))
11

12
Effect.runPromiseExit(program.pipe(Effect.provide(NodeSdkLive))).then(console.log)
13
/*
14
Example Output:
15
{
16
  resource: {
17
    attributes: {
18
      'service.name': 'example',
19
      'telemetry.sdk.language': 'nodejs',
20
      'telemetry.sdk.name': '@effect/opentelemetry',
21
      'telemetry.sdk.version': '1.28.0'
22
    }
23
  },
24
  instrumentationScope: { name: 'example', version: undefined, schemaUrl: undefined },
25
  traceId: 'eee9619866179f209b7aae277283e71f',
26
  parentId: undefined,
27
  traceState: undefined,
28
  name: 'myspan',
29
  id: '3a5725c91884c9e1',
30
  kind: 0,
31
  timestamp: 1733220830575626,
32
  duration: 106578.042,
33
  attributes: {
34
    'code.stacktrace': 'at <anonymous> (/Users/giuliocanti/Documents/GitHub/website/content/dev/index.ts:10:10)'
35
  },
36
  status: { code: 2, message: 'Oh no!' },
37
  events: [
38
    {
39
      name: 'exception',
40
      attributes: {
41
        'exception.type': 'Error',
42
        'exception.message': 'Oh no!',
43
        'exception.stacktrace': 'Error: Oh no!'
44
      },
45
      time: [ 1733220830, 682204083 ],
46
      droppedAttributesCount: 0
47
    }
48
  ],
49
  links: []
50
}
51
{
52
  _id: 'Exit',
53
  _tag: 'Failure',
54
  cause: { _id: 'Cause', _tag: 'Fail', failure: 'Oh no!' }
55
}
56
*/

In this example, the span’s status code is 2, indicating an error. The message in the status provides more details about the failure.

Adding Annotations

You can provide extra information to a span by utilizing the Effect.annotateCurrentSpan function. This function allows you to attach key-value pairs, offering more context about the execution of the span.

Example (Annotating a Span)

1
import { Effect } from "effect"
2
import { NodeSdk } from "@effect/opentelemetry"
3
import { ConsoleSpanExporter, BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"
4

5
const program = Effect.void.pipe(
6
  Effect.delay("100 millis"),
7
  // Annotate the span with a key-value pair
8
  Effect.tap(() => Effect.annotateCurrentSpan("key", "value")),
9
  // Wrap the effect in a span named 'myspan'
10
  Effect.withSpan("myspan"),
11
)
12

13
// Set up tracing with the OpenTelemetry SDK
14
const NodeSdkLive = NodeSdk.layer(() => ({
15
  resource: { serviceName: "example" },
16
  spanProcessor: new BatchSpanProcessor(new ConsoleSpanExporter()),
17
}))
18

19
// Run the effect, providing the tracing layer
20
Effect.runPromise(program.pipe(Effect.provide(NodeSdkLive)))
21
/*
22
Example Output:
23
{
24
  resource: {
25
    attributes: {
26
      'service.name': 'example',
27
      'telemetry.sdk.language': 'nodejs',
28
      'telemetry.sdk.name': '@effect/opentelemetry',
29
      'telemetry.sdk.version': '1.28.0'
30
    }
31
  },
32
  instrumentationScope: { name: 'example', version: undefined, schemaUrl: undefined },
33
  traceId: 'c8120e01c0f1ea83ccc1d388e5cdebd3',
34
  parentId: undefined,
35
  traceState: undefined,
36
  name: 'myspan',
37
  id: '81c430ba4979f1db',
38
  kind: 0,
39
  timestamp: 1733220874356084,
40
  duration: 102821.417,
41
  attributes: { key: 'value' },
42
  status: { code: 1 },
43
  events: [],
44
  links: []
45
}
46
*/

Logs as events

In the context of tracing, logs are converted into “Span Events.” These events offer structured insights into your application’s activities and provide a timeline of when specific operations occurred.

1
import { Effect } from "effect"
2
import { NodeSdk } from "@effect/opentelemetry"
3
import { ConsoleSpanExporter, BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"
4

5
// Define a program that logs a message and delays for 100 milliseconds
6
const program = Effect.log("Hello").pipe(Effect.delay("100 millis"), Effect.withSpan("myspan"))
7

8
// Set up tracing with the OpenTelemetry SDK
9
const NodeSdkLive = NodeSdk.layer(() => ({
10
  resource: { serviceName: "example" },
11
  spanProcessor: new BatchSpanProcessor(new ConsoleSpanExporter()),
12
}))
13

14
// Run the effect, providing the tracing layer
15
Effect.runPromise(program.pipe(Effect.provide(NodeSdkLive)))
16
/*
17
Example Output:
18
{
19
  resource: {
20
    attributes: {
21
      'service.name': 'example',
22
      'telemetry.sdk.language': 'nodejs',
23
      'telemetry.sdk.name': '@effect/opentelemetry',
24
      'telemetry.sdk.version': '1.28.0'
25
    }
26
  },
27
  instrumentationScope: { name: 'example', version: undefined, schemaUrl: undefined },
28
  traceId: 'b0f4f012b5b13c0a040f7002a1d7b020',
29
  parentId: undefined,
30
  traceState: undefined,
31
  name: 'myspan',
32
  id: 'b9ba8472002715a8',
33
  kind: 0,
34
  timestamp: 1733220905504162.2,
35
  duration: 103790,
36
  attributes: {},
37
  status: { code: 1 },
38
  events: [
39
    {
40
      name: 'Hello',
41
      attributes: { 'effect.fiberId': '#0', 'effect.logLevel': 'INFO' }, // Log attributes
42
      time: [ 1733220905, 607761042 ], // Event timestamp
43
      droppedAttributesCount: 0
44
    }
45
  ],
46
  links: []
47
}
48
*/

Each span can include events, which capture specific moments during the execution of a span. In this example, a log message "Hello" is recorded as an event within the span. Key details of the event include:

Field	Description
`name`	The name of the event, which corresponds to the logged message (e.g., `'Hello'`).
`attributes`	Key-value pairs that provide additional context about the event, such as `fiberId` and log level.
`time`	The timestamp of when the event occurred, shown in a high-precision format.
`droppedAttributesCount`	Indicates how many attributes were discarded, if any. In this case, no attributes were dropped.

Nesting Spans

Spans can be nested to represent a hierarchy of operations. This allows you to track how different parts of your application relate to one another during execution. The following example demonstrates how to create and manage nested spans.

Example (Nesting Spans in a Trace)

1
import { Effect } from "effect"
2
import { NodeSdk } from "@effect/opentelemetry"
3
import { ConsoleSpanExporter, BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"
4

5
const child = Effect.void.pipe(Effect.delay("100 millis"), Effect.withSpan("child"))
6

7
const parent = Effect.gen(function* () {
8
  yield* Effect.sleep("20 millis")
9
  yield* child
10
  yield* Effect.sleep("10 millis")
11
}).pipe(Effect.withSpan("parent"))
12

13
// Set up tracing with the OpenTelemetry SDK
14
const NodeSdkLive = NodeSdk.layer(() => ({
15
  resource: { serviceName: "example" },
16
  spanProcessor: new BatchSpanProcessor(new ConsoleSpanExporter()),
17
}))
18

19
// Run the effect, providing the tracing layer
20
Effect.runPromise(parent.pipe(Effect.provide(NodeSdkLive)))
21
/*
22
Example Output:
23
{
24
  resource: {
25
    attributes: {
26
      'service.name': 'example',
27
      'telemetry.sdk.language': 'nodejs',
28
      'telemetry.sdk.name': '@effect/opentelemetry',
29
      'telemetry.sdk.version': '1.28.0'
30
    }
31
  },
32
  instrumentationScope: { name: 'example', version: undefined, schemaUrl: undefined },
33
  traceId: 'a9cd69ad70698a0c7b7b774597c77d39',
34
  parentId: 'a09e5c3fdfdbbc1d', // This indicates the span is a child of 'parent'
35
  traceState: undefined,
36
  name: 'child',
37
  id: '210d2f9b648389a4', // Unique ID for the child span
38
  kind: 0,
39
  timestamp: 1733220970590126.2,
40
  duration: 101579.875,
41
  attributes: {},
42
  status: { code: 1 },
43
  events: [],
44
  links: []
45
}
46
{
47
  resource: {
48
    attributes: {
49
      'service.name': 'example',
50
      'telemetry.sdk.language': 'nodejs',
51
      'telemetry.sdk.name': '@effect/opentelemetry',
52
      'telemetry.sdk.version': '1.28.0'
53
    }
54
  },
55
  instrumentationScope: { name: 'example', version: undefined, schemaUrl: undefined },
56
  traceId: 'a9cd69ad70698a0c7b7b774597c77d39',
57
  parentId: undefined, // Indicates this is the root span
58
  traceState: undefined,
59
  name: 'parent',
60
  id: 'a09e5c3fdfdbbc1d', // Unique ID for the parent span
61
  kind: 0,
62
  timestamp: 1733220970569015.2,
63
  duration: 132612.208,
64
  attributes: {},
65
  status: { code: 1 },
66
  events: [],
67
  links: []
68
}
69
*/

The parent-child relationship is evident in the span output, where the parentId of the child span matches the id of the parent span. This structure helps track how operations are related within a single trace.

Tutorial: Visualizing Traces

In this tutorial, we will guide you through visualizing traces generated by a sample Effect application. The sample application has also been configured to export traces and/or metrics via HTTP using OTLP format.

To visualize the traces being exported by our application, we will use a Docker image that contains a preconfigured OpenTelemetry backend based on the OpenTelemetry Collector, Prometheus, Loki, Tempo, and Grafana.

Tools Explained

Let’s understand the tools we’ll be using in simple terms:

Docker: Docker allows us to run applications in containers. Think of a container as a lightweight and isolated environment where your application can run consistently, regardless of the host system. It’s a bit like a virtual machine but more efficient.
Prometheus: Prometheus is a monitoring and alerting toolkit. It collects metrics and data about your applications and stores them for further analysis. This helps in identifying performance issues and understanding the behavior of your applications.
Loki: Loki is a log aggregation system inspired by Prometheus. It does not index the contents of the logs, but rather a set of labels for each log stream.
Grafana: Grafana is a visualization and analytics platform. It helps in creating beautiful and interactive dashboards to visualize your application’s data. You can use it to graphically represent metrics collected by Prometheus.
Tempo: Tempo is a distributed tracing system that allows you to trace the journey of a request as it flows through your application. It provides insights into how requests are processed and helps in debugging and optimizing your applications.

Getting Docker

To get Docker, follow these steps:

Visit the Docker website at https://www.docker.com/.
Download Docker Desktop for your operating system (Windows or macOS) and install it.
After installation, open Docker Desktop, and it will run in the background.

Simulating Traces

Start the OpenTelemetry Backend

Before we begin generating and exporting traces from our sample application, we will need to get our OpenTelemetry backend running in Docker.

This can be done using the following command:
Terminal window
```
docker run -p 3000:3000 -p 4317:4317 -p 4318:4318 --rm -it docker.io/grafana/otel-lgtm
```

Install Dependencies

We also need to install a few additional dependencies, as well as the latest version of effect:

# If not already installed
npm install effect
# Required to integrate Effect with OpenTelemetry
npm install @effect/opentelemetry
# Required to export traces over HTTP in OTLP format
npm install @opentelemetry/exporter-trace-otlp-http
# Required by all applications
npm install @opentelemetry/sdk-trace-base
# For NodeJS applications
npm install @opentelemetry/sdk-trace-node
# For browser applications
npm install @opentelemetry/sdk-trace-web
# If you also need to export metrics
npm install @opentelemetry/sdk-metrics

# If not already installed
pnpm add effect
# Required to integrate Effect with OpenTelemetry
pnpm add @effect/opentelemetry
# Required to export traces over HTTP in OTLP format
pnpm add @opentelemetry/exporter-trace-otlp-http
# Required by all applications
pnpm add @opentelemetry/sdk-trace-base
# For NodeJS applications
pnpm add @opentelemetry/sdk-trace-node
# For browser applications
pnpm add @opentelemetry/sdk-trace-web
# If you also need to export metrics
pnpm add @opentelemetry/sdk-metrics

# If not already installed
yarn add effect
# Required to integrate Effect with OpenTelemetry
yarn add @effect/opentelemetry
# Required to export traces over HTTP in OTLP format
yarn add @opentelemetry/exporter-trace-otlp-http
# Required by all applications
yarn add @opentelemetry/sdk-trace-base
# For NodeJS applications
yarn add @opentelemetry/sdk-trace-node
# For browser applications
yarn add @opentelemetry/sdk-trace-web
# If you also need to export metrics
yarn add @opentelemetry/sdk-metrics

# If not already installed
bun add effect
# Required to integrate Effect with OpenTelemetry
bun add @effect/opentelemetry
# Required to export traces over HTTP in OTLP format
bun add @opentelemetry/exporter-trace-otlp-http
# Required by all applications
bun add @opentelemetry/sdk-trace-base
# For NodeJS applications
bun add @opentelemetry/sdk-trace-node
# For browser applications
bun add @opentelemetry/sdk-trace-web
# If you also need to export metrics
bun add @opentelemetry/sdk-metrics

Simulate Traces

Now, let’s simulate traces using a sample Node.js application.

The following code simulates a set of tasks and generates traces for each task. It also sets up a Layer which will export traces from our application to our OpenTelemetry backend over HTTP in OTLP format.

1
import { Effect } from "effect"
2
import { NodeSdk } from "@effect/opentelemetry"
3
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"
4
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"
5

6
// Function to simulate a task with possible subtasks
7
const task = (name: string, delay: number, children: ReadonlyArray<Effect.Effect<void>> = []) =>
8
  Effect.gen(function* () {
9
    yield* Effect.log(name)
10
    yield* Effect.sleep(`${delay} millis`)
11
    for (const child of children) {
12
      yield* child
13
    }
14
    yield* Effect.sleep(`${delay} millis`)
15
  }).pipe(Effect.withSpan(name))
16

17
const poll = task("/poll", 1)
18

19
// Create a program with tasks and subtasks
20
const program = task("client", 2, [
21
  task("/api", 3, [
22
    task("/authN", 4, [task("/authZ", 5)]),
23
    task("/payment Gateway", 6, [task("DB", 7), task("Ext. Merchant", 8)]),
24
    task("/dispatch", 9, [
25
      task("/dispatch/search", 10),
26
      Effect.all([poll, poll, poll], { concurrency: "inherit" }),
27
      task("/pollDriver/{id}", 11),
28
    ]),
29
  ]),
30
])
31

32
const NodeSdkLive = NodeSdk.layer(() => ({
33
  resource: { serviceName: "example" },
34
  spanProcessor: new BatchSpanProcessor(new OTLPTraceExporter()),
35
}))
36

37
Effect.runPromise(
38
  program.pipe(Effect.provide(NodeSdkLive), Effect.catchAllCause(Effect.logError)),
39
)
40
/*
41
Output:
42
timestamp=... level=INFO fiber=#0 message=client
43
timestamp=... level=INFO fiber=#0 message=/api
44
timestamp=... level=INFO fiber=#0 message=/authN
45
timestamp=... level=INFO fiber=#0 message=/authZ
46
timestamp=... level=INFO fiber=#0 message="/payment Gateway"
47
timestamp=... level=INFO fiber=#0 message=DB
48
timestamp=... level=INFO fiber=#0 message="Ext. Merchant"
49
timestamp=... level=INFO fiber=#0 message=/dispatch
50
timestamp=... level=INFO fiber=#0 message=/dispatch/search
51
timestamp=... level=INFO fiber=#3 message=/poll
52
timestamp=... level=INFO fiber=#4 message=/poll
53
timestamp=... level=INFO fiber=#5 message=/poll
54
timestamp=... level=INFO fiber=#0 message=/pollDriver/{id}
55
*/

Visualize Traces

Open your web browser and go to http://localhost:3000/explore. You should see the Grafana Tempo TraceQL interface.

To get a list of all available traces, we can select the "Search" query type to get a list of all available traces.

Clicking the generated Trace ID will allow us to inspect the details of the trace.

Integrations

Sentry

To send span data directly to Sentry for analysis, replace the default span processor with Sentry’s implementation. This allows you to use Sentry as a backend for tracing and debugging.

Example (Configuring Sentry for Tracing)

1
import { NodeSdk } from "@effect/opentelemetry"
2
import { SentrySpanProcessor } from "@sentry/opentelemetry"
3

4
const NodeSdkLive = NodeSdk.layer(() => ({
5
  resource: { serviceName: "example" },
6
  spanProcessor: new SentrySpanProcessor(),
7
}))