# Semantic conventions for RPC spans LLMS index: [llms.txt](/llms.txt) --- **Status**: [Release Candidate][DocumentStatus], unless otherwise specified This document defines how to describe remote procedure calls (also called "remote method invocations" / "RMI") with spans. > [!IMPORTANT] > Existing RPC instrumentations that are using > [v1.37.0 of this document](https://github.com/open-telemetry/semantic-conventions/blob/v1.37.0/docs/rpc/rpc-spans.md) > (or prior): > > * SHOULD NOT change the version of the RPC conventions that they emit by > default in their existing major version. Conventions include (but are not > limited to) attributes, metric and span names, and unit of measure. > * SHOULD introduce an environment variable `OTEL_SEMCONV_STABILITY_OPT_IN` > in their existing major version as a comma-separated list of category-specific values > (e.g., http, databases, rpc). The list of values includes: > * `rpc` - emit the stable RPC conventions, and stop emitting > the experimental RPC conventions that the instrumentation emitted > previously. > * `rpc/dup` - emit both the experimental and stable RPC conventions, > allowing for a phased rollout of the stable semantic conventions. > * The default behavior (in the absence of one of these values) is to continue > emitting whatever version of the old experimental RPC conventions > the instrumentation was emitting previously. > * Note: `rpc/dup` has higher precedence than `rpc` in case both values are present > * SHOULD maintain (security patching at a minimum) their existing major version > for at least six months after it starts emitting both sets of conventions. > * MAY drop the environment variable in their next major version and emit only > the stable RPC conventions. ## Name RPC spans MUST follow the overall [guidelines for span names](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.54.0/specification/trace/api.md#span). The *span name* SHOULD be `{rpc.method}` if it is available and not set to `_OTHER`. If `rpc.method` is unavailable or set to `_OTHER`, the span name SHOULD be `{rpc.system.name}`. Semantic conventions for individual RPC systems MAY specify different span name format. ## RPC client span **Status:** ![Release Candidate](https://img.shields.io/badge/-rc-mediumorchid) This span represents an outgoing Remote Procedure Call (RPC). RPC client spans SHOULD cover the entire client-side lifecycle of an RPC, starting when the RPC is initiated and ending when the response is received or the RPC is terminated due to an error or cancellation. For streaming RPCs, the span covers the full lifetime of the request and/or response streams until they are closed or terminated. If a transient issue happened and was retried within this RPC, the corresponding span SHOULD cover the duration of the logical call with all retries. **Span name:** refer to the [Span Name](/docs/specs/semconv/rpc/rpc-spans.md#name) section. **Span kind** MUST be `CLIENT`. **Span status**: refer to the [Recording Errors](/docs/specs/semconv/general/recording-errors.md) document for details on how to record span status. **Attributes:** | Key | Stability | [Requirement Level](/docs/specs/semconv/general/attribute-requirement-level/) | Value Type | Description | Example Values | | --- | --- | --- | --- | --- | --- | | [`rpc.system.name`](/docs/specs/semconv/registry/attributes/rpc.md) | ![Release Candidate](https://img.shields.io/badge/-rc-mediumorchid) | `Required` | string | The Remote Procedure Call (RPC) system. [1] | `grpc`; `dubbo`; `connectrpc` | | [`error.type`](/docs/specs/semconv/registry/attributes/error.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Conditionally Required` If and only if the operation failed. | string | Describes a class of error the operation ended with. [2] | `DEADLINE_EXCEEDED`; `java.net.UnknownHostException`; `-32602` | | [`rpc.method`](/docs/specs/semconv/registry/attributes/rpc.md) | ![Release Candidate](https://img.shields.io/badge/-rc-mediumorchid) | `Conditionally Required` if available. | string | The fully-qualified logical name of the method from the RPC interface perspective. [3] | `com.example.ExampleService/exampleMethod`; `EchoService/Echo`; `_OTHER` | | [`rpc.method_original`](/docs/specs/semconv/registry/attributes/rpc.md) | ![Release Candidate](https://img.shields.io/badge/-rc-mediumorchid) | `Conditionally Required` If and only if it's different than `rpc.method`. | string | The original name of the method used by the client. | `com.myservice.EchoService/catchAll`; `com.myservice.EchoService/unknownMethod`; `InvalidMethod` | | [`rpc.response.status_code`](/docs/specs/semconv/registry/attributes/rpc.md) | ![Release Candidate](https://img.shields.io/badge/-rc-mediumorchid) | `Conditionally Required` if available. | string | Status code of the RPC returned by the RPC server or generated by the client [4] | `OK`; `DEADLINE_EXCEEDED`; `-32602` | | [`server.address`](/docs/specs/semconv/registry/attributes/server.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Conditionally Required` If available. | string | A string identifying a group of RPC server instances request is sent to. [5] | `example.com`; `10.1.2.80`; `/tmp/my.sock` | | [`server.port`](/docs/specs/semconv/registry/attributes/server.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Conditionally Required` if applicable and if `server.address` is set. | int | Server port number. [6] | `80`; `8080`; `443` | | [`network.peer.address`](/docs/specs/semconv/registry/attributes/network.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Recommended` | string | Peer address of the network connection - IP address or Unix domain socket name. [7] | `10.1.2.80`; `/tmp/my.sock` | | [`network.peer.port`](/docs/specs/semconv/registry/attributes/network.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Recommended` If `network.peer.address` is set. | int | Peer port number of the network connection. | `65123` | **[1] `rpc.system.name`:** The client and server RPC systems may differ for the same RPC interaction. For example, a client may use Apache Dubbo or Connect RPC to communicate with a server that uses gRPC since both protocols provide compatibility with gRPC. **[2] `error.type`:** If the RPC fails with an error before status code is returned, `error.type` SHOULD be set to the exception type (its fully-qualified class name, if applicable) or a component-specific, low cardinality error identifier. If a response status code is returned and status indicates an error, `error.type` SHOULD be set to that status code. Check system-specific conventions for the details on which values of `rpc.response.status_code` are considered errors. The `error.type` value SHOULD be predictable and SHOULD have low cardinality. Instrumentations SHOULD document the list of errors they report. If the request has completed successfully, instrumentations SHOULD NOT set `error.type`. **[3] `rpc.method`:** The method name MAY have unbounded cardinality in edge or error cases. Some RPC frameworks or libraries provide a fixed set of recognized methods for client stubs and server implementations. Instrumentations for such frameworks MUST set this attribute to the original method name only when the method is recognized by the framework or library. When the method is not recognized, for example, when the server receives a request for a method that is not predefined on the server, or when instrumentation is not able to reliably detect if the method is predefined, the attribute MUST be set to `_OTHER`. In such cases, tracing instrumentations MUST also set `rpc.method_original` attribute to the original method value. If the RPC instrumentation could end up converting valid RPC methods to `_OTHER`, then it SHOULD provide a way to configure the list of recognized RPC methods. The `rpc.method` can be different from the name of any implementing method/function. The `code.function.name` attribute may be used to record the fully-qualified method actually executing the call on the server side, or the RPC client stub method on the client side. **[4] `rpc.response.status_code`:** Usually it represents an error code, but may also represent partial success, warning, or differentiate between various types of successful outcomes. Semantic conventions for individual RPC frameworks SHOULD document what `rpc.response.status_code` means in the context of that system and which values are considered to represent errors. **[5] `server.address`:** May contain a DNS name, an endpoint and path in the service registry, local socket name or an IP address. Semantic conventions for individual RPC systems SHOULD document how to populate this attribute. When address is an IP address, instrumentations SHOULD NOT do a reverse DNS lookup to obtain a DNS name and SHOULD set `server.address` to the provided IP address. **[6] `server.port`:** When observed from the client side, and when communicating through an intermediary, `server.port` SHOULD represent the server port behind any intermediaries, for example proxies, if it's available. **[7] `network.peer.address`:** If a RPC involved multiple network calls (for example retries), the last contacted address SHOULD be used. The following attributes can be important for making sampling decisions and SHOULD be provided **at span creation time** (if provided at all): * [`rpc.method`](/docs/specs/semconv/registry/attributes/rpc.md) * [`rpc.system.name`](/docs/specs/semconv/registry/attributes/rpc.md) * [`server.address`](/docs/specs/semconv/registry/attributes/server.md) * [`server.port`](/docs/specs/semconv/registry/attributes/server.md) --- `error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. | Value | Description | Stability | | --- | --- | --- | | `_OTHER` | A fallback error value to be used when the instrumentation doesn't define a custom value. | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | --- `rpc.system.name` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. | Value | Description | Stability | | --- | --- | --- | | `connectrpc` | [Connect RPC](https://connectrpc.com/) | ![Development](https://img.shields.io/badge/-development-blue) | | `dubbo` | [Apache Dubbo](https://dubbo.apache.org/) | ![Release Candidate](https://img.shields.io/badge/-rc-mediumorchid) | | `grpc` | [gRPC](https://grpc.io/) | ![Release Candidate](https://img.shields.io/badge/-rc-mediumorchid) | | `jsonrpc` | [JSON-RPC](https://www.jsonrpc.org/) | ![Development](https://img.shields.io/badge/-development-blue) | ## RPC server span **Status:** ![Release Candidate](https://img.shields.io/badge/-rc-mediumorchid) This span represents an incoming Remote Procedure Call (RPC). RPC server spans SHOULD cover the entire server-side lifecycle of an RPC, starting when the request is received and ending when the response is sent or the RPC is terminated due to an error or cancellation. For streaming RPCs, the span SHOULD cover the full lifetime of the request and/or response streams until they are closed or terminated. **Span name:** refer to the [Span Name](/docs/specs/semconv/rpc/rpc-spans.md#name) section. **Span kind** MUST be `SERVER`. **Span status**: refer to the [Recording Errors](/docs/specs/semconv/general/recording-errors.md) document for details on how to record span status. **Attributes:** | Key | Stability | [Requirement Level](/docs/specs/semconv/general/attribute-requirement-level/) | Value Type | Description | Example Values | | --- | --- | --- | --- | --- | --- | | [`rpc.system.name`](/docs/specs/semconv/registry/attributes/rpc.md) | ![Release Candidate](https://img.shields.io/badge/-rc-mediumorchid) | `Required` | string | The Remote Procedure Call (RPC) system. [1] | `grpc`; `dubbo`; `connectrpc` | | [`error.type`](/docs/specs/semconv/registry/attributes/error.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Conditionally Required` If and only if the operation failed. | string | Describes a class of error the operation ended with. [2] | `DEADLINE_EXCEEDED`; `java.net.UnknownHostException`; `-32602` | | [`rpc.method`](/docs/specs/semconv/registry/attributes/rpc.md) | ![Release Candidate](https://img.shields.io/badge/-rc-mediumorchid) | `Conditionally Required` if available. | string | The fully-qualified logical name of the method from the RPC interface perspective. [3] | `com.example.ExampleService/exampleMethod`; `EchoService/Echo`; `_OTHER` | | [`rpc.method_original`](/docs/specs/semconv/registry/attributes/rpc.md) | ![Release Candidate](https://img.shields.io/badge/-rc-mediumorchid) | `Conditionally Required` If and only if it's different than `rpc.method`. | string | The original name of the method used by the client. | `com.myservice.EchoService/catchAll`; `com.myservice.EchoService/unknownMethod`; `InvalidMethod` | | [`rpc.response.status_code`](/docs/specs/semconv/registry/attributes/rpc.md) | ![Release Candidate](https://img.shields.io/badge/-rc-mediumorchid) | `Conditionally Required` if available. | string | Status code of the RPC returned by the RPC server or generated by the client [4] | `OK`; `DEADLINE_EXCEEDED`; `-32602` | | [`server.address`](/docs/specs/semconv/registry/attributes/server.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Conditionally Required` If available. | string | A string identifying a group of RPC server instances request is sent to. [5] | `example.com`; `10.1.2.80`; `/tmp/my.sock` | | [`server.port`](/docs/specs/semconv/registry/attributes/server.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Conditionally Required` if applicable and if `server.address` is set. | int | Server port number. [6] | `80`; `8080`; `443` | | [`client.address`](/docs/specs/semconv/registry/attributes/client.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Recommended` | string | Client address - domain name if available without reverse DNS lookup; otherwise, IP address or Unix domain socket name. [7] | `client.example.com`; `10.1.2.80`; `/tmp/my.sock` | | [`client.port`](/docs/specs/semconv/registry/attributes/client.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Recommended` | int | Client port number. [8] | `65123` | | [`network.peer.address`](/docs/specs/semconv/registry/attributes/network.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Recommended` | string | Peer address of the network connection - IP address or Unix domain socket name. [9] | `10.1.2.80`; `/tmp/my.sock` | | [`network.peer.port`](/docs/specs/semconv/registry/attributes/network.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Recommended` If `network.peer.address` is set. | int | Peer port number of the network connection. | `65123` | **[1] `rpc.system.name`:** The client and server RPC systems may differ for the same RPC interaction. For example, a client may use Apache Dubbo or Connect RPC to communicate with a server that uses gRPC since both protocols provide compatibility with gRPC. **[2] `error.type`:** If the RPC fails with an error before status code is returned, `error.type` SHOULD be set to the exception type (its fully-qualified class name, if applicable) or a component-specific, low cardinality error identifier. If a response status code is returned and status indicates an error, `error.type` SHOULD be set to that status code. Check system-specific conventions for the details on which values of `rpc.response.status_code` are considered errors. The `error.type` value SHOULD be predictable and SHOULD have low cardinality. Instrumentations SHOULD document the list of errors they report. If the request has completed successfully, instrumentations SHOULD NOT set `error.type`. **[3] `rpc.method`:** The method name MAY have unbounded cardinality in edge or error cases. Some RPC frameworks or libraries provide a fixed set of recognized methods for client stubs and server implementations. Instrumentations for such frameworks MUST set this attribute to the original method name only when the method is recognized by the framework or library. When the method is not recognized, for example, when the server receives a request for a method that is not predefined on the server, or when instrumentation is not able to reliably detect if the method is predefined, the attribute MUST be set to `_OTHER`. In such cases, tracing instrumentations MUST also set `rpc.method_original` attribute to the original method value. If the RPC instrumentation could end up converting valid RPC methods to `_OTHER`, then it SHOULD provide a way to configure the list of recognized RPC methods. The `rpc.method` can be different from the name of any implementing method/function. The `code.function.name` attribute may be used to record the fully-qualified method actually executing the call on the server side, or the RPC client stub method on the client side. **[4] `rpc.response.status_code`:** Usually it represents an error code, but may also represent partial success, warning, or differentiate between various types of successful outcomes. Semantic conventions for individual RPC frameworks SHOULD document what `rpc.response.status_code` means in the context of that system and which values are considered to represent errors. **[5] `server.address`:** May contain a DNS name, an endpoint and path in the service registry, local socket name or an IP address. Semantic conventions for individual RPC systems SHOULD document how to populate this attribute. When address is an IP address, instrumentations SHOULD NOT do a reverse DNS lookup to obtain a DNS name and SHOULD set `server.address` to the provided IP address. **[6] `server.port`:** When observed from the client side, and when communicating through an intermediary, `server.port` SHOULD represent the server port behind any intermediaries, for example proxies, if it's available. **[7] `client.address`:** When observed from the server side, and when communicating through an intermediary, `client.address` SHOULD represent the client address behind any intermediaries, for example proxies, if it's available. **[8] `client.port`:** When observed from the server side, and when communicating through an intermediary, `client.port` SHOULD represent the client port behind any intermediaries, for example proxies, if it's available. **[9] `network.peer.address`:** If a RPC involved multiple network calls (for example retries), the last contacted address SHOULD be used. The following attributes can be important for making sampling decisions and SHOULD be provided **at span creation time** (if provided at all): * [`rpc.method`](/docs/specs/semconv/registry/attributes/rpc.md) * [`rpc.system.name`](/docs/specs/semconv/registry/attributes/rpc.md) * [`server.address`](/docs/specs/semconv/registry/attributes/server.md) * [`server.port`](/docs/specs/semconv/registry/attributes/server.md) --- `error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. | Value | Description | Stability | | --- | --- | --- | | `_OTHER` | A fallback error value to be used when the instrumentation doesn't define a custom value. | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | --- `rpc.system.name` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. | Value | Description | Stability | | --- | --- | --- | | `connectrpc` | [Connect RPC](https://connectrpc.com/) | ![Development](https://img.shields.io/badge/-development-blue) | | `dubbo` | [Apache Dubbo](https://dubbo.apache.org/) | ![Release Candidate](https://img.shields.io/badge/-rc-mediumorchid) | | `grpc` | [gRPC](https://grpc.io/) | ![Release Candidate](https://img.shields.io/badge/-rc-mediumorchid) | | `jsonrpc` | [JSON-RPC](https://www.jsonrpc.org/) | ![Development](https://img.shields.io/badge/-development-blue) | [DocumentStatus]: /docs/specs/otel/document-status