# Troubleshooting > Troubleshooting OBI common issues and errors --- LLMS index: [llms.txt](/llms.txt) --- On this page, you can learn how to diagnose and resolve common OBI errors and issues. ## Troubleshooting tools OBI provides a variety of tools and configuration options to help diagnose and troubleshoot issues. ### Detailed logging You can increase the logging verbosity of OBI by setting the `log_level` configuration or the `OTEL_EBPF_LOG_LEVEL` environment variable to `debug`. This provides more detailed logs that may help in diagnosing issues. To enable logging from the BPF programs, set the `ebpf.bpf_debug` configuration or the `OTEL_EBPF_BPF_DEBUG` environment variable to `true`. **Use this only for debugging**, as it can generate a significant number of logs. ### Configuration logging By default, OBI merges its configuration from three different sources, from least to most priority: - Built-in default configuration - Configuration file, provided using the `--config` flag or `OTEL_EBPF_CONFIG_PATH` - Environment variables, usually starting with `OTEL_EBPF_` It is often helpful to view the final merged configuration. Using the `log_config` configuration value (or `OTEL_EBPF_LOG_CONFIG` environment variable), you can instruct OBI to log the final configuration at startup. `log_config` supports the following values: - `yaml` — logs the final configuration in YAML format; best for human readability since it matches the config file structure - `json` — logs the final configuration in JSON format; best for log shippers since it is a single structured line ### Internal metrics You can configure and use [OBI internal metrics](../metrics/#internal-metrics) to monitor performance and internal state. To turn on internal metrics, configure `internal_metrics.exporter` with one of the following values: - `none` (default): disables internal metrics - `prometheus`: exports internal metrics in Prometheus format via an HTTP server - `otlp`: exports internal metrics via an OTLP exporter ### Debug traces exporter To debug the raw trace spans generated by OBI, you can set the `otel_traces_exporter.protocol` configuration value or the `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` environment variable to `debug`. This logs the raw trace spans to the console in a human-readable format, matching the OTel Collector debug exporter with `verbosity: detailed`. Example spans to the console look like this: ```text Traces {"resource spans": 1, "spans": 1} ResourceSpans #0 Resource SchemaURL: Resource attributes: -> service.name: Str(flagd) -> telemetry.sdk.language: Str(go) -> telemetry.sdk.name: Str(opentelemetry) -> telemetry.distro.name: Str(opentelemetry-ebpf-instrumentation) -> telemetry.sdk.version: Str(main) -> host.name: Str(flagd-5cccb4c4f5-sfkcm) -> os.type: Str(linux) -> service.namespace: Str(opentelemetry-demo) -> k8s.owner.name: Str(flagd) -> k8s.kind: Str(Deployment) -> k8s.replicaset.name: Str(flagd-5cccb4c4f5) -> k8s.pod.name: Str(flagd-5cccb4c4f5-sfkcm) -> k8s.container.name: Str(flagd) -> k8s.deployment.name: Str(flagd) -> service.version: Str(2.0.2) -> k8s.namespace.name: Str(default) -> otel.library.name: Str(go.opentelemetry.io/obi) ScopeSpans #0 ScopeSpans SchemaURL: InstrumentationScope Span #0 Trace ID : 63a2723a58e0033170e58b1ff27ef03d Parent ID : ID : fab47609b60cc4e0 Name : /opentelemetry.proto.collector.metrics.v1.MetricsService/Export Kind : Client Start time : 2025-11-28 16:10:35.4241749 +0000 UTC End time : 2025-11-28 16:10:35.42555658 +0000 UTC Status code : Unset Status message : Attributes: -> rpc.method: Str(/opentelemetry.proto.collector.metrics.v1.MetricsService/Export) -> rpc.system: Str(grpc) -> rpc.grpc.status_code: Int(0) -> server.address: Str(otel-collector.default) -> peer.service: Str(otel-collector.default) -> server.port: Int(4317) ``` Starting with OBI v0.6.0, `telemetry.sdk.name` reflects the underlying SDK when available, and OBI identifies itself using `telemetry.distro.name`. ### Performance profiler (pprof) OBI can expose a `pprof` port to allow performance profiling. To enable it, set the `profile_port` configuration value or the `OTEL_EBPF_PROFILE_PORT` environment variable to the desired port. This is an advanced use case and typically not required. ## Common OBI issues This section covers how to resolve common OBI issues. ### ClickHouse instances crash when OBI is running If you're running [Clickhouse](https://github.com/ClickHouse/ClickHouse) on the same node with OBI, you might see ClickHouse crashing with logs such as: ```text Application: Code: 246. DB::Exception: Calculated checksum of the executable (...) does not correspond to the reference checksum ... ``` The issue is likely caused by OBI attaching eBPF uprobes to the ClickHouse binary. [A relevant GitHub](https://github.com/ClickHouse/ClickHouse/issues/83637) issue explains this behavior: > When attaching a uprobe, the kernel will modify the target process memory to > insert a trap instruction at the attachment address. This causes the > ClickHouse binary checksum validation to fail during startup. **Solution:** Start ClickHouse with the [skip_binary_checksum_checks](https://clickhouse.com/docs/operations/server-configuration-parameters/settings#skip_binary_checksum_checks) flag ### Missing telemetry data for Go applications or TLS requests If you are missing telemetry coming from Go applications or TLS requests (like HTTPS communication), it might be due to insufficient privileges for attaching uprobes. Due to some recent kernel security changes which were backported to many older kernel versions, uprobes now require `CAP_SYS_ADMIN` capability. OBI uses uprobes to instrument Golang applications and TLS requests, along with other runtime/language specific instrumentations. If your OBI deployment security configuration isn't using privileged operation (for example, `privileged:true` or Docker and Kubernetes) or it doesn't provide `CAP_SYS_ADMIN` as a security capability, you might not see some or all of your telemetry. To troubleshoot this issue, enable detailed OBI logging with `OTEL_EBPF_LOG_LEVEL=debug`. If you see all the uprobe injections failing with the error "setting uprobe (offset)..." then you are likely experiencing this issue. **Solutions:** You can either: - Run OBI as privileged. - Add `CAP_SYS_ADMIN` to the list of capabilities in your deployment security configuration. ### Client metrics or spans attributed to the wrong service When OBI runs with access to the host PID namespace (for example, `pid: host` in Docker Compose or `hostPID: true` in Kubernetes) and you select services by [open port](../configure/service-discovery/#open-ports), you might see unexpected outgoing (client) metrics or spans attributed to one of your services. A common symptom is Docker Engine API calls, such as `GET /v1.43/containers/json`, reported as client requests with a meaningless `server.port` and `telemetry.sdk.language=go`, even when the target service isn't written in Go: ```text http_client_request_body_size_bytes_sum{ http_request_method="GET", http_route="/v1.43/containers/json", server_address="docker", server_port="4", service_name="python-service", telemetry_sdk_language="go", ... } ``` **Cause:** When you publish a container port (for example, Docker's `-p 7773:7773` or a Compose `ports:` entry), a host-side forwarder process listens on that port in the host network namespace. Depending on your container runtime this is `docker-proxy` (Docker with the userland proxy enabled) or an equivalent agent. Because OBI can see host processes, and the forwarder listens on the same port you selected, OBI matches the forwarder against your `open_ports` criteria and instruments it under your service's identity. Both the forwarder (in the host network namespace) and your real service (in its container network namespace) genuinely listen on that port, so they can't be told apart by the port alone. The forwarder is typically a Go binary, which is why the spans carry `telemetry.sdk.language=go`. Any traffic the forwarder itself generates — such as the Docker Engine API calls it makes over the Docker socket — is then attributed to your service. **Solutions:** Exclude the forwarder from instrumentation by its executable path: ```yaml discovery: exclude_instrument: - exe_path: '{*/docker-proxy,*/scon-agent}' ``` Alternatively, select your services with a criterion more specific than the open port, so a host-side forwarder that happens to share the port isn't matched. For example, use the [executable path](../configure/service-discovery/#executable-path), [Kubernetes metadata](../configure/service-discovery/#k8s-namespace), or the [container name](../configure/service-discovery/#container-name) selector. ## Migration to v0.7.0: Network port guessing changes OBI v0.7.0 introduces a breaking change: **network port guessing is now disabled by default**. This change improves network metrics accuracy by not making assumptions about unknown initiators in network flows. ### What changed In v0.6.0 and earlier, OBI would attempt to guess which endpoint was the client and which was the server in network flows where the initiator couldn't be determined. This guessing was based on ordinal heuristics (typically assuming the lower port number was the server and the higher port number was the client). In v0.7.0, this guessing is disabled by default, which means: - `client.port` and `server.port` attributes may be empty for flows where OBI cannot determine the initiator - Network metrics will be more accurate but may lose information for unknown flows ### How to migrate If you depend on the old behavior and want `client.port` and `server.port` to be inferred even when the initiator is unknown, re-enable port guessing with ordinal heuristics: **YAML configuration:** ```yaml network: guess_ports: ordinal ``` **Environment variable:** ```sh OTEL_EBPF_NETWORK_GUESS_PORTS=ordinal ``` For more details, see the [network configuration documentation](../network/config/). ### Recommendation We recommend leaving port guessing disabled unless you have a specific use case that requires it. The default behavior provides cleaner, more accurate network metrics that are less prone to misclassification.