BTrace Troubleshooting Guide

This guide helps you diagnose and resolve common BTrace issues.

Note: Examples use btrace.jar -- the single masked JAR (BTrace 2.2+). If using a legacy multi-JAR distribution, replace btrace.jar with btrace-agent.jar (and add -Xbootclasspath/a:btrace-boot.jar where needed).

JVM Attachment Issues
Script Compilation Errors
Script Verification Errors
No Output from Scripts
Class/Method Not Found
Performance Issues
Agent Loading Problems
Compatibility Issues
Debugging BTrace Scripts
Known Limitations

JVM Attachment Issues

Unable to attach to target VM

Error message:

com.sun.tools.attach.AttachNotSupportedException: Unable to attach to target VM

Causes and solutions:

Permission mismatch
- Cause: BTrace and target JVM run as different users
- Solution: Run BTrace as the same user as the target JVM
```
# Check process owner
ps aux | grep <PID>

# Run as correct user
sudo -u <username> btrace <PID> script.java
```
Attach mechanism disabled or restricted

For JDK 8-20:
- Cause: Target JVM started with -XX:+DisableAttachMechanism
- Solution: Remove this flag and restart the application
For JDK 21+ (JEP 451):
- Cause: Dynamic agent loading triggers warnings (still works but shows warning)
- Warning message: WARNING: A Java agent has been loaded dynamically
- Solution: Add -XX:+EnableDynamicAgentLoading to target JVM startup to suppress warnings:
```
java -XX:+EnableDynamicAgentLoading -jar your-application.jar
```
- Note: In a future JDK release, dynamic agent loading will be disabled by default. The -XX:+EnableDynamicAgentLoading flag will be required to use BTrace's attach mode.
- Alternative: Use BTrace in agent mode (attach at startup) instead of dynamic attach:
```
java -javaagent:/path/to/btrace.jar=script=YourScript.class -jar your-application.jar
```
JRE instead of JDK
- Cause: JRE doesn't include attach API tools
- Solution: Install JDK and ensure JAVA_HOME points to it
```
# Verify JDK installation
which javac
echo $JAVA_HOME
```
Process in container/namespace
- Cause: Target process in Docker/different namespace
- Solution: Run BTrace in same container/namespace
```
# For Docker
docker exec -it <container> btrace <PID> script.java
```
SELinux/AppArmor restrictions
- Cause: Security policies prevent ptrace
- Solution: Adjust security policies or disable temporarily
```
# Check SELinux
getenforce

# Temporarily disable (not recommended for production)
sudo setenforce 0
```

Connection refused

Error message:

java.net.ConnectException: Connection refused

Solutions:

Check if BTrace client port (default 2020) is available
Specify different port: btrace -p 3030 <PID> script.java
Check firewall rules

Script Compilation Errors

Compilation failed

Error message:

error: cannot find symbol

Common causes:

Missing imports

// Wrong
@BTrace
public class MyTrace {
    @OnMethod(...)
    public static void handler() {
        println("test");  // ERROR: cannot find symbol
    }
}

// Correct
import io.btrace.core.annotations.*;
import static io.btrace.core.BTraceUtils.*;

@BTrace
public class MyTrace {
    @OnMethod(...)
    public static void handler() {
        println("test");  // OK
    }
}

Wrong package for annotations

// Old (pre-2.0)
import com.sun.btrace.annotations.*;

// New (2.0+)
import io.btrace.core.annotations.*;

Classpath issues
- Solution: Add required JARs to classpath
```
btrace -cp /path/to/lib.jar <PID> script.java
```

Script Verification Errors

BTrace script verification failed

Error message:

BTrace : verification failed

BTrace enforces safety restrictions. Common violations:

1. Creating new threads

Wrong:

@OnMethod(...)
public static void handler() {
    new Thread(() -> {}).start();  // ERROR: Cannot create threads
}

Right:

@OnMethod(...)
public static void handler() {
    // Use @OnTimer instead
}

@OnTimer(1000)
public static void periodic() {
    // Executes every second
}

2. Using synchronization

Wrong:

private static Object lock = new Object();

@OnMethod(...)
public static void handler() {
    synchronized (lock) {  // ERROR: Cannot use synchronization
        // ...
    }
}

Right:

import static io.btrace.core.BTraceUtils.Atomic.*;

private static AtomicInteger counter = newAtomicInteger(0);

@OnMethod(...)
public static void handler() {
    incrementAndGet(counter);  // Thread-safe without synchronization
}

3. Loops

Wrong:

@OnMethod(...)
public static void handler() {
    // ERROR: All loops forbidden in safe mode
    while (condition) { }
    for (int i = 0; i < 100; i++) { }
    do { } while (condition);
    for (String s : collection) { }
}

Right:

@OnMethod(...)
public static void handler() {
    // Use BTraceUtils methods instead
    // For iteration, use unsafe mode (-u flag) if absolutely necessary
}

Note: ALL loop constructs (for, while, do-while, enhanced for) are forbidden in safe mode. Use -u (unsafe mode) only in controlled environments if loops are required.

4. File I/O operations

Wrong:

@OnMethod(...)
public static void handler() {
    FileWriter fw = new FileWriter("out.txt");  // ERROR: No file I/O
}

Right:

@OnMethod(...)
public static void handler() {
    println("Output");  // Use BTraceUtils output functions
}

// Or run in unsafe mode
// btrace -u <PID> script.java

5. Calling external methods

Wrong:

@OnMethod(...)
public static void handler() {
    MyOtherClass.someMethod();  // ERROR: Cannot call external class methods
    System.out.println("test");  // ERROR: Cannot call System methods
}

Right:

@BTrace
public class MyTrace {
    @OnMethod(...)
    public static void handler() {
        // Call helper methods within same BTrace class - OK
        myCustomMethod();

        // Use BTraceUtils methods
        println(str(timeMillis()));
    }

    private static void myCustomMethod() {
        // Helper methods in same class are allowed
        println("Helper method called");
    }
}

Note: You CAN call private static methods within the same BTrace class. You CANNOT call methods from external classes (except BTraceUtils).

6. Catching exceptions

Wrong:

@OnMethod(...)
public static void handler() {
    try {
        // ...
    } catch (Exception e) {  // ERROR: Cannot catch exceptions
        // ...
    }
}

Right:

// BTrace handles exceptions automatically
@OnMethod(...)
public static void handler() {
    // Any exception here is caught and logged by BTrace
}

// Or use @OnError to track exceptions
@OnMethod(..., location = @Location(Kind.ERROR))
public static void onError(Throwable t) {
    println("Exception: " + str(t));
}

No Output from Scripts

Script attaches but produces no output

Diagnostic checklist:

Verify method is being called

// Add entry point logging
@OnMethod(clazz = "com.example.MyClass", method = "<init>")
public static void onInit() {
    println("Constructor called - script is working!");
}

Check class and method names

# Class names are case-sensitive and must be fully qualified

# Wrong
@OnMethod(clazz = "MyClass", method = "process")

# Right
@OnMethod(clazz = "com.example.pkg.MyClass", method = "process")

Test with wildcard patterns

// Too specific - might not match
@OnMethod(clazz = "com.example.MyClass", method = "processData")

// Broader - helps identify the issue
@OnMethod(clazz = "/com\\.example\\..*/", method = "/.*/")
public static void catchAll() {
    println("Matched something!");
}

Verify regex escaping

// Wrong - dots match any character
@OnMethod(clazz = "/com.example..*/")

// Right - dots are escaped
@OnMethod(clazz = "/com\\.example\\..*/")

Check for inner classes

// Inner classes use $ separator
@OnMethod(clazz = "com.example.Outer$Inner", method = "method")

Enable verbose output
```
btrace -v <PID> script.java
```

Verify instrumention occurred

# Dump instrumented classes
btrace -d /tmp/btrace-dump <PID> script.java

# Check if classes were modified
ls -la /tmp/btrace-dump/

Class/Method Not Found

No methods matched

Error message:

No methods matched for probe: ...

Solutions:

Use jcmd to find exact class names

# List loaded classes
jcmd <PID> VM.classes | grep -i myclass

# Find specific methods
jcmd <PID> VM.class_hierarchy | grep -A 5 MyClass

Verify BTrace is retransforming loaded classes
- BTrace scans ALL already-loaded classes when attaching
- It retransforms matching classes via Instrumentation.retransformClasses()
- It also listens for newly loaded classes
- If classes aren't being instrumented, check if they are modifiable:
```
# Some classes cannot be retransformed (e.g., native methods, JVM internals)
# Use -v flag to see which classes are being instrumented
btrace -v <PID> script.java
```

Check class loader hierarchy

// Match classes loaded by any classloader
@OnMethod(clazz = "+com.example.MyClass", method = "method")
//         ^ plus sign matches subclasses and all classloaders

Verify method signature for overloaded methods

// Multiple methods with same name - need signature
@OnMethod(clazz = "com.example.MyClass",
         method = "process",
         type = "void (java.lang.String, int)")

Performance Issues

BTrace causes significant slowdown

Symptoms:

Application becomes unresponsive
High CPU usage
Increased latency

Solutions:

Use sampling

@Sampled(kind = Sampled.Sampler.Adaptive)
@OnMethod(...)
public static void handler() {
    // Only executes on sampled invocations
}

Add level filtering

@OnMethod(clazz = "...", enableAt = @Level(">=2"))
public static void heavyHandler() {
    // Only active when level >= 2
}

// Control at runtime:
// Press Ctrl-C in btrace console, type: level 0

Avoid tracing high-frequency methods

// BAD - called millions of times per second
@OnMethod(clazz = "java.lang.String", method = "charAt")

// GOOD - application-specific methods
@OnMethod(clazz = "com.example.Service", method = "handleRequest")

Minimize instrumentation scope

// Too broad - matches thousands of methods
@OnMethod(clazz = "/java\\..*/", method = "/.*/")

// Focused - matches specific target
@OnMethod(clazz = "com.example.Service", method = "slowMethod")

Reduce output volume

// Bad - prints every invocation
@OnMethod(...)
public static void handler() {
    println("Called");
}

// Better - periodic summary
private static AtomicInteger count = newAtomicInteger(0);

@OnMethod(...)
public static void handler() {
    incrementAndGet(count);
}

@OnTimer(5000)
public static void report() {
    println("Calls in last 5s: " + str(get(count)));
    set(count, 0);
}

Use aggregations instead of individual logs

private static Aggregation duration =
    Aggregations.newAggregation(AggregationFunction.QUANTIZE);

@OnMethod(..., location = @Location(Kind.RETURN))
public static void onReturn(@Duration long d) {
    Aggregations.addToAggregation(duration, d / 1000000);
}

@OnTimer(10000)
public static void printStats() {
    Aggregations.printAggregation("Durations", duration);
    Aggregations.clearAggregation(duration);
}

Agent Loading Problems

Agent JAR not found

Error message:

Error opening zip file or JAR manifest missing : /path/to/btrace.jar

Solutions:

# Verify JAR exists
ls -la /path/to/btrace.jar

# Use absolute path
java -javaagent:/absolute/path/to/btrace.jar=script=Script.class MyApp

# Or set BTRACE_HOME
export BTRACE_HOME=/path/to/btrace
java -javaagent:$BTRACE_HOME/libs/btrace.jar=script=Script.class MyApp

Script class not found in agent mode

Error message:

ClassNotFoundException: MyTrace

Solutions:

# Specify absolute path to script
java -javaagent:btrace.jar=script=/absolute/path/MyTrace.class MyApp

# Or add to classpath
java -javaagent:btrace.jar=script=MyTrace.class \
     -cp /path/to/scripts:app.jar MyApp

Compatibility Issues

JDK 25+ Thread Method Exclusions

BTrace automatically excludes certain java.lang.Thread methods from instrumentation on JDK 25+ to prevent infinite recursion and interference with JVM internals:

ThreadLocal accessors — threadLocals(), setThreadLocals(), inheritableThreadLocals(), setInheritableThreadLocals(), terminatingThreadLocals(), setTerminatingThreadLocals(): JDK 25 changed direct field access to accessor methods. Instrumenting these causes infinite recursion when BTrace runtime uses ThreadLocal.
Interrupt methods — getAndClearInterrupt(), setInterrupt(), clearInterrupt(): Used internally by Thread.interrupted(), which BTrace's own runtime calls.
Exception dispatch — dispatchUncaughtException(), getUncaughtExceptionHandler(): Instrumenting these interferes with thread exception handling.

These exclusions are enforced in ClassFilter.SENSITIVE_METHODS and cannot be overridden by user scripts. If you see "method not instrumented" warnings for these methods, this is expected behavior.

Java Version Mismatch

Error message:

UnsupportedClassVersionError

Solutions:

Ensure BTrace supports your Java version (BTrace supports Java 8+)
Compile scripts with target version matching JVM
Check JAVA_HOME points to correct version

Module System Issues (Java 9+)

Error message:

IllegalAccessError: class X cannot access class Y

Solutions:

# Add required --add-opens flags
btrace --add-opens java.base/jdk.internal.misc=ALL-UNNAMED <PID> script.java

# Or add to target JVM startup
java --add-opens java.base/java.lang=ALL-UNNAMED \
     --add-opens java.base/jdk.internal.misc=ALL-UNNAMED \
     MyApp

Old Script Syntax

Error: Scripts written for BTrace 1.x fail with 2.x

Solution: Update imports:

// Old (1.x)
import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;

// New (2.x)
import io.btrace.core.annotations.*;
import static io.btrace.core.BTraceUtils.*;

Debugging BTrace Scripts

Enable verbose logging

btrace -v <PID> script.java

Dump instrumented classes

btrace -d /tmp/btrace-classes <PID> script.java

Then examine with:

javap -c /tmp/btrace-classes/com/example/MyClass.class

Print diagnostic info

@OnMethod(clazz = "com.example.MyClass", method = "myMethod")
public static void handler(@ProbeClassName String clazz,
                          @ProbeMethodName String method) {
    println("Class: " + clazz);
    println("Method: " + method);
    println("Thread: " + threadName());
    println("Stack:");
    jstack(5);  // Print 5 stack frames
}

Test incrementally

Start simple and add complexity:

// Step 1: Verify attachment
@OnTimer(1000)
public static void heartbeat() {
    println("BTrace alive");
}

// Step 2: Verify class matching
@OnMethod(clazz = "com.example.MyClass", method = "/.*/")
public static void anyMethod() {
    println("Matched");
}

// Step 3: Add specific logic
@OnMethod(clazz = "com.example.MyClass", method = "specificMethod")
public static void specific(String arg) {
    println("Arg: " + arg);
}

Kubernetes and Cloud Deployments

Finding Process ID in Pods

Problem: Need to find Java process ID inside a pod

Solutions:

# List Java processes in pod
kubectl exec <pod-name> -- jps

# Or use ps
kubectl exec <pod-name> -- ps aux | grep java

# For multi-container pods, specify container
kubectl exec <pod-name> -c <container-name> -- jps

Multi-Container Pod Considerations

Problem: Cannot attach to process in different container

Solution: Kubernetes pods with multiple containers have separate process namespaces by default.

# Enable process namespace sharing
apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  shareProcessNamespace: true  # Required for cross-container attachment
  containers:
  - name: app
    image: myapp:latest
  - name: btrace-sidecar
    image: btrace:latest

Note: With shareProcessNamespace: true, all containers can see each other's processes.

Port Conflicts in Pods

Problem: BTrace fails with port already in use

Error:

java.net.BindException: Address already in use

Cause: BTrace uses port 2020 by default; multiple BTrace instances or port conflicts

Solutions:

# Use different port
kubectl exec <pod> -- btrace -p 3030 <PID> script.java

# Check port usage
kubectl exec <pod> -- netstat -an | grep 2020

Security Policies and RBAC

Problem: Pod Security Policy blocks BTrace attachment

Error:

Operation not permitted

Common causes:

Pod Security Standards (PSP replacement in K8s 1.25+):

apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  securityContext:
    # May need to adjust based on policy
    runAsNonRoot: true
    capabilities:
      add:
      - SYS_PTRACE  # Required for process attachment

SELinux/AppArmor:

# Check if SELinux is blocking
kubectl exec <pod> -- getenforce

# Check AppArmor profile
kubectl exec <pod> -- cat /proc/self/attr/current

Solutions:

Add SYS_PTRACE capability if allowed by security policy
Run as same user as target JVM
Adjust security context constraints (OpenShift)

JDK Not Available in Container

Problem: BTrace requires JDK but only JRE in container

Error:

Error: tools.jar not found

Solutions:

Option 1: Use JDK base image

# Instead of JRE
FROM openjdk:11-jre
# Use JDK
FROM bellsoft/liberica-openjdk-debian:11-cds

Option 2: Install JDK in running pod (temporary)

kubectl exec <pod> -- apt-get update && apt-get install -y openjdk-11-jdk

Option 3: Sidecar with JDK

spec:
  shareProcessNamespace: true
  containers:
  - name: app
    image: myapp-jre:latest  # Can use JRE
  - name: btrace
    image: bellsoft/liberica-openjdk-debian:11-cds     # Sidecar has JDK

Service Mesh Compatibility

Istio/Linkerd:

BTrace works with service meshes but some considerations apply:

Traffic Interception:

Service mesh sidecars don't interfere with BTrace (different layer)
BTrace communication port (2020) not intercepted by mesh

mTLS:

BTrace uses local socket communication (not HTTP)
mTLS between services doesn't affect BTrace

Resource Limits:

# BTrace overhead may trigger limits
resources:
  requests:
    cpu: "100m"
    memory: "128Mi"
  limits:
    cpu: "500m"      # Increase if BTrace causes throttling
    memory: "512Mi"   # Increase if needed

ConfigMap for BTrace Scripts

Pattern: Store scripts in ConfigMap for easy distribution

apiVersion: v1
kind: ConfigMap
metadata:
  name: btrace-scripts
data:
  TraceMethod.java: |
    import io.btrace.core.annotations.*;
    import static io.btrace.core.BTraceUtils.*;

    @BTrace
    // Classname 'TraceMethod' must correspond to the file name 'TraceMethod.java'
    public class TraceMethod {
        @OnMethod(clazz = "com.example.Service", method = "process")
        public static void onProcess() {
            println("Method called");
        }
    }
---
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    image: myapp:latest
    volumeMounts:
    - name: scripts
      mountPath: /btrace-scripts
  volumes:
  - name: scripts
    configMap:
      name: btrace-scripts

Usage:

kubectl exec <pod> -- btrace <PID> /btrace-scripts/TraceMethod.java

Batch Tracing Across Pods

Pattern: Trace all pods in a deployment

#!/bin/bash
# trace-deployment.sh

DEPLOYMENT=$1
SCRIPT=$2

# Get all pod names for deployment
PODS=$(kubectl get pods -l app=$DEPLOYMENT -o jsonpath='{.items[*].metadata.name}')

for POD in $PODS; do
  echo "Attaching to $POD..."
  kubectl exec $POD -- btrace 1 $SCRIPT &
done

wait
echo "All traces complete"

Usage:

./trace-deployment.sh myapp /tmp/trace.java

Persistent Tracing with StatefulSets

Issue: BTrace doesn't persist across pod restarts

Solutions:

Store output in persistent volume:

kubectl exec <pod> -- btrace -o /data/trace.log <PID> script.java

Stream to external system:

// Use StatSD integration
@OnMethod(...)
public static void handler() {
    Statsd.send("metric.name", value);
}

Agent mode in pod spec:

spec:
  containers:
  - name: app
    image: myapp:latest
    env:
    - name: JAVA_TOOL_OPTIONS
      value: "-javaagent:/opt/btrace/lib/btrace.jar=script=/scripts/trace.class"

Cloud-Specific Issues

AWS EKS:

No specific issues; standard K8s patterns apply
Use same region for kubectl access

Google GKE:

GKE Autopilot: May restrict SYS_PTRACE capability
Use standard GKE for full BTrace support

Azure AKS:

No specific issues; standard K8s patterns apply
Ensure node has sufficient resources

OpenShift:

Security Context Constraints (SCC) may be stricter
May need to add SCC for SYS_PTRACE:

oc adm policy add-scc-to-user anyuid -z default

For more K8s deployment patterns, see Getting Started: K8s and FAQ: Microservices.

Known Limitations

Cannot instrument

Native methods:

Methods implemented in native code cannot be instrumented (JVM limitation)

Sensitive classes (to avoid infinite recursion):

Core classes BTrace depends on: java.lang.Object, String, ThreadLocal, Integer, Number
Package prefixes: java.lang.instrument.*, java.lang.invoke.*, java.lang.ref.*
Lock classes: java.util.concurrent.locks.LockSupport, AbstractQueuedSynchronizer, etc.
JDK internals: jdk.internal.*, sun.invoke.*
BTrace itself: io.btrace.*

Classes annotated with @BTrace (BTrace scripts themselves)

Note: Classes already transformed by other agents CAN be retransformed. Boot classpath and JDK classes CAN be instrumented (except the sensitive classes listed above).

Cannot trace

Very early VM initialization (use agent mode instead of attach mode)
Class loading before agent initialization
JVM shutdown sequence (partially)

Platform-specific issues

Windows:

Attach API may require admin privileges
Path separators: use \\ or /

macOS:

SIP (System Integrity Protection) may block attachment
Disable for debugging: csrutil disable (not recommended for production)

Linux:

Some distros have ptrace restrictions in /proc/sys/kernel/yama/ptrace_scope
Set to 0 to allow: echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

Getting Help

If issues persist:

Check BTrace version
```
btrace --version
```
Search existing issues
- GitHub Issues
Provide details when asking for help:
- BTrace version
- Java version (java -version)
- OS and version
- Complete error message
- Minimal reproducing script
- Target application details
Community channels:
- Slack: btrace.slack.com
- Gitter: gitter.im/btraceio/btrace
- GitHub Issues: github.com/btraceio/btrace/issues

FilesExpand file tree

Troubleshooting.md

Latest commit

History

Troubleshooting.md

File metadata and controls

BTrace Troubleshooting Guide

Table of Contents

JVM Attachment Issues

Unable to attach to target VM

Connection refused

Script Compilation Errors

Compilation failed

Script Verification Errors

BTrace script verification failed

1. Creating new threads

2. Using synchronization

3. Loops

4. File I/O operations

5. Calling external methods

6. Catching exceptions

No Output from Scripts

Script attaches but produces no output

Class/Method Not Found

No methods matched

Performance Issues

BTrace causes significant slowdown

Agent Loading Problems

Agent JAR not found

Script class not found in agent mode

Compatibility Issues

JDK 25+ Thread Method Exclusions

Java Version Mismatch

Module System Issues (Java 9+)

Old Script Syntax

Debugging BTrace Scripts

Enable verbose logging

Dump instrumented classes

Print diagnostic info

Test incrementally

Kubernetes and Cloud Deployments

Finding Process ID in Pods

Multi-Container Pod Considerations

Port Conflicts in Pods

Security Policies and RBAC

JDK Not Available in Container

Service Mesh Compatibility

ConfigMap for BTrace Scripts

Batch Tracing Across Pods

Persistent Tracing with StatefulSets

Cloud-Specific Issues

Known Limitations

Cannot instrument

Cannot trace

Platform-specific issues

Getting Help

See Also