Skip to content
This repository was archived by the owner on Oct 3, 2023. It is now read-only.

Commit 2a9d2a7

Browse files
songy23songy23
authored
Add gRPC metrics that allow real-time reporting. (#212)
* Add measures that are updated for each message For streaming RPCs, especially long-lived ones, we want to have stats for each message as it is sent/received, rather than waiting until the end of the RPC to record stats. * Have separate bytes and count measures. * Update the desciptions for *_bytes measure. * Make *_per_method views optional.
1 parent cb9a2a7 commit 2a9d2a7

1 file changed

Lines changed: 58 additions & 34 deletions

File tree

‎stats/gRPC.md‎

Lines changed: 58 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -29,27 +29,32 @@ Buckets for distributions in default views are as follows:
2929

3030
Client stats are recorded at the end of each outbound RPC.
3131

32-
| Measure name | Unit | Description |
33-
|------------------------------------------|------|-----------------------------------------------------------------------------------------------|
34-
| grpc.io/client/sent_messages_per_rpc | 1 | Number of messages sent in the RPC (always 1 for non-streaming RPCs). |
35-
| grpc.io/client/sent_bytes_per_rpc | By | Total bytes sent across all request messages per RPC. |
36-
| grpc.io/client/received_messages_per_rpc | 1 | Number of response messages received per RPC (always 1 for non-streaming RPCs). |
37-
| grpc.io/client/received_bytes_per_rpc | By | Total bytes received across all response messages per RPC. |
38-
| grpc.io/client/roundtrip_latency | ms | Time between first byte of request sent to last byte of response received, or terminal error. |
39-
| grpc.io/client/server_latency | ms | Propagated from the server and should have the same value as "grpc.io/server/latency". |
40-
| grpc.io/client/started_rpcs | 1 | The total number of client RPCs ever opened, including those that have not completed. |
32+
| Measure name | Unit | Description |
33+
|-------------------------------------------|------|-----------------------------------------------------------------------------------------------|
34+
| grpc.io/client/sent_messages_per_rpc | 1 | Number of messages sent in the RPC (always 1 for non-streaming RPCs). |
35+
| grpc.io/client/sent_bytes_per_rpc | By | Total bytes sent across all request messages per RPC. |
36+
| grpc.io/client/received_messages_per_rpc | 1 | Number of response messages received per RPC (always 1 for non-streaming RPCs). |
37+
| grpc.io/client/received_bytes_per_rpc | By | Total bytes received across all response messages per RPC. |
38+
| grpc.io/client/roundtrip_latency | ms | Time between first byte of request sent to last byte of response received, or terminal error. |
39+
| grpc.io/client/server_latency | ms | Propagated from the server and should have the same value as "grpc.io/server/latency". |
40+
| grpc.io/client/started_rpcs | 1 | The total number of client RPCs ever opened, including those that have not completed. |
41+
| grpc.io/client/sent_messages_per_method | 1 | Total messages sent per method. |
42+
| grpc.io/client/received_messages_per_method | 1 | Total messages received per method. |
43+
| grpc.io/client/sent_bytes_per_method | By | Total bytes sent per method, recorded real-time as bytes are sent. |
44+
| grpc.io/client/received_bytes_per_method | By | Total bytes received per method, recorded real-time as bytes are received. |
45+
4146

4247
### Tags
4348

44-
All client metrics should be tagged with the following.
49+
All client metrics should be tagged with the following.
4550

4651
| Tag name | Description |
4752
|--------------------|------------------------------------------------------------------------------------------------------------------|
4853
| grpc_client_method | Full gRPC method name, including package, service and method, e.g. google.bigtable.v2.Bigtable/CheckAndMutateRow |
4954
| grpc_client_status | gRPC server status code received, e.g. OK, CANCELLED, DEADLINE_EXCEEDED |
5055

5156
`grpc_client_method` is set when an outgoing request starts and is available in all the recorded
52-
metrics.
57+
metrics.
5358
`grpc_client_status` is set when an outgoing request finishes and is only available around metrics
5459
recorded at the end of the outgoing request.
5560
Status codes should be stringified according to:
@@ -59,13 +64,13 @@ https://github.com/grpc/grpc/blob/master/doc/statuscodes.md
5964

6065
The following set of views are considered minimum required to monitor client-side performance:
6166

62-
| View name | Measure | Aggregation | Tags |
63-
|---------------------------------------|---------------------------------------|--------------|----------------------------------------|
64-
| grpc.io/client/sent_bytes_per_rpc | grpc.io/client/sent_bytes_per_rpc | distribution | grpc_client_method |
65-
| grpc.io/client/received_bytes_per_rpc | grpc.io/client/received_bytes_per_rpc | distribution | grpc_client_method |
66-
| grpc.io/client/roundtrip_latency | grpc.io/client/roundtrip_latency | distribution | grpc_client_method |
67-
| grpc.io/client/completed_rpcs | grpc.io/client/roundtrip_latency | count | grpc_client_method, grpc_client_status |
68-
| grpc.io/client/started_rpcs | grpc.io/client/started_rpcs | count | grpc_client_method |
67+
| View name | Measure | Aggregation | Tags |
68+
|-------------------------------------------|-------------------------------------------|--------------|----------------------------------------|
69+
| grpc.io/client/sent_bytes_per_rpc | grpc.io/client/sent_bytes_per_rpc | distribution | grpc_client_method |
70+
| grpc.io/client/received_bytes_per_rpc | grpc.io/client/received_bytes_per_rpc | distribution | grpc_client_method |
71+
| grpc.io/client/roundtrip_latency | grpc.io/client/roundtrip_latency | distribution | grpc_client_method |
72+
| grpc.io/client/completed_rpcs | grpc.io/client/roundtrip_latency | count | grpc_client_method, grpc_client_status |
73+
| grpc.io/client/started_rpcs | grpc.io/client/started_rpcs | count | grpc_client_method |
6974

7075
### Extra views
7176

@@ -76,19 +81,27 @@ The following set of views are considered useful but not mandatory to monitor cl
7681
| grpc.io/client/sent_messages_per_rpc | grpc.io/client/sent_messages_per_rpc | distribution | grpc_client_method |
7782
| grpc.io/client/received_messages_per_rpc | grpc.io/client/received_messages_per_rpc | distribution | grpc_client_method |
7883
| grpc.io/client/server_latency | grpc.io/client/server_latency | distribution | grpc_client_method |
84+
| grpc.io/client/sent_messages_per_method | grpc.io/client/sent_messages_per_method | count | grpc_client_method |
85+
| grpc.io/client/received_messages_per_method | grpc.io/client/received_messages_per_method | count | grpc_client_method |
86+
| grpc.io/client/sent_bytes_per_method | grpc.io/client/sent_bytes_per_method | sum | grpc_client_method |
87+
| grpc.io/client/received_bytes_per_method | grpc.io/client/received_bytes_per_method | sum | grpc_client_method |
7988

8089
## Server
8190

8291
Server stats are recorded at the end of processing each RPC.
8392

84-
| Measure name | Unit | Description |
85-
|------------------------------------------|------|-----------------------------------------------------------------------------------------------|
86-
| grpc.io/server/received_messages_per_rpc | 1 | Number of messages received in each RPC. Has value 1 for non-streaming RPCs. |
87-
| grpc.io/server/received_bytes_per_rpc | By | Total bytes received across all messages per RPC. |
88-
| grpc.io/server/sent_messages_per_rpc | 1 | Number of messages sent in each RPC. Has value 1 for non-streaming RPCs. |
89-
| grpc.io/server/sent_bytes_per_rpc | By | Total bytes sent in across all response messages per RPC. |
90-
| grpc.io/server/server_latency | ms | Time between first byte of request received to last byte of response sent, or terminal error. |
91-
| grpc.io/server/started_rpcs | 1 | The total number of server RPCs ever opened, including those that have not completed. |
93+
| Measure name | Unit | Description |
94+
|-------------------------------------------|------|-----------------------------------------------------------------------------------------------|
95+
| grpc.io/server/received_messages_per_rpc | 1 | Number of messages received in each RPC. Has value 1 for non-streaming RPCs. |
96+
| grpc.io/server/received_bytes_per_rpc | By | Total bytes received across all messages per RPC. |
97+
| grpc.io/server/sent_messages_per_rpc | 1 | Number of messages sent in each RPC. Has value 1 for non-streaming RPCs. |
98+
| grpc.io/server/sent_bytes_per_rpc | By | Total bytes sent in across all response messages per RPC. |
99+
| grpc.io/server/server_latency | ms | Time between first byte of request received to last byte of response sent, or terminal error. |
100+
| grpc.io/server/started_rpcs | 1 | The total number of server RPCs ever opened, including those that have not completed. |
101+
| grpc.io/server/sent_messages_per_method | 1 | Total messages sent per method. |
102+
| grpc.io/server/received_messages_per_method | 1 | Total messages received per method. |
103+
| grpc.io/server/sent_bytes_per_method | By | Total bytes sent per method, recorded real-time as bytes are sent. |
104+
| grpc.io/server/received_bytes_per_method | By | Total bytes received per method, recorded real-time as bytes are received. |
92105

93106

94107
### Tags
@@ -101,7 +114,7 @@ All server metrics should be tagged with the following.
101114
| grpc_server_status | gRPC server status code returned, e.g. OK, CANCELLED, DEADLINE_EXCEEDED |
102115

103116
`grpc_server_method` is set when an incoming request starts and is available in the context for
104-
the entire RPC call handling.
117+
the entire RPC call handling.
105118

106119
`grpc_server_status` is set when an incoming request finishes and is only available around metrics
107120
recorded at the end of the incoming request.
@@ -112,13 +125,13 @@ https://github.com/grpc/grpc/blob/master/doc/statuscodes.md
112125

113126
The following set of views are considered minimum required to monitor server side performance:
114127

115-
| View name | Measure | Aggregation | Tags |
116-
|---------------------------------------|---------------------------------------|--------------|----------------------------------------|
117-
| grpc.io/server/received_bytes_per_rpc | grpc.io/server/received_bytes_per_rpc | distribution | grpc_server_method |
118-
| grpc.io/server/sent_bytes_per_rpc | grpc.io/server/sent_bytes_per_rpc | distribution | grpc_server_method |
119-
| grpc.io/server/server_latency | grpc.io/server/server_latency | distribution | grpc_server_method |
120-
| grpc.io/server/completed_rpcs | grpc.io/server/server_latency | count | grpc_server_method, grpc_server_status |
121-
| grpc.io/server/started_rpcs | grpc.io/server/started_rpcs | count | grpc_server_method |
128+
| View name | Measure | Aggregation | Tags |
129+
|-------------------------------------------|---------------------------------------|--------------|----------------------------------------|
130+
| grpc.io/server/received_bytes_per_rpc | grpc.io/server/received_bytes_per_rpc | distribution | grpc_server_method |
131+
| grpc.io/server/sent_bytes_per_rpc | grpc.io/server/sent_bytes_per_rpc | distribution | grpc_server_method |
132+
| grpc.io/server/server_latency | grpc.io/server/server_latency | distribution | grpc_server_method |
133+
| grpc.io/server/completed_rpcs | grpc.io/server/server_latency | count | grpc_server_method, grpc_server_status |
134+
| grpc.io/server/started_rpcs | grpc.io/server/started_rpcs | count | grpc_server_method |
122135

123136
### Extra views
124137

@@ -128,6 +141,10 @@ The following set of views are considered useful but not mandatory to monitor se
128141
|------------------------------------------|------------------------------------------|--------------|--------------------|
129142
| grpc.io/server/received_messages_per_rpc | grpc.io/server/received_messages_per_rpc | distribution | grpc_server_method |
130143
| grpc.io/server/sent_messages_per_rpc | grpc.io/server/sent_messages_per_rpc | distribution | grpc_server_method |
144+
| grpc.io/server/sent_messages_per_method | grpc.io/server/sent_messages_per_method | count | grpc_server_method |
145+
| grpc.io/server/received_messages_per_method | grpc.io/server/received_messages_per_method | count | grpc_server_method |
146+
| grpc.io/server/sent_bytes_per_method | grpc.io/server/sent_bytes_per_method | sum | grpc_server_method |
147+
| grpc.io/server/received_bytes_per_method | grpc.io/server/received_bytes_per_method | sum | grpc_server_method |
131148

132149
## FAQ
133150

@@ -147,3 +164,10 @@ Error counts can be computed on your metrics backend by totalling the different
147164
### Why are ".../completed_rpcs" views defined over latency measures?
148165
They can be defined over any measure recorded once per RPC (since it's just a count aggregation over the measure).
149166
It would be unnecessary to use a separate "count" measure.
167+
168+
### Why are "*_per_method" views not default?
169+
170+
These views are useful for real-time reporting for streaming RPCs. However, for unary calls
171+
they are not particularly useful, and data volume for these views could be huge compared to
172+
default views. Only enable these views if you are using streaming RPCs and want real-time
173+
metrics.

0 commit comments

Comments
 (0)