Skip to content
This repository was archived by the owner on Oct 3, 2023. It is now read-only.

Commit 6e91b23

Browse files
author
Bogdan Drutu
authored
Initial Stats API overview. (#51)
* Initial Stats API overview. * Relax requirements and respond to first round of comments. * Fix more comments. * Fix more comments. * Fix comment about recording multiple Measurements at once. * Fix more comments. * Add aggregations that allow us to support max and gauges. * Temporary drop Mean aggregation.
1 parent 6004dac commit 6e91b23

5 files changed

Lines changed: 187 additions & 1 deletion

File tree

‎README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,7 @@ The key elements of the API can be broken down as:
9999

100100
##### Links
101101

102-
* Trace API is described at the [Trace API][TraceAPI] document.
102+
* Trace API is [here][TraceAPI].
103103
* Data model is defined at the [Trace Data Model][TraceDataModel] document.
104104

105105
#### Tags
@@ -146,6 +146,7 @@ The key elements the API MUST provide are:
146146

147147
##### Links
148148

149+
* Stats API is [here][StatsAPI].
149150
* TODO: Add links to API definition and data model.
150151

151152
### Supported propagation formats
@@ -161,6 +162,7 @@ The key elements the API MUST provide are:
161162
[NamespaceAndPackage]: https://github.com/census-instrumentation/opencensus-specs/blob/master/NamespaceAndPackage.md
162163
[RFC2119]: https://www.ietf.org/rfc/rfc2119.txt
163164
[TraceAPI]: https://github.com/census-instrumentation/opencensus-specs/blob/master/trace/README.md
165+
[StatsAPI]: https://github.com/census-instrumentation/opencensus-specs/blob/master/stats/README.md
164166
[TraceContextSpecs]: https://github.com/TraceContext/tracecontext-spec
165167
[TraceDataModel]: https://github.com/census-instrumentation/opencensus-proto/blob/master/opencensus/proto/trace/trace.proto
166168
[BinaryEncoding]: https://github.com/census-instrumentation/opencensus-specs/blob/master/encodings/BinaryEncoding.md

‎stats/DataAggregation.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Data Aggregation API Overview
2+
The stats library allows users to define views to configure how data are aggregated and broken down.
3+
The core data types used are:
4+
* `Aggregation`: defines the aggregation type (e.g. `Distribution`, `Sum`, `Mean`).
5+
* `View`: defines how individual measurements are broken down by tags (similar to labels in some
6+
other systems) and aggregated.
7+
8+
## Aggregation
9+
An Aggregation describes how data collected is aggregated.
10+
11+
The library SHOULD provide support for multiple types of Aggregations:
12+
* `Count`: counts the number of measurements recorded.
13+
* `Sum`: indicates that data collected and aggregated with this `Aggregation` will be summed up.
14+
* `Max`: indicates that data collected and aggregated with this `Aggregation` will calculate the
15+
maximum value recorded.
16+
* `LastValue`: indicates that data collected and aggregated with this `Aggregation` will
17+
represent the last recorded value. This is useful to support Gauges.
18+
* `Distribution`: indicates that the desired `Aggregation` is a histogram distribution. A
19+
distribution `Aggregation` may contain a histogram of the values in the population. User should
20+
define the bucket boundaries for that histogram.
21+
22+
## View
23+
A View describes how individual measurements are both broken down (by a set of tag keys) and
24+
aggregated (e.g. into Distributions). A single `Measure` can be used by multiple Views.
25+
26+
A View is defined from the following:
27+
* `name`: a string by which the View will be referred to, e.g. "rpc_latency". Names MUST be unique
28+
within the library.
29+
* `description`: a free format descriptive string, e.g. "RPC latency distribution".
30+
* `Measure`: the Measure to which this view is applied.
31+
* `columns`: an array of tag keys. These values associated with tags of this name form the basis
32+
by which individual stats will be aggregated (one aggregation per unique tag value). If none are
33+
provided, then all data is recorded in a single aggregation.
34+
* `aggregation`: `Distribution`, `Count`, `Sum`, `Mean`.
35+
36+
Implementations SHOULD define a View data type, constructed from the parameters above. Views MAY
37+
have getters for retrieving all of the information used in View definition. Once created, View
38+
metadata is immutable.
39+
40+
Example in Java
41+
```java
42+
// Define a list of tags to break down view by RPC method.
43+
private static final ArrayList tagKeyListForMethodViews = new ArrayList();
44+
tagKeyListForMethodViews.add(RPC_METHOD_KEY);
45+
46+
// Create latency buckets from 100us to 10s.
47+
private static final BucketBoundaries LATENCY_DISTRIBUTION_BUCKETS =
48+
new BucketBoundaries(Arrays.asList(0, 0.01, 0.1, 1.0, 10.0, 1000.0, 10000.0));
49+
50+
// Create a distribution with the latency buckets.
51+
private static final Distribution LATENCY_DISTRIBUTION =
52+
Distribution.create(LATENCY_DISTRIBUTION_BUCKETS);
53+
54+
// Define a view to collect RPC latency stats by RPC method.
55+
private static final View RPC_LATENCY_BY_METHOD_VIEW = new View(
56+
View.Name.create("rpc_latency_by_method"),
57+
"Distribution of RPC latency broken down by Method", // description
58+
RPC_LATENCY, // Measure
59+
tagKeyListForMethodViews, // aggregation_tags
60+
LATENCY_DISTRIBUTION);
61+
```
62+
63+
References to Views MAY be obtained from querying of registered Views at runtime. This
64+
functionality enables decoupling the definition of the data aggregation from the exporting
65+
of the aggregated data.
66+
67+
For languages that do not allow private properties/metadata and if they are needed implementations
68+
MAY define a `ViewDescription` data type which contains all the read-only fields from the `View`
69+
definition such as: `name`, `description`, `MeasureDescription` and `columns` and `Aggregation`.

‎stats/Export.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Export API Overview
2+
The stats library aggregates measurements into Views (which are essentially a collection of
3+
metrics, each with a different set of labels). All output objects are immutable. The core data
4+
types used are:
5+
* `AggregationData`: contains data from an `Aggregation`.
6+
* `ViewData`: encapsulates `View` data aggregation.
7+
8+
### AggregationData
9+
An AggregationData describes the result of data aggregation. The library SHOULD define an
10+
AggregationData for each Aggregation types defined.
11+
12+
The library SHOULD provide support for multiple types of Aggregations:
13+
* `CountData`: data generated for a `Count` aggregation.
14+
* `SumDataDouble` and `SumDataInt64`: data generated for a `Sum` aggregation based on the `Measure`
15+
type.
16+
* `MaxData`: data generated for a `Max` aggregation.
17+
* `LastValueData`: data generated for a `LastValue` aggregation.
18+
* `DistributionData`: data generated for a `Distribution` aggregation.
19+
20+
### ViewData
21+
A ViewData is defined from the following:
22+
* `View`: the exported `View`.
23+
* `rows`: a map of repeated tag values, and, dependent on Aggregation type in the `View` an
24+
`AggregationData` object. These contain the respective tag values corresponding to the `columns`
25+
parameter from the `View` and the aggregated data associated with those `columns`.
26+
* `start_time`: a timestamp, describing the start time of the current stats snapshot.
27+
* `end_time`: a timestamp, describing the end time of the current stats snapshot.
28+
29+
The library SHOULD provide a means of retrieving the ViewData for any registered view in the system.

‎stats/README.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# OpenCensus Library Stats API
2+
This documentation serves to document the "look and feel" of the open source stats API's. It
3+
describes the key types and the overall behavior.
4+
5+
## Overview
6+
OpenCensus allows users to create typed measures, record measurements, define data aggregation, and
7+
export aggregated data.
8+
9+
### Main APIs
10+
* [Record API](Record.md): used by the application and libraries (e.g gRPC) owners to record
11+
metrics.
12+
* [Data Aggregation API](DataAggregation.md): used by the application owners to define and enable
13+
data aggregation.
14+
* [Export API](Export.md): used by the specific vendors to define vendor specific exporters (e.g.
15+
Prometheus, Stackdriver, SignalFx).

‎stats/Record.md‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# Record API Overview
2+
The stats library allows users to record metrics for their applications or libraries. The core
3+
data types used are:
4+
* `Measure`: describes the type of the individual values recorded by an application.
5+
* `Measurement`: describes a data point to be collected for a `Measure`.
6+
7+
## Measure
8+
A `Measure` describes the type of the individual values/measurements recorded by an application. It
9+
includes information such as the type of measurement, the units of measurement and descriptive names
10+
for the data. This provides the fundamental type used for recording data.
11+
12+
A Measure describes a value with the following metadata:
13+
* `name`: a string by which the measure will be referred to, e.g. "rpc_server_latency", or
14+
"vm_cpu_cycles". Names MUST be unique within the library. It is recommended to use names
15+
compatible with the intended end usage, e.g, use host/path pattern.
16+
* `description`: a string describes the measure, e.g. "RPC latency in seconds", "Virtual cycles
17+
executed on VM".
18+
* `unit`: a string describing the unit used for the `Measure`. Follows the format described by
19+
[Unified Code for Units of Measure](http://unitsofmeasure.org/ucum.html).
20+
* `type`: the only supported types are `int64` and `double`.
21+
22+
Implementations MAY define a `Measure` data type, constructed from the parameters above.
23+
Measure MAY have getters for retrieving all of the information used in `Measure` definition.
24+
Once created, Measure metadata is immutable.
25+
26+
Example in Java
27+
```java
28+
private static final MeasureDouble RPC_LATENCY =
29+
MeasureDouble.create("grpc.io/latency", "latency", "ms");
30+
```
31+
32+
References to Measures in the system MAY be obtained from querying of registered Measures. This
33+
functionality is required to decouple the recording of the data from the exporting of the data.
34+
35+
For languages that do not allow private properties/metadata and if they are needed implementations
36+
MAY define a `MeasureDescription` data type which contains all the read-only fields from the
37+
`Measure` definition such as: `name`, `description`, `unit` and `type`.
38+
39+
## Measurement
40+
A `Measurement` is defined from the following:
41+
* `Measure`: the `Measure` to which this `value` is applied.
42+
* `value`: recorded value, MUST have the appropriate type to match the `Measure` definition.
43+
44+
Implementations MAY define a `MeasurementMap` which describes a set of data points to be collected
45+
for a set of Measures. Adding this functionality may improve the efficiency of the record usage API.
46+
47+
## Recording Stats
48+
49+
Users should record Measurements against a context, either an explicit context or the implicit
50+
current context. Tags from the context are recorded with the Measurements if they are any.
51+
52+
Implementations SHOULD provide a means of recording multiple Measurements at once. This
53+
functionality can be provided through one of the following options:
54+
* As a method in a class/package (recommended name Stats/stats), taking a list of Measurement as
55+
argument. e.g. `record(List<Measurement>)` or `record(...Measurement)`.
56+
* As a `record` method of the appropriate data type. e.g. `MeasurementMap.record()`.
57+
58+
Example in Java
59+
60+
```java
61+
private static final MeasureDouble RPC_LATENCY =
62+
MeasureDouble.create("grpc.io/client/latency", "latency", "ms");
63+
private static final MeasureLong RPC_BYTES_SENT =
64+
MeasureLong.create("grpc.io/client/bytes_sent", "bytes sent", "kb");
65+
66+
MeasurementMap measurementMap = new MeasurementMap();
67+
measurementMap.put(RPC_LATENCY, 10.3);
68+
measurementMap.put(RPC_BYTES_SENT, 124);
69+
measurementMap.record(); // reads context from thread-local.
70+
}
71+
```

0 commit comments

Comments
 (0)