diff --git a/.editorconfig b/.editorconfig new file mode 100644 index 0000000..05caea0 --- /dev/null +++ b/.editorconfig @@ -0,0 +1,13 @@ +# http://editorconfig.org +root = true + +[*] +indent_style = space +indent_size = 2 +end_of_line = lf +charset = utf-8 +trim_trailing_whitespace = true +insert_final_newline = true + +[*.rs] +indent_size = 4 diff --git a/.gitignore b/.gitignore index 6936990..4fb5d9e 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ /target **/*.rs.bk Cargo.lock +/.vscode diff --git a/Cargo.toml b/Cargo.toml index 0c416b5..c026295 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,12 +1,10 @@ [workspace] members = [ - "metrics-core", "metrics", - "metrics-runtime", + "metrics-macros", "metrics-util", - "metrics-exporter-log", - "metrics-exporter-http", - "metrics-observer-yaml", - "metrics-observer-prometheus", - "metrics-observer-json", + "metrics-benchmark", + "metrics-exporter-tcp", + "metrics-exporter-prometheus", + "metrics-tracing-context", ] diff --git a/README.md b/README.md index af7ca0a..8448b4d 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,7 @@ The Metrics project: a metrics ecosystem for Rust. Running applications in production can be hard when you don't have insight into what the application is doing. We're lucky to have so many good system monitoring programs and services to show us how our servers are performing, but we still have to do the work of instrumenting our applications to gain deep insight into their behavior and performance. -_Metrics_ makes it easy to instrument your application to provide real-time insight into what's happening. It provides a number of practical features that make it easy for library and application authors to start collecting and exporting metrics from their codebase. +`metrics` makes it easy to instrument your application to provide real-time insight into what's happening. It provides a number of practical features that make it easy for library and application authors to start collecting and exporting metrics from their codebase. # why would I collect metrics? @@ -32,50 +32,35 @@ Some of the most common scenarios for collecting metrics from an application: Importantly, this works for both library authors and application authors. If the libraries you use are instrumented, you unlock the power of being able to collect those metrics in your application for free, without any extra configuration. Everyone wins, and learns more about their application performance at the end of the day. -# project goals - -Firstly, we want to establish standardized interfaces by which everyone can interoperate: this is the goal of the `metrics` and `metrics-core` crates. - -`metrics` provides macros similar to `log`, which are essentially zero cost and invisible when not in use, but automatically funnel their data when a user opts in and installs a metrics recorder. This allows library authors to instrument their libraries without needing to care which metrics system end users will be utilizing. - -`metrics-core` provides foundational traits for core components of the metrics ecosystem, primarily the output side. There are a large number of output formats and transports that application authors may consider or want to use. By focusing on the API boundary between the systems that collect metrics and the systems they're exported to, these pieces can be easily swapped around depending on the needs of the end user. - -Secondly, we want to provide a best-in-class reference runtime: this is the goal of the `metrics-runtime` crate. - -Unfortunately, a great interface is no good without a suitable implementation, and we want to make sure that for users looking to instrument their applications for the first time, that they have a batteries-included option that gets them off to the races quickly. The `metrics-runtime` crate provides a best-in-class implementation of a metrics collection system, including support for the core metric types -- counters, gauges, and histograms -- as well as support for important features such as scoping, labels, flexible approaches to recording, and more. - -On top of that, collecting metrics isn't terribly useful unless you can export those values, and so `metrics-runtime` pulls in a small set of default observers and exporters to allow users to quickly set up their application to be observable by their existing downstream metrics aggregation/storage. - # project layout The Metrics project provides a number of crates for both library and application authors. -If you're a library author, you'll only care about using [`metrics`] to instrument your library. If you're an application author, you'll primarily care about [`metrics-runtime`], but you may also want to use [`metrics`] to make instrumenting your own code even easier. +If you're a library author, you'll only care about using [`metrics`] to instrument your library. If +you're an application author, you'll likely also want to instrument your application, but you'll +care about "exporters" as a means to take those metrics and ship them somewhere for analysis. Overall, this repository is home to the following crates: * [`metrics`][metrics]: A lightweight metrics facade, similar to [`log`](https://docs.rs/log). -* [`metrics-core`][metrics-core]: Foundational traits for interoperable metrics libraries. -* [`metrics-runtime`][metrics-runtime]: A batteries-included metrics library. -* [`metrics-exporter-http`][metrics-exporter-http]: A metrics-core compatible exporter for serving metrics over HTTP. -* [`metrics-exporter-log`][metrics-exporter-log]: A metrics-core compatible exporter for forwarding metrics to logs. -* [`metrics-observer-json`][metrics-observer-json]: A metrics-core compatible observer that outputs JSON. -* [`metrics-observer-yaml`][metrics-observer-yaml]: A metrics-core compatible observer that outputs YAML. -* [`metrics-observer-prometheus`][metrics-observer-prometheus]: A metrics-core compatible observer that outputs the Prometheus exposition format. -* [`metrics-util`][metrics-util]: Helper types/functions used by the metrics ecosystem. +* [`metrics-macros`][metrics-macros]: Procedural macros that power `metrics`. +* [`metrics-tracing-context`][metrics-tracing-context]: Allow capturing [`tracing`][tracing] span + fields as metric labels. +* [`metrics-exporter-tcp`][metrics-exporter-tcp]: A `metrics`-compatible exporter for serving metrics over TCP. +* [`metrics-exporter-prometheus`][metrics-exporter-prometheus]: A `metrics`-compatible exporter for + serving a Prometheus scrape endpoint. +* [`metrics-util`][metrics-util]: Helper types/functions used by the `metrics` ecosystem. # contributing -We're always looking for users who have thoughts on how to make metrics better, or users with interesting use cases. Of course, we're also happy to accept code contributions for outstanding feature requests! 😀 +We're always looking for users who have thoughts on how to make `metrics` better, or users with interesting use cases. Of course, we're also happy to accept code contributions for outstanding feature requests! 😀 We'd love to chat about any of the above, or anything else, really! You can find us over on [Discord](https://discord.gg/eTwKyY9). [metrics]: https://github.com/metrics-rs/metrics/tree/master/metrics -[metrics-core]: https://github.com/metrics-rs/metrics/tree/master/metrics-core -[metrics-runtime]: https://github.com/metrics-rs/metrics/tree/master/metrics-runtime -[metrics-exporter-http]: https://github.com/metrics-rs/metrics/tree/master/metrics-exporter-http -[metrics-exporter-log]: https://github.com/metrics-rs/metrics/tree/master/metrics-exporter-log -[metrics-observer-json]: https://github.com/metrics-rs/metrics/tree/master/metrics-observer-json -[metrics-observer-yaml]: https://github.com/metrics-rs/metrics/tree/master/metrics-observer-yaml -[metrics-observer-prometheus]: https://github.com/metrics-rs/metrics/tree/master/metrics-observer-prometheus +[metrics-macros]: https://github.com/metrics-rs/metrics/tree/master/metrics-macros +[metrics-tracing-context]: https://github.com/metrics-rs/metrics/tree/master/metrics-tracing-context +[metrics-exporter-tcp]: https://github.com/metrics-rs/metrics/tree/master/metrics-exporter-tcp +[metrics-exporter-prometheus]: https://github.com/metrics-rs/metrics/tree/master/metrics-exporter-prometheus [metrics-util]: https://github.com/metrics-rs/metrics/tree/master/metrics-util +[tracing]: https://tracing.rs \ No newline at end of file diff --git a/ci/azure-test-minimum.yaml b/ci/azure-test-minimum.yaml index c605d95..5047afa 100644 --- a/ci/azure-test-minimum.yaml +++ b/ci/azure-test-minimum.yaml @@ -15,6 +15,6 @@ jobs: steps: - template: azure-install-rust.yml parameters: - rust_version: 1.39.0 + rust_version: 1.40.0 - script: cargo test displayName: cargo test diff --git a/metrics-benchmark/Cargo.toml b/metrics-benchmark/Cargo.toml new file mode 100644 index 0000000..ecd6450 --- /dev/null +++ b/metrics-benchmark/Cargo.toml @@ -0,0 +1,15 @@ +[package] +name = "metrics-benchmark" +version = "0.1.1-alpha.1" +authors = ["Toby Lawrence "] +edition = "2018" + +[dependencies] +log = "0.4" +env_logger = "0.7" +getopts = "0.2" +hdrhistogram = "7.0" +quanta = "0.6" +atomic-shim = "0.1" +metrics = { version = "0.13.0-alpha.0", path = "../metrics" } +metrics-util = { version = "0.4.0-alpha.0", path = "../metrics-util" } diff --git a/metrics-core/LICENSE b/metrics-benchmark/LICENSE similarity index 100% rename from metrics-core/LICENSE rename to metrics-benchmark/LICENSE diff --git a/metrics-runtime/examples/facade.rs b/metrics-benchmark/src/main.rs similarity index 75% rename from metrics-runtime/examples/facade.rs rename to metrics-benchmark/src/main.rs index b1d2202..4dbb138 100644 --- a/metrics-runtime/examples/facade.rs +++ b/metrics-benchmark/src/main.rs @@ -1,22 +1,12 @@ -#[macro_use] -extern crate log; -extern crate env_logger; -extern crate getopts; -extern crate hdrhistogram; -extern crate metrics_core; -extern crate metrics_runtime; -extern crate tokio; - -#[macro_use] -extern crate metrics; - use atomic_shim::AtomicU64; use getopts::Options; use hdrhistogram::Histogram; -use metrics_runtime::{exporters::HttpExporter, observers::JsonBuilder, Receiver}; -use quanta::Clock; +use log::{error, info}; +use metrics::{gauge, histogram, increment}; +use metrics_util::DebuggingRecorder; use std::{ env, + ops::Sub, sync::{ atomic::{AtomicBool, Ordering}, Arc, @@ -28,23 +18,21 @@ use std::{ const LOOP_SAMPLE: u64 = 1000; struct Generator { - t0: Option, + t0: Option, gauge: i64, hist: Histogram, done: Arc, rate_counter: Arc, - clock: Clock, } impl Generator { - fn new(done: Arc, rate_counter: Arc, clock: Clock) -> Generator { + fn new(done: Arc, rate_counter: Arc) -> Generator { Generator { t0: None, gauge: 0, hist: Histogram::::new_with_bounds(1, u64::max_value(), 3).unwrap(), done, rate_counter, - clock, } } @@ -59,22 +47,22 @@ impl Generator { self.gauge += 1; - let t1 = self.clock.now(); + let t1 = Instant::now(); if let Some(t0) = self.t0 { - let start = if counter % LOOP_SAMPLE == 0 { - self.clock.now() + let start = if counter % 1000 == 0 { + Some(Instant::now()) } else { - 0 + None }; - counter!("ok.gotem", 1); - timing!("ok.gotem", t0, t1); - gauge!("total", self.gauge); + increment!("ok"); + gauge!("total", self.gauge as f64); + histogram!("ok", t1.sub(t0)); - if start != 0 { - let delta = self.clock.now() - start; - self.hist.saturating_record(delta); + if let Some(val) = start { + let delta = Instant::now() - val; + self.hist.saturating_record(delta.as_nanos() as u64); // We also increment our global counter for the sample rate here. self.rate_counter @@ -121,8 +109,7 @@ pub fn opts() -> Options { opts } -#[tokio::main] -async fn main() { +fn main() { env_logger::init(); let args: Vec = env::args().collect(); @@ -159,35 +146,23 @@ async fn main() { info!("duration: {}s", seconds); info!("producers: {}", producers); - let receiver = Receiver::builder() - .histogram(Duration::from_secs(5), Duration::from_millis(100)) - .build() - .expect("failed to build receiver"); + let recorder = DebuggingRecorder::new(); + let snapshotter = recorder.snapshotter(); + recorder.install().expect("failed to install recorder"); - let controller = receiver.controller(); - - let addr = "0.0.0.0:23432" - .parse() - .expect("failed to parse http listen address"); - let builder = JsonBuilder::new().set_pretty_json(true); - let exporter = HttpExporter::new(controller.clone(), builder, addr); - tokio::spawn(exporter.async_run()); - - receiver.install(); - info!("receiver configured"); + info!("sink configured"); // Spin up our sample producers. let done = Arc::new(AtomicBool::new(false)); let rate_counter = Arc::new(AtomicU64::new(0)); let mut handles = Vec::new(); - let clock = Clock::new(); for _ in 0..producers { let d = done.clone(); let r = rate_counter.clone(); - let c = clock.clone(); let handle = thread::spawn(move || { - Generator::new(d, r, c).run(); + let mut gen = Generator::new(d, r); + gen.run(); }); handles.push(handle); @@ -202,7 +177,7 @@ async fn main() { let t1 = Instant::now(); let start = Instant::now(); - let _snapshot = controller.snapshot(); + let _snapshot = snapshotter.snapshot(); let end = Instant::now(); snapshot_hist.saturating_record(duration_as_nanos(end - start) as u64); @@ -219,7 +194,7 @@ async fn main() { info!("--------------------------------------------------------------------------------"); info!(" ingested samples total: {}", total); info!( - "snapshot end-to-end: min: {:9} p50: {:9} p95: {:9} p99: {:9} p999: {:9} max: {:9}", + "snapshot retrieval: min: {:9} p50: {:9} p95: {:9} p99: {:9} p999: {:9} max: {:9}", nanos_to_readable(snapshot_hist.min()), nanos_to_readable(snapshot_hist.value_at_percentile(50.0)), nanos_to_readable(snapshot_hist.value_at_percentile(95.0)), diff --git a/metrics-core/CHANGELOG.md b/metrics-core/CHANGELOG.md deleted file mode 100644 index 89afc27..0000000 --- a/metrics-core/CHANGELOG.md +++ /dev/null @@ -1,47 +0,0 @@ -# Changelog -All notable changes to this project will be documented in this file. - -The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), -and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - -## [Unreleased] - -## [0.5.2] - 2019-11-21 -### Changed -- Fixed a bug with the display output for `Key`. (#59) - -## [0.5.1] - 2019-08-08 -### Changed -- Fixed a bug with macros calling inner macros without a fully qualified name. - -## [0.5.0] - 2019-07-29 -### Added -- `Key` now supports labels. (#27) -- `Builder` for building observers in a more standardized way. (#30) - -### Changed -- `Recorder` is now `Observer`. (#35) - -## [0.4.0] - 2019-06-11 -### Added -- Add `Key` as the basis for metric names. (#20) -- Add `AsNanoseconds` for defining types that can be used for start/end times. (#20) - -## [0.3.1] - 2019-04-30 -### Removed -- Removed extraneous import. - -## [0.3.0] - 2019-04-30 -### Added -- Added snapshot traits for composable snapshotting. (#8) - -### Changed -- Reduced stuttering in type names. (#8) - -## [0.2.0] - 2019-04-23 -### Changed -- Changed from "exporter" to "recorder" in type names, documentation, etc. - -## [0.1.2] - 2019-03-26 -### Added -- Effective birth of the crate -- earlier versions were purely for others to experiment with. (#1) diff --git a/metrics-core/CODE_OF_CONDUCT.md b/metrics-core/CODE_OF_CONDUCT.md deleted file mode 100644 index 21dae9f..0000000 --- a/metrics-core/CODE_OF_CONDUCT.md +++ /dev/null @@ -1,30 +0,0 @@ -# The Code of Conduct - -This document is based on the [Rust Code of Conduct](https://www.rust-lang.org/conduct.html) and outlines the standard of conduct which is both expected and enforced as part of this project. - -## Conduct - -* We are committed to providing a friendly, safe and welcoming environment for all, regardless of level of experience, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, nationality, or other similar characteristic. -* Avoid using overtly sexual nicknames or other nicknames that might detract from a friendly, safe and welcoming environment for all. -* Please be kind and courteous. There's no need to be mean or rude. -* Respect that people have differences of opinion and that every design or implementation choice carries a trade-off and numerous costs. There is seldom a right answer. -* Please keep unstructured critique to a minimum. If you have solid ideas you want to experiment with, make a fork and see how it works. -* We will exclude you from interaction if you insult, demean or harass anyone. That is not welcome behaviour. We interpret the term "harassment" as including the definition in the [Citizen Code of Conduct](http://citizencodeofconduct.org/); if you have any lack of clarity about what might be included in that concept, please read their definition. In particular, we don't tolerate behavior that excludes people in socially marginalized groups. -* Private harassment is also unacceptable. No matter who you are, if you feel you have been or are being harassed or made uncomfortable by a community member, please contact one of the repository Owners immediately. Whether you're a regular contributor or a newcomer, we care about making this community a safe place for you and we've got your back. -* Likewise any spamming, trolling, flaming, baiting or other attention-stealing behaviour is not welcome. - -## Moderation - -These are the policies for upholding our community's standards of conduct. If you feel that a thread needs moderation, please use the contact information above, or mention @tobz or @LucioFranco in the thread. - -1. Remarks that violate this Code of Conduct, including hateful, hurtful, oppressive, or exclusionary remarks, are not allowed. (Cursing is allowed, but never targeting another user, and never in a hateful manner.) -2. Remarks that moderators find inappropriate, whether listed in the code of conduct or not, are also not allowed. - -In the Rust community we strive to go the extra step to look out for each other. Don't just aim to be technically unimpeachable, try to be your best self. In particular, avoid flirting with offensive or sensitive issues, particularly if they're off-topic; this all too often leads to unnecessary fights, hurt feelings, and damaged trust; worse, it can drive people away from the community entirely. - -And if someone takes issue with something you said or did, resist the urge to be defensive. Just stop doing what it was they complained about and apologize. Even if you feel you were misinterpreted or unfairly accused, chances are good there was something you could've communicated better — remember that it's your responsibility to make your fellow Rustaceans comfortable. Everyone wants to get along and we are all here first and foremost because we want to talk about cool technology. You will find that people will be eager to assume good intent and forgive as long as you earn their trust. - -## Contacts: - -- Toby Lawrence ([toby@nuclearfurnace.com](mailto:toby@nuclearfurnace.com)) -- Lucio Franco ([luciofranco14@gmail.com](mailto:luciofranco14@gmail.com)) diff --git a/metrics-core/Cargo.toml b/metrics-core/Cargo.toml deleted file mode 100644 index 9b310fd..0000000 --- a/metrics-core/Cargo.toml +++ /dev/null @@ -1,16 +0,0 @@ -[package] -name = "metrics-core" -version = "0.5.2" -authors = ["Toby Lawrence "] -edition = "2018" - -license = "MIT" - -description = "Foundational traits for interoperable metrics libraries." -homepage = "https://github.com/metrics-rs/metrics" -repository = "https://github.com/metrics-rs/metrics" -documentation = "https://docs.rs/metrics-core" -readme = "README.md" - -categories = ["development-tools::debugging"] -keywords = ["metrics", "interface", "common"] diff --git a/metrics-core/README.md b/metrics-core/README.md deleted file mode 100644 index f2d1dd2..0000000 --- a/metrics-core/README.md +++ /dev/null @@ -1,24 +0,0 @@ -# metrics-core - -[![conduct-badge][]][conduct] [![downloads-badge][] ![release-badge][]][crate] [![docs-badge][]][docs] [![license-badge][]](#license) - -[conduct-badge]: https://img.shields.io/badge/%E2%9D%A4-code%20of%20conduct-blue.svg -[downloads-badge]: https://img.shields.io/crates/d/metrics-core.svg -[release-badge]: https://img.shields.io/crates/v/metrics-core.svg -[license-badge]: https://img.shields.io/crates/l/metrics-core.svg -[docs-badge]: https://docs.rs/metrics-core/badge.svg -[conduct]: https://github.com/metrics-rs/metrics/blob/master/CODE_OF_CONDUCT.md -[crate]: https://crates.io/crates/metrics-core -[docs]: https://docs.rs/metrics-core - -__metrics-core__ defines foundational traits for interoperable metrics libraries in Rust. - -## code of conduct - -**NOTE**: All conversations and contributions to this project shall adhere to the [Code of Conduct][conduct]. - -## mandate / goals - -This crate acts as the minimum viable trait for metrics libraries, and consumers of that data, for interoperating with each other. - -If your library allows users to collect metrics, it should support metrics-core to allow for flexibility in output targets. If your library provides support for a target metrics backend, it should support metrics-core so that it can be easily plugged into applications using a supported metrics library. diff --git a/metrics-core/src/lib.rs b/metrics-core/src/lib.rs deleted file mode 100644 index 5f3e9fa..0000000 --- a/metrics-core/src/lib.rs +++ /dev/null @@ -1,347 +0,0 @@ -//! Foundational traits for interoperable metrics libraries in Rust. -//! -//! # Common Ground -//! Most libraries, under the hood, are all based around a core set of data types: counters, -//! gauges, and histograms. While the API surface may differ, the underlying data is the same. -//! -//! # Metric Types -//! -//! ## Counters -//! Counters represent a single value that can only ever be incremented over time, or reset to -//! zero. -//! -//! Counters are useful for tracking things like operations completed, or errors raised, where -//! the value naturally begins at zero when a process or service is started or restarted. -//! -//! ## Gauges -//! Gauges represent a single value that can go up _or_ down over time. -//! -//! Gauges are useful for tracking things like the current number of connected users, or a stock -//! price, or the temperature outside. -//! -//! ## Histograms -//! Histograms measure the distribution of values for a given set of measurements. -//! -//! Histograms are generally used to derive statistics about a particular measurement from an -//! operation or event that happens over and over, such as the duration of a request, or number of -//! rows returned by a particular database query. -//! -//! Histograms allow you to answer questions of these measurements, such as: -//! - "What were the fastest and slowest requests in this window?" -//! - "What is the slowest request we've seen out of 90% of the requests measured? 99%?" -//! -//! Histograms are a convenient way to measure behavior not only at the median, but at the edges of -//! normal operating behavior. -#![deny(missing_docs)] -use std::{borrow::Cow, fmt, slice::Iter, time::Duration}; - -/// An allocation-optimized string. -/// -/// We specify `ScopedString` to attempt to get the best of both worlds: flexibility to provide a -/// static or dynamic (owned) string, while retaining the performance benefits of being able to -/// take ownership of owned strings and borrows of completely static strings. -pub type ScopedString = Cow<'static, str>; - -/// A key/value pair used to further describe a metric. -#[derive(PartialEq, Eq, Hash, Clone, Debug)] -pub struct Label(ScopedString, ScopedString); - -impl Label { - /// Creates a `Label` from a key and value. - pub fn new(key: K, value: V) -> Self - where - K: Into, - V: Into, - { - Label(key.into(), value.into()) - } - - /// The key of this label. - pub fn key(&self) -> &str { - self.0.as_ref() - } - - /// The value of this label. - pub fn value(&self) -> &str { - self.1.as_ref() - } - - /// Consumes this `Label`, returning the key and value. - pub fn into_parts(self) -> (ScopedString, ScopedString) { - (self.0, self.1) - } -} - -/// A metric key. -/// -/// A key always includes a name, but can optional include multiple labels used to further describe -/// the metric. -#[derive(PartialEq, Eq, Hash, Clone, Debug)] -pub struct Key { - name: ScopedString, - labels: Vec