metrics/metrics-runtime/src/builder.rs

97 lines
3.4 KiB
Rust
Raw Normal View History

2019-06-01 12:55:28 -07:00
use crate::{config::Configuration, Receiver};
2019-07-17 06:06:45 -07:00
use std::{error::Error, fmt, time::Duration};
Entirely remove the event loop and switch to pure atomics. Originally, metrics (and `hotmic` before it was converted) was based on an event loop that centered around `mio`'s `Poll` interface with a custom channel to read and write metrics to. That model required a dedicated thread to run to poll for writes, and ingest them, managing the internal data structures in turn. Eventually, I rewrote that portion to be based on `crossbeam-channel` but we still depended on a background thread to pop samples off the channel and process them. Instead, we've rewritten the core of metrics to be based purely on atomics, with the caveat that we still do have a background thread. Instead of a single channel that metrics are funneled into, each underlying metric becomes a single-track codepath: each metric is backed by an atomic structure which means we can pass handles to that storage as far as the callers themselves, eliminating the need to funnel metrics into the "core" where they all contend for processing. Counters are gauges are now, effectively, wrapped atomic integers, which means we can process over 100 million counter/gauge updates per core. Histograms are based on a brand-new atomic "bucket" that allows for fast, unbounded writes and the ability to snapshot at any time. The end result is that we can process a mixed workload (counter, gauge, and histogram) at sample rates of up to 30 million samples per second per core, with p999 ingest latencies of in the low hundreds of nanoseconds. Taking snapshots also now avoids stalling the event loop and driving up tail latencies for ingest, and writers can proceed with no interruption. There is still a background thread that is part of quanta's new "recent" time support, which allows a background thread to incrementally update a shared global time, which can be accessed more quickly than grabbing the time directly. For our purposes, only histograms need the time to perform the window upkeep inherent to the sliding windows we use, and the time they need can be far less precise than what quanta is capable of. This background thread is spawned automatically when creating a receiver, and drops when the receiver goes away. By default, it updates 20 times a second performing an operation which itself takes less than 100ns, so all in all, this background thread should be imperceptible, performance-wise, on all systems*. On top of all of this, we've embraced the natural pattern of defining metrics individually at the variable/field level, and added supported for proxy types, which can be acquired from a sink and embedded as fields within your existing types, which lets you directly update metrics with the ease of accessing a field in an object. Sinks still have the ability to have metrics pushed directly into them, but this just opens up more possibilities. * - famous last words
2019-05-27 17:41:42 -07:00
/// Errors during receiver creation.
#[derive(Debug, Clone)]
pub enum BuilderError {
/// Failed to spawn the upkeep thread.
///
/// As histograms are windowed, reads and writes require getting the current time so they can
/// perform the required maintenance, or upkeep, on the internal structures to roll over old
/// buckets, etc.
///
/// Acquiring the current time is fast compared to most operations, but is a significant
/// portion of the other time it takes to write to a histogram, which limits overall throughput
/// under high load.
///
/// We spin up a background thread, or the "upkeep thread", which updates a global time source
/// that the read and write operations exclusively rely on. While this source is not as
/// up-to-date as the real clock, it is much faster to access.
UpkeepFailure,
2019-06-01 12:55:28 -07:00
#[doc(hidden)]
_NonExhaustive,
Entirely remove the event loop and switch to pure atomics. Originally, metrics (and `hotmic` before it was converted) was based on an event loop that centered around `mio`'s `Poll` interface with a custom channel to read and write metrics to. That model required a dedicated thread to run to poll for writes, and ingest them, managing the internal data structures in turn. Eventually, I rewrote that portion to be based on `crossbeam-channel` but we still depended on a background thread to pop samples off the channel and process them. Instead, we've rewritten the core of metrics to be based purely on atomics, with the caveat that we still do have a background thread. Instead of a single channel that metrics are funneled into, each underlying metric becomes a single-track codepath: each metric is backed by an atomic structure which means we can pass handles to that storage as far as the callers themselves, eliminating the need to funnel metrics into the "core" where they all contend for processing. Counters are gauges are now, effectively, wrapped atomic integers, which means we can process over 100 million counter/gauge updates per core. Histograms are based on a brand-new atomic "bucket" that allows for fast, unbounded writes and the ability to snapshot at any time. The end result is that we can process a mixed workload (counter, gauge, and histogram) at sample rates of up to 30 million samples per second per core, with p999 ingest latencies of in the low hundreds of nanoseconds. Taking snapshots also now avoids stalling the event loop and driving up tail latencies for ingest, and writers can proceed with no interruption. There is still a background thread that is part of quanta's new "recent" time support, which allows a background thread to incrementally update a shared global time, which can be accessed more quickly than grabbing the time directly. For our purposes, only histograms need the time to perform the window upkeep inherent to the sliding windows we use, and the time they need can be far less precise than what quanta is capable of. This background thread is spawned automatically when creating a receiver, and drops when the receiver goes away. By default, it updates 20 times a second performing an operation which itself takes less than 100ns, so all in all, this background thread should be imperceptible, performance-wise, on all systems*. On top of all of this, we've embraced the natural pattern of defining metrics individually at the variable/field level, and added supported for proxy types, which can be acquired from a sink and embedded as fields within your existing types, which lets you directly update metrics with the ease of accessing a field in an object. Sinks still have the ability to have metrics pushed directly into them, but this just opens up more possibilities. * - famous last words
2019-05-27 17:41:42 -07:00
}
impl Error for BuilderError {}
impl fmt::Display for BuilderError {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
match self {
BuilderError::UpkeepFailure => write!(f, "failed to spawn quanta upkeep thread"),
2019-06-01 12:55:28 -07:00
BuilderError::_NonExhaustive => write!(f, "non-exhaustive matching"),
Entirely remove the event loop and switch to pure atomics. Originally, metrics (and `hotmic` before it was converted) was based on an event loop that centered around `mio`'s `Poll` interface with a custom channel to read and write metrics to. That model required a dedicated thread to run to poll for writes, and ingest them, managing the internal data structures in turn. Eventually, I rewrote that portion to be based on `crossbeam-channel` but we still depended on a background thread to pop samples off the channel and process them. Instead, we've rewritten the core of metrics to be based purely on atomics, with the caveat that we still do have a background thread. Instead of a single channel that metrics are funneled into, each underlying metric becomes a single-track codepath: each metric is backed by an atomic structure which means we can pass handles to that storage as far as the callers themselves, eliminating the need to funnel metrics into the "core" where they all contend for processing. Counters are gauges are now, effectively, wrapped atomic integers, which means we can process over 100 million counter/gauge updates per core. Histograms are based on a brand-new atomic "bucket" that allows for fast, unbounded writes and the ability to snapshot at any time. The end result is that we can process a mixed workload (counter, gauge, and histogram) at sample rates of up to 30 million samples per second per core, with p999 ingest latencies of in the low hundreds of nanoseconds. Taking snapshots also now avoids stalling the event loop and driving up tail latencies for ingest, and writers can proceed with no interruption. There is still a background thread that is part of quanta's new "recent" time support, which allows a background thread to incrementally update a shared global time, which can be accessed more quickly than grabbing the time directly. For our purposes, only histograms need the time to perform the window upkeep inherent to the sliding windows we use, and the time they need can be far less precise than what quanta is capable of. This background thread is spawned automatically when creating a receiver, and drops when the receiver goes away. By default, it updates 20 times a second performing an operation which itself takes less than 100ns, so all in all, this background thread should be imperceptible, performance-wise, on all systems*. On top of all of this, we've embraced the natural pattern of defining metrics individually at the variable/field level, and added supported for proxy types, which can be acquired from a sink and embedded as fields within your existing types, which lets you directly update metrics with the ease of accessing a field in an object. Sinks still have the ability to have metrics pushed directly into them, but this just opens up more possibilities. * - famous last words
2019-05-27 17:41:42 -07:00
}
}
}
/// Builder for [`Receiver`].
#[derive(Clone)]
pub struct Builder {
pub(crate) histogram_window: Duration,
pub(crate) histogram_granularity: Duration,
2019-06-01 12:55:28 -07:00
pub(crate) upkeep_interval: Duration,
Entirely remove the event loop and switch to pure atomics. Originally, metrics (and `hotmic` before it was converted) was based on an event loop that centered around `mio`'s `Poll` interface with a custom channel to read and write metrics to. That model required a dedicated thread to run to poll for writes, and ingest them, managing the internal data structures in turn. Eventually, I rewrote that portion to be based on `crossbeam-channel` but we still depended on a background thread to pop samples off the channel and process them. Instead, we've rewritten the core of metrics to be based purely on atomics, with the caveat that we still do have a background thread. Instead of a single channel that metrics are funneled into, each underlying metric becomes a single-track codepath: each metric is backed by an atomic structure which means we can pass handles to that storage as far as the callers themselves, eliminating the need to funnel metrics into the "core" where they all contend for processing. Counters are gauges are now, effectively, wrapped atomic integers, which means we can process over 100 million counter/gauge updates per core. Histograms are based on a brand-new atomic "bucket" that allows for fast, unbounded writes and the ability to snapshot at any time. The end result is that we can process a mixed workload (counter, gauge, and histogram) at sample rates of up to 30 million samples per second per core, with p999 ingest latencies of in the low hundreds of nanoseconds. Taking snapshots also now avoids stalling the event loop and driving up tail latencies for ingest, and writers can proceed with no interruption. There is still a background thread that is part of quanta's new "recent" time support, which allows a background thread to incrementally update a shared global time, which can be accessed more quickly than grabbing the time directly. For our purposes, only histograms need the time to perform the window upkeep inherent to the sliding windows we use, and the time they need can be far less precise than what quanta is capable of. This background thread is spawned automatically when creating a receiver, and drops when the receiver goes away. By default, it updates 20 times a second performing an operation which itself takes less than 100ns, so all in all, this background thread should be imperceptible, performance-wise, on all systems*. On top of all of this, we've embraced the natural pattern of defining metrics individually at the variable/field level, and added supported for proxy types, which can be acquired from a sink and embedded as fields within your existing types, which lets you directly update metrics with the ease of accessing a field in an object. Sinks still have the ability to have metrics pushed directly into them, but this just opens up more possibilities. * - famous last words
2019-05-27 17:41:42 -07:00
}
impl Default for Builder {
fn default() -> Self {
Self {
histogram_window: Duration::from_secs(10),
histogram_granularity: Duration::from_secs(1),
2019-06-01 12:55:28 -07:00
upkeep_interval: Duration::from_millis(50),
Entirely remove the event loop and switch to pure atomics. Originally, metrics (and `hotmic` before it was converted) was based on an event loop that centered around `mio`'s `Poll` interface with a custom channel to read and write metrics to. That model required a dedicated thread to run to poll for writes, and ingest them, managing the internal data structures in turn. Eventually, I rewrote that portion to be based on `crossbeam-channel` but we still depended on a background thread to pop samples off the channel and process them. Instead, we've rewritten the core of metrics to be based purely on atomics, with the caveat that we still do have a background thread. Instead of a single channel that metrics are funneled into, each underlying metric becomes a single-track codepath: each metric is backed by an atomic structure which means we can pass handles to that storage as far as the callers themselves, eliminating the need to funnel metrics into the "core" where they all contend for processing. Counters are gauges are now, effectively, wrapped atomic integers, which means we can process over 100 million counter/gauge updates per core. Histograms are based on a brand-new atomic "bucket" that allows for fast, unbounded writes and the ability to snapshot at any time. The end result is that we can process a mixed workload (counter, gauge, and histogram) at sample rates of up to 30 million samples per second per core, with p999 ingest latencies of in the low hundreds of nanoseconds. Taking snapshots also now avoids stalling the event loop and driving up tail latencies for ingest, and writers can proceed with no interruption. There is still a background thread that is part of quanta's new "recent" time support, which allows a background thread to incrementally update a shared global time, which can be accessed more quickly than grabbing the time directly. For our purposes, only histograms need the time to perform the window upkeep inherent to the sliding windows we use, and the time they need can be far less precise than what quanta is capable of. This background thread is spawned automatically when creating a receiver, and drops when the receiver goes away. By default, it updates 20 times a second performing an operation which itself takes less than 100ns, so all in all, this background thread should be imperceptible, performance-wise, on all systems*. On top of all of this, we've embraced the natural pattern of defining metrics individually at the variable/field level, and added supported for proxy types, which can be acquired from a sink and embedded as fields within your existing types, which lets you directly update metrics with the ease of accessing a field in an object. Sinks still have the ability to have metrics pushed directly into them, but this just opens up more possibilities. * - famous last words
2019-05-27 17:41:42 -07:00
}
}
}
impl Builder {
/// Creates a new [`Builder`] with default values.
pub fn new() -> Self {
Default::default()
}
/// Sets the histogram configuration.
///
/// Defaults to a 10 second window with 1 second granularity.
///
/// This controls both how long of a time window we track histogram data for, and the
/// granularity in which we roll off old data.
///
/// As an example, with the default values, we would keep the last 10 seconds worth of
/// histogram data, and would remove 1 seconds worth of data at a time as the window rolled
/// forward.
pub fn histogram(mut self, window: Duration, granularity: Duration) -> Self {
self.histogram_window = window;
self.histogram_granularity = granularity;
self
}
2019-06-01 12:55:28 -07:00
/// Sets the upkeep interval.
///
/// Defaults to 50 milliseconds.
///
/// This controls how often the time source, used internally by histograms, is updated with the
/// real time. For performance reasons, histograms use a sampled time source when they perform
/// checks to see if internal maintenance needs to occur. If the histogram granularity is set
/// very low, then this interval might need to be similarly reduced to make sure we're able to
/// update the time more often than histograms need to perform upkeep.
pub fn upkeep_interval(mut self, interval: Duration) -> Self {
self.upkeep_interval = interval;
self
}
Entirely remove the event loop and switch to pure atomics. Originally, metrics (and `hotmic` before it was converted) was based on an event loop that centered around `mio`'s `Poll` interface with a custom channel to read and write metrics to. That model required a dedicated thread to run to poll for writes, and ingest them, managing the internal data structures in turn. Eventually, I rewrote that portion to be based on `crossbeam-channel` but we still depended on a background thread to pop samples off the channel and process them. Instead, we've rewritten the core of metrics to be based purely on atomics, with the caveat that we still do have a background thread. Instead of a single channel that metrics are funneled into, each underlying metric becomes a single-track codepath: each metric is backed by an atomic structure which means we can pass handles to that storage as far as the callers themselves, eliminating the need to funnel metrics into the "core" where they all contend for processing. Counters are gauges are now, effectively, wrapped atomic integers, which means we can process over 100 million counter/gauge updates per core. Histograms are based on a brand-new atomic "bucket" that allows for fast, unbounded writes and the ability to snapshot at any time. The end result is that we can process a mixed workload (counter, gauge, and histogram) at sample rates of up to 30 million samples per second per core, with p999 ingest latencies of in the low hundreds of nanoseconds. Taking snapshots also now avoids stalling the event loop and driving up tail latencies for ingest, and writers can proceed with no interruption. There is still a background thread that is part of quanta's new "recent" time support, which allows a background thread to incrementally update a shared global time, which can be accessed more quickly than grabbing the time directly. For our purposes, only histograms need the time to perform the window upkeep inherent to the sliding windows we use, and the time they need can be far less precise than what quanta is capable of. This background thread is spawned automatically when creating a receiver, and drops when the receiver goes away. By default, it updates 20 times a second performing an operation which itself takes less than 100ns, so all in all, this background thread should be imperceptible, performance-wise, on all systems*. On top of all of this, we've embraced the natural pattern of defining metrics individually at the variable/field level, and added supported for proxy types, which can be acquired from a sink and embedded as fields within your existing types, which lets you directly update metrics with the ease of accessing a field in an object. Sinks still have the ability to have metrics pushed directly into them, but this just opens up more possibilities. * - famous last words
2019-05-27 17:41:42 -07:00
/// Create a [`Receiver`] based on this configuration.
pub fn build(self) -> Result<Receiver, BuilderError> {
2019-06-01 12:55:28 -07:00
let config = Configuration::from_builder(&self);
Receiver::from_config(config)
Entirely remove the event loop and switch to pure atomics. Originally, metrics (and `hotmic` before it was converted) was based on an event loop that centered around `mio`'s `Poll` interface with a custom channel to read and write metrics to. That model required a dedicated thread to run to poll for writes, and ingest them, managing the internal data structures in turn. Eventually, I rewrote that portion to be based on `crossbeam-channel` but we still depended on a background thread to pop samples off the channel and process them. Instead, we've rewritten the core of metrics to be based purely on atomics, with the caveat that we still do have a background thread. Instead of a single channel that metrics are funneled into, each underlying metric becomes a single-track codepath: each metric is backed by an atomic structure which means we can pass handles to that storage as far as the callers themselves, eliminating the need to funnel metrics into the "core" where they all contend for processing. Counters are gauges are now, effectively, wrapped atomic integers, which means we can process over 100 million counter/gauge updates per core. Histograms are based on a brand-new atomic "bucket" that allows for fast, unbounded writes and the ability to snapshot at any time. The end result is that we can process a mixed workload (counter, gauge, and histogram) at sample rates of up to 30 million samples per second per core, with p999 ingest latencies of in the low hundreds of nanoseconds. Taking snapshots also now avoids stalling the event loop and driving up tail latencies for ingest, and writers can proceed with no interruption. There is still a background thread that is part of quanta's new "recent" time support, which allows a background thread to incrementally update a shared global time, which can be accessed more quickly than grabbing the time directly. For our purposes, only histograms need the time to perform the window upkeep inherent to the sliding windows we use, and the time they need can be far less precise than what quanta is capable of. This background thread is spawned automatically when creating a receiver, and drops when the receiver goes away. By default, it updates 20 times a second performing an operation which itself takes less than 100ns, so all in all, this background thread should be imperceptible, performance-wise, on all systems*. On top of all of this, we've embraced the natural pattern of defining metrics individually at the variable/field level, and added supported for proxy types, which can be acquired from a sink and embedded as fields within your existing types, which lets you directly update metrics with the ease of accessing a field in an object. Sinks still have the ability to have metrics pushed directly into them, but this just opens up more possibilities. * - famous last words
2019-05-27 17:41:42 -07:00
}
}