Commit Graph

74 Commits

Author SHA1 Message Date
Michael Vines 3f4731b37f Standardize thread names
Tenets:
1. Limit thread names to 15 characters
2. Prefix all Solana-controlled threads with "sol"
3. Use Camel case. It's more character dense than Snake or Kebab case
2022-08-20 07:49:39 -07:00
behzad nouri ba785cf8ab
removes erroneous uses of std::mem::swap (#26536)
All instances should be replace by std::mem::{replace,take},
or just plain assignment.
2022-07-11 11:33:15 +00:00
steviez 136eb43f7d
Allow metric tag values to be runtime configurable (#25615) 2022-06-06 12:14:38 -05:00
Yueh-Hsuan Chiang 6de2884969
Extend the datapoint macro to support group-by operations. (#25385)
#### Problem
Our existing datapoint macro syntax does directly not support group-by tags.
The existing workaround is to embed the group-by tags into the metric name which does not scale well.

#### Summary of Changes
This PR extends the existing syntax to support group-by tags as follows.  The new syntax is also compatible with the existing syntax:
```
datapoint_debug!(
   "metric_name",
   "tag" => "tag-value",
   "tag2" => "tag-value2",
   ....
   ("field1", 100, i64),  // field syntax is the same as the current syntax.
   ("field2", "hello", String),  
   ...
);
```
2022-05-26 17:00:58 -07:00
Michael Vines b05c7d91ed Fix derive_partial_eq_without_eq clippy lint 2022-05-22 22:22:21 -07:00
Yueh-Hsuan Chiang d121a06826
Add comment block and examples for datapoint.rs (#25393)
#### Summary of Changes
Add comment block and examples for datapoint.rs
2022-05-21 16:41:51 -07:00
sakridge a63d99fd11
Use write! to fix clippy and save heap allocation (#25220) 2022-05-16 22:39:33 +02:00
HaoranYi 591bfc3e0c
optimize counter submission: avoid copy when converting counters to datapoints (#24802) 2022-04-29 21:21:57 -05:00
HaoranYi 71ad121282
metric optimization: simply data structure used for metric datapoints and counter storage (#24447)
* remove redudant if

* check loglevel enable before submit metrics

* optimize metric data structure

* add metrics benches

* clippy: add default impl

* remove clone

* add random benches

* use mem::swap to exchange points vec
2022-04-27 09:11:29 -05:00
HaoranYi 3abd46010c
Report metric command queue size (#24661)
* report metric command queue size

* rename buffered_points
2022-04-25 19:18:38 -05:00
sakridge e7fcda1424
Quic client stats (#24195)
* Add metrics to connection-cache to measure cache hits and misses

* Add congestion stats

* Add more client stats

* Review comments

Co-authored-by: Ryan Leung <ryan.leung@solana.com>
2022-04-13 05:04:40 +02:00
Michael Vines a2be810dbc Resolve new clippy complaints 2022-04-11 18:15:03 -04:00
HaoranYi ba770832d0
Poh timing service (#23736)
* initial work for poh timing report service

* add poh_timing_report_service to validator

* fix comments

* clippy

* imrove test coverage

* delete record when complete

* rename shred full to slot full.

* debug logging

* fix slot full

* remove debug comments

* adding fmt trait

* derive default

* default for poh timing reporter

* better comments

* remove commented code

* fix test

* more test fixes

* delete timestamps for slot that are older than root_slot

* debug log

* record poh start end in bank reset

* report full to start time instead

* fix poh slot offset

* report poh start for normal ticks

* fix typo

* refactor out poh point report fn

* rename

* optimize delete - delete only when last_root changed

* change log level to trace

* convert if to match

* remove redudant check

* fix SlotPohTiming comments

* review feedback on poh timing reporter

* review feedback on poh_recorder

* add test case for out-of-order arrival of timing points and incomplete timing points

* refactor poh_timing_points into its own mod

* remove option for poh_timing_report service

* move poh_timing_point_sender to constructor

* clippy

* better comments

* more clippy

* more clippy

* add slot poh timing point macro

* clippy

* assert in test

* comments and display fmt

* fix check

* assert format

* revise comments

* refactor

* extrac send fn

* revert reporting_poh_timing_point

* align loggin

* small refactor

* move type declaration to the top of the module

* replace macro with constructor

* clippy: remove redundant closure

* review comments

* simplify poh timing point creation

Co-authored-by: Haoran Yi <hyi@Haorans-MacBook-Air.local>
2022-03-30 09:04:49 -05:00
HaoranYi ac8b662413
reduce metric write log level (#23966) 2022-03-29 12:00:42 -05:00
behzad nouri 78089941ff
adds validator version to set_panic_hook (#23082) 2022-02-11 18:04:10 +00:00
Michael Vines 6d5bbca630 Pacify clippy 2022-01-21 19:12:57 -08:00
Jeff Biseda 8b66625c95
convert std::sync::mpsc to crossbeam_channel (#22264) 2022-01-11 02:44:46 -08:00
Michael Vines b8837c04ec Reformat imports to a consistent style for imports
rustfmt.toml configuration:
  imports_granularity = "One"
  group_imports = "One"
2021-12-03 09:19:13 -08:00
behzad nouri cd87525f54
uses nanos precision for timestamp when submitting metrics to influxdb (#20623)
Current datapoint_info! is apparently overwriting itself when run inside
a loop. For example in
https://github.com/solana-labs/solana/blob/005d6863f/core/src/window_service.rs#L101-L107
only one of the slots will show up in influxdb.

This is apparently because of metrics code using milliseconds as the
timestamp, as mentioned here:
https://github.com/solana-labs/solana/issues/19789#issuecomment-922482013
2021-10-13 13:58:02 +00:00
Pavel Strakhov 65227f44dc
Optimize RPC pubsub for multiple clients with the same subscription (#18943)
* reimplement rpc pubsub with a broadcast queue

* update tests for new pubsub implementation

* fix: fix review suggestions

* chore(rpc): add additional pubsub metrics

* integrate max subscriptions check into SubscriptionTracker to reduce locking

* separate subscription control from tracker

* limit memory usage of items in pubsub broadcast queue, improve error handling

* add more pubsub metrics

* add final count metrics to pubsub

* add metric for total number of subscriptions

* fix small review suggestions

* remove by_params from SubscriptionTracker and add node_progress_watchers map instead

* add subscription tracker tests

* add metrics for number of pubsub notifications as a counter

* ignore clippy lint in TokenCounter

* fix underflow in token counter

* reduce queue capacity in pubsub tests

* fix(rpc): fix test timeouts

* fix race in account subscription test

* Add RpcSubscriptions::new_for_tests

Co-authored-by: Pavel Strakhov <p.strakhov@iconic.vc>
Co-authored-by: Nikita Podoliako <n.podoliako@zubr.io>
Co-authored-by: Tyera Eulberg <tyera@solana.com>
2021-09-17 13:40:14 -06:00
Alexander Meißner 6514096a67 chore: cargo +nightly clippy --fix -Z unstable-options 2021-06-18 10:42:46 -07:00
Trent Nelson 5cc073420a metrics: Don't unwrap client instantiation errors 2021-06-16 18:32:17 +00:00
Ruud van Asseldonk 3f92abedd5
Make metrics tests independent of RUST_LOG env var (#16710)
Previously, running the tests with RUST_LOG=none would fail, because the
env logger would set its filter level to reject all log messages, and
incrementing a counter only happens if the global logger has at least
the specified log level. Having the tests behave differently when
RUST_LOG is set is surprising, they should be self-contained.

Nix' buildRustPackage sets RUST_LOG="" to make the build logs less
verbose. I have trouble packaging Solana for Nix because of this, and I
believe making the tests independent of the environment is a good
solution for this.
2021-04-21 18:14:49 -06:00
Trent Nelson 7f7370c306 Re-allow clippy::integer_arithmetic at crate-level 2021-02-17 13:55:08 -07:00
Tom Parker-Shemilt 01230a0105
Remove serial_test_derive dependency (#14891) 2021-01-28 22:35:31 -07:00
Michael Vines cbffab7850 Upgrade to Rust v1.49.0 2021-01-23 19:16:36 -08:00
Kristofer Peterson e23340d89e
Clippy cleanup for all targets and nighly rust (also support 1.44.0) (#10445)
* address warnings from 'rustup run beta cargo clippy --workspace'

minor refactoring in:
- cli/src/cli.rs
- cli/src/offline/blockhash_query.rs
- logger/src/lib.rs
- runtime/src/accounts_db.rs

expect some performance improvement AccountsDB::clean_accounts()

* address warnings from 'rustup run beta cargo clippy --workspace --tests'

* address warnings from 'rustup run nightly cargo clippy --workspace --all-targets'

* rustfmt

* fix warning stragglers

* properly fix clippy warnings test_vote_subscribe()
replace ref-to-arc with ref parameters where arc not cloned

* Remove lock around JsonRpcRequestProcessor (#10417)

automerge

* make ancestors parameter optional to avoid forcing construction of empty hash maps

Co-authored-by: Greg Fitzgerald <greg@solana.com>
2020-06-09 09:38:14 +09:00
Kristofer Peterson 58ef02f02b
9951 clippy errors in the test suite (#10030)
automerge
2020-05-15 09:35:43 -07:00
Michael Vines 3acfe42622
Exit cleanly on panic! so the process don't limp along in a half-dead state (#9690) 2020-04-23 12:05:13 -07:00
Michael Vines f142451a33 Reduce metrics log output 2020-04-17 23:38:14 -07:00
sakridge 7079559c2d
Fix windows build by removing sys-info (#8860)
Doesn't build for windows.
2020-03-16 12:53:13 -07:00
Michael Vines f4622d67e9 Submit all metrics in one HTTP POST rather than a HTTP POST per level 2020-02-20 18:12:30 -07:00
anatoly yakovenko 17fb8258e5
Datapoints overwhelm the metrics queue and blow up ram usage. (#8272)
automerge
2020-02-14 11:11:55 -08:00
dependabot-preview[bot] 57858b8015 Bump reqwest from 0.9.24 to 0.10.0 (#7642)
* Bump reqwest from 0.9.24 to 0.10.0

Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.9.24 to 0.10.0.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/commits)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Make reqwest::blocking specific

Co-authored-by: Tyera Eulberg <teulberg@gmail.com>
2020-01-08 13:31:43 -07:00
Rob Walker a7040896f0
Update to rust 1.40.0 (#7572)
* Update to rust 1.40.0

* fixups
2019-12-19 23:27:54 -08:00
Michael Vines 9c00ad9ff2
Remove some low-hanging TODOs (#6839) 2019-11-08 16:41:36 -07:00
sakridge 5ef012b2c1
Tweak debug to remove unreadable datapoints (#6060) 2019-10-04 16:25:22 -07:00
Michael Vines c1c7e0ff08
Remove reduntant semicolon 2019-10-03 16:25:00 -07:00
Tyera Eulberg 7babfd00c1
Revert back to reqwest, using rustls feature (#6041)
* Revert back to reqwest, using rustls feature

* Cargo.lock and crate-features

* Ignore test
2019-09-24 14:10:59 -06:00
Michael Vines 3d44cffcda
Beautify metrics datapoint logging (#5998) 2019-09-20 12:00:43 -07:00
Michael Vines a6479eb6e9
Data points are now logged according to their level, instead of always debug! (#5906)
Note that Counters remain at debug! to avoid excessive default logging
2019-09-14 08:52:09 -07:00
Michael Vines df205f8752
Use ureq instead of influx_db_client (#5839) 2019-09-07 12:48:45 -07:00
Tyera Eulberg b19d9a50d3
Transition to ureq http client (#5777)
* Transition to ureq http client

* Remove unwrap
2019-09-04 12:11:44 -07:00
Michael Vines 6089c8030b Validator/replicator metrics host id is no longer set by bash (#5755)
automerge
2019-08-30 15:33:30 -07:00
Michael Vines f6b63a7dbc
Decode SOLANA_METRICS_CONFIG instead of relying on some bash to do it (#5633) 2019-08-23 21:17:10 -07:00
Greg Fitzgerald 471bc73a23
Fix Rust 1.37.0 compiler warnings (#5530)
Looks like most usages of trait objects should have introduced
a type variable instead.
2019-08-15 14:00:09 -06:00
Justin Starry 0fde19239b
Rate limit counter metrics points to one per second (#5496)
* Rate limit counter metrics points to one per second

* Remove old env var

* Test that metrics counter is incrementing

* Fix typo
2019-08-12 18:15:34 -04:00
Justin Starry 4f86c0b74a
Rate limit transaction counters (#5447)
* Rate limit transaction counters

* @sakridge feedback

* Set default high metrics rate for multinode demo

* Fix tests

* Swap defaults and fix env var tests

* Only set metrics rate if not already set
2019-08-08 17:05:06 -04:00
Justin Starry 3ab1b46ef7
Fix vote metrics (#5377) 2019-08-01 09:11:49 -04:00
Justin Starry 84368697af
Fix metrics when leader does not report metrics (#5291) 2019-07-30 16:18:33 -04:00