Benchmark plot script (#356)
* add first version of benchmark post * add benchmarks table * document plot.py * Mention cargo-criterion installation in plot.py pydoc --------- Co-authored-by: Conrado Gouvea <conradoplg@gmail.com>
|
@ -0,0 +1,183 @@
|
|||
# FROST Performance
|
||||
|
||||
|
||||
## What is FROST?
|
||||
|
||||
FROST is a threshold Schnorr signature scheme
|
||||
[invented](https://eprint.iacr.org/2020/852) by Chelsea Komlo (researcher at the
|
||||
Zcash Foundation) and Ian Goldberg, and in the process of becoming an [IETF
|
||||
RFC](https://datatracker.ietf.org/doc/draft-irtf-cfrg-frost/). Threshold
|
||||
signatures allows a private key being split into shares given to multiple
|
||||
participants, allowing a subgroup of them (e.g. 3 out of 5, or whatever
|
||||
threshold specified at key generation) to generate a signature that can be
|
||||
verified by the group public key, as if it were signed by the original unsplit
|
||||
private key. It has many applications such as allowing multiple entities to
|
||||
manage a cryptocurrency wallet in a safer and more resilient manner.
|
||||
|
||||
Currently, the RFC is close to completion, and we also completed [a Rust
|
||||
implementation](https://github.com/ZcashFoundation/frost/) of all ciphersuites
|
||||
specified in the RFC, and are now doing final cleanups and improvements prior to
|
||||
the first release of the crates (which will soon be audited).
|
||||
|
||||
|
||||
## Benchmarking and investigating the Aggregate step
|
||||
|
||||
When we presented FROST at Zcon 3, we were asked how FROST performed in larger
|
||||
settings, such as a 667-of-1000 signers. (This is motivated by a mechanism
|
||||
proposed by Christopher Goes for [bridging Zcash with other ecosystems using
|
||||
FROST](https://forum.zcashcommunity.com/t/proposed-architecture-for-a-zcash-namada-ibc-ecosystem-ethereum-ecosystem-non-custodial-bridge-using-frost-multisignatures/42749).)
|
||||
We set out to benchmark our Rust implementation, and I was a bit surprised about
|
||||
one particular step, “Aggregate”.
|
||||
|
||||
The FROST scheme can be split into steps. The first one is Key Generation, which
|
||||
only needs to be done once, while the rest are carried out each time the group
|
||||
wishes to generate a new signature. In Round 1, the participant generates
|
||||
commitments which are broadcast to all other participants via a Coordinator. In
|
||||
Round 2, using these commitments and their respective key shares generated
|
||||
during Key Generation, they produce a signature share which is sent to the
|
||||
Coordinator. Finally, the Coordinator carries out the final step, Aggregate,
|
||||
which produces the final signatures from all the signatures shares received.
|
||||
|
||||
The benchmark for the Ristretto255 suite looked like the following. (Benchmarks
|
||||
were run on an AMD Ryzen 9 5900X 3.7GHZ, Ubuntu 22.04, Rust 1.66.0.)
|
||||
|
||||
![](times-by-size-and-function-ristretto255-all-shares.png)
|
||||
|
||||
(Note that Round 1 and 2 timings in this post refer to per-signer timings, while
|
||||
Key Generation and Aggregate are performed by the Coordinator.)
|
||||
|
||||
It was expected that the timings would increase with the larger number of
|
||||
participants (with the exception of Round 1, which does not depend on that
|
||||
number), but the Aggregate timings appeared too high, surpassing 400ms for the
|
||||
667-of-1000 case (which may not seem much but it’s unusual for a signing
|
||||
procedure).
|
||||
|
||||
I intended to investigate this but I didn’t even need to. Coincidentally, while
|
||||
the RFC was in the last call for feedback, Tim Ruffing [pointed
|
||||
out](https://mailarchive.ietf.org/arch/msg/cfrg/QQhyjvvcoaqLslaX3gWwABqHN-s/)
|
||||
that Aggregate can be sped up significantly. Originally, it was specified that
|
||||
each share received from the participants should be verified (each signature
|
||||
share can be verified separately to ensure it is correct) and then aggregated.
|
||||
Tim’s observation is that the shares can be simply aggregated and the final
|
||||
signature verified with the group public key. If the verification fails, then
|
||||
it’s possible to find which participant generated an incorrect share by
|
||||
verifying them one by one (if desired). This greatly speeds up the case where
|
||||
all shares are correct, which should be the most common.
|
||||
|
||||
This is how the Ristretto255 timings look like with that optimization
|
||||
implemented:
|
||||
|
||||
![](times-by-size-and-function-ristretto255-aggregated.png)
|
||||
|
||||
Now the Aggregate performance is very similar to the Round 2 step, which makes
|
||||
sense since they have a very similar structure.
|
||||
|
||||
Here’s the Aggregate performance comparison for all ciphersuites, in three
|
||||
different scenarios:
|
||||
|
||||
|
||||
![](verify-aggregated-vs-all-shares-10.png)
|
||||
|
||||
![](verify-aggregated-vs-all-shares-100.png)
|
||||
|
||||
![](verify-aggregated-vs-all-shares-1000.png)
|
||||
|
||||
|
||||
## Examining overall performance
|
||||
|
||||
With the benchmark machinery in place (we used
|
||||
[criterion.rs](https://github.com/bheisler/criterion.rs)) we can provide
|
||||
benchmark results for all supported ciphersuites in different scenarios. These
|
||||
all use the optimization described above.
|
||||
|
||||
![](times-by-ciphersuite-and-function-10.png)
|
||||
|
||||
![](times-by-ciphersuite-and-function-100.png)
|
||||
|
||||
![](times-by-ciphersuite-and-function-1000.png)
|
||||
|
||||
The same data in table format:
|
||||
|
||||
<!-- Benchmarks -->
|
||||
### ed448
|
||||
|
||||
| | Key Generation with Dealer | Round 1 | Round 2 | Aggregate |
|
||||
| :---------- | -------------------------: | ------: | ------: | --------: |
|
||||
| 2-of-3 | 1.56 | 0.51 | 0.75 | 1.39 |
|
||||
| 7-of-10 | 4.65 | 0.53 | 2.36 | 2.88 |
|
||||
| 67-of-100 | 46.05 | 0.52 | 21.04 | 20.37 |
|
||||
| 667-of-1000 | 693.45 | 0.53 | 211.68 | 197.00 |
|
||||
|
||||
### ristretto255
|
||||
|
||||
| | Key Generation with Dealer | Round 1 | Round 2 | Aggregate |
|
||||
| :---------- | -------------------------: | ------: | ------: | --------: |
|
||||
| 2-of-3 | 0.24 | 0.08 | 0.13 | 0.22 |
|
||||
| 7-of-10 | 0.71 | 0.08 | 0.42 | 0.47 |
|
||||
| 67-of-100 | 7.61 | 0.08 | 3.77 | 3.40 |
|
||||
| 667-of-1000 | 179.43 | 0.08 | 38.32 | 32.54 |
|
||||
|
||||
### p256
|
||||
|
||||
| | Key Generation with Dealer | Round 1 | Round 2 | Aggregate |
|
||||
| :---------- | -------------------------: | ------: | ------: | --------: |
|
||||
| 2-of-3 | 0.56 | 0.18 | 0.33 | 0.58 |
|
||||
| 7-of-10 | 1.71 | 0.19 | 1.08 | 1.24 |
|
||||
| 67-of-100 | 16.51 | 0.18 | 10.03 | 9.38 |
|
||||
| 667-of-1000 | 206.85 | 0.19 | 97.49 | 90.82 |
|
||||
|
||||
### secp256k1
|
||||
|
||||
| | Key Generation with Dealer | Round 1 | Round 2 | Aggregate |
|
||||
| :---------- | -------------------------: | ------: | ------: | --------: |
|
||||
| 2-of-3 | 0.26 | 0.09 | 0.15 | 0.25 |
|
||||
| 7-of-10 | 0.78 | 0.09 | 0.48 | 0.52 |
|
||||
| 67-of-100 | 7.50 | 0.09 | 4.41 | 3.82 |
|
||||
| 667-of-1000 | 123.74 | 0.09 | 46.11 | 37.48 |
|
||||
|
||||
### ed25519
|
||||
|
||||
| | Key Generation with Dealer | Round 1 | Round 2 | Aggregate |
|
||||
| :---------- | -------------------------: | ------: | ------: | --------: |
|
||||
| 2-of-3 | 0.24 | 0.08 | 0.12 | 0.22 |
|
||||
| 7-of-10 | 0.73 | 0.08 | 0.39 | 0.45 |
|
||||
| 67-of-100 | 7.70 | 0.08 | 3.64 | 3.28 |
|
||||
| 667-of-1000 | 181.45 | 0.08 | 36.92 | 31.88 |
|
||||
|
||||
|
||||
<!-- Benchmarks -->
|
||||
|
||||
The time-consuming part of each step is elliptic curve point multiplication.
|
||||
Here’s a breakdown:
|
||||
|
||||
- Key Generation with Trusted Dealer:
|
||||
|
||||
- One base point multiplication to derive the group public key from the group
|
||||
private key;
|
||||
- One base point multiplication per MIN_PARTICIPANTS to derive a commitment
|
||||
for each polynomial coefficient;
|
||||
- One base point multiplication per MAX_PARTICIPANTS to derive their
|
||||
individual public keys.
|
||||
|
||||
- Round 1:
|
||||
|
||||
- Two base point multiplications to generate commitments to the pair of
|
||||
nonces.
|
||||
|
||||
- Round 2:
|
||||
|
||||
- One point multiplication per NUM_PARTICIPANTS to compute the group
|
||||
commitment.
|
||||
|
||||
- Aggregate:
|
||||
|
||||
- One point multiplication per NUM_PARTICIPANTS to compute the group
|
||||
commitment. If the Coordinator is also a participant, they could reuse the
|
||||
value from Round 2, but we didn’t assume that in our benchmark (and our
|
||||
implementation does not support this for now);
|
||||
- One base point multiplication and one general point multiplication to verify
|
||||
the aggregated signature;
|
||||
- Verifying all shares (i.e. in our original approach, or to find a corrupt
|
||||
signer if the aggregated signature failed) additionally requires one base
|
||||
point multiplication and two general point multiplications per
|
||||
NUM_PARTICIPANTS to actually verify the share.
|
|
@ -0,0 +1,171 @@
|
|||
"""
|
||||
|
||||
Generate the graphs for the FROST perfomance blog post.
|
||||
|
||||
Install cargo-criterion:
|
||||
|
||||
cargo install cargo-criterion
|
||||
|
||||
Run the benchmarks with:
|
||||
|
||||
(check out old code)
|
||||
|
||||
cargo criterion --message-format=json 'FROST' | tee > benchmark-verify-all-shares.txt
|
||||
|
||||
(check out new code)
|
||||
|
||||
cargo criterion --message-format=json 'FROST' | tee > benchmark-verify-aggregate.txt
|
||||
|
||||
And then run:
|
||||
|
||||
python3 plot.py
|
||||
|
||||
It will generate the figures (names are partially hardcoded in each functions)
|
||||
and will insert/update the tables inside `performance.md`
|
||||
"""
|
||||
|
||||
import matplotlib.pyplot as plt
|
||||
import numpy as np
|
||||
import json
|
||||
|
||||
|
||||
def load_data(filename):
|
||||
ciphersuite_lst = []
|
||||
fn_lst = []
|
||||
size_lst = []
|
||||
data = {}
|
||||
with open(filename, 'r') as f:
|
||||
for line in f:
|
||||
line_data = json.loads(line)
|
||||
if line_data['reason'] == 'benchmark-complete':
|
||||
ciphersuite, fn, size = line_data['id'].split('/')
|
||||
ciphersuite = ciphersuite.replace('FROST Signing ', '')
|
||||
size = int(size)
|
||||
unit = line_data['typical']['unit']
|
||||
time = line_data['typical']['estimate']
|
||||
assert unit == 'ns'
|
||||
if unit == 'ns':
|
||||
time = time / 1e6
|
||||
if ciphersuite not in ciphersuite_lst:
|
||||
ciphersuite_lst.append(ciphersuite)
|
||||
if fn not in fn_lst:
|
||||
fn_lst.append(fn)
|
||||
if size in (2, 7, 67, 667):
|
||||
size = {2: 3, 7: 10, 67: 100, 667: 1000}[size]
|
||||
if size not in size_lst:
|
||||
size_lst.append(size)
|
||||
data.setdefault(ciphersuite, {}).setdefault(fn, {})[size] = time
|
||||
return ciphersuite_lst, fn_lst, size_lst, data
|
||||
|
||||
|
||||
def plot(title, filename, get_group_value, group_lst, series_lst, fmt, figsize):
|
||||
x = np.arange(len(group_lst)) # the label locations
|
||||
total_width = 0.8
|
||||
bar_width = total_width / len(series_lst) # the width of the bars
|
||||
|
||||
fig, ax = plt.subplots(figsize=figsize)
|
||||
|
||||
offsets = [-total_width / 2 + bar_width / 2 + (bar_width * i) for i in range(len(series_lst))]
|
||||
rect_lst = []
|
||||
for series_idx, series in enumerate(series_lst):
|
||||
values = [get_group_value(series_idx, series, group_idx, group) for group_idx, group in enumerate(group_lst)]
|
||||
rect = ax.bar(x + offsets[series_idx], values, bar_width, label=series)
|
||||
rect_lst.append(rect)
|
||||
|
||||
# Add some text for labels, title and custom x-axis tick labels, etc.
|
||||
ax.set_ylabel('Time (ms)')
|
||||
ax.set_title(title)
|
||||
ax.set_xticks(x, group_lst)
|
||||
ax.legend()
|
||||
|
||||
for rect in rect_lst:
|
||||
ax.bar_label(rect, padding=3, fmt=fmt)
|
||||
|
||||
fig.tight_layout()
|
||||
|
||||
plt.savefig(filename)
|
||||
plt.close()
|
||||
|
||||
|
||||
def times_by_size_and_function(data, ciphersuite, fn_lst, size_lst, fmt, suffix):
|
||||
group_lst = [str(int((size * 2 + 2) / 3)) + "-of-" + str(size) for size in size_lst]
|
||||
series_lst = fn_lst
|
||||
title = f'Times by number of signers and functions; {ciphersuite} ciphersuite'
|
||||
filename = f'times-by-size-and-function-{ciphersuite}-{suffix}.png'
|
||||
|
||||
def get_group_value(series_idx, series, group_idx, group):
|
||||
return data[ciphersuite][series][size_lst[group_idx]]
|
||||
|
||||
plot(title, filename, get_group_value, group_lst, series_lst, fmt, (8, 6))
|
||||
|
||||
|
||||
def times_by_ciphersuite_and_function(data, ciphersuite_lst, fn_lst, size, fmt):
|
||||
ciphersuite_lst = ciphersuite_lst.copy()
|
||||
ciphersuite_lst.sort(key=lambda cs: data[cs]['Aggregate'][size])
|
||||
group_lst = fn_lst
|
||||
series_lst = ciphersuite_lst
|
||||
min_signers = int((size * 2 + 2) / 3)
|
||||
title = f'Times by ciphersuite and function; {min_signers}-of-{size}'
|
||||
filename = f'times-by-ciphersuite-and-function-{size}.png'
|
||||
|
||||
def get_group_value(series_idx, series, group_idx, group):
|
||||
return data[series][group][size]
|
||||
|
||||
plot(title, filename, get_group_value, group_lst, series_lst, fmt, (12, 6))
|
||||
|
||||
|
||||
def verify_aggregated_vs_all_shares(data_aggregated, data_all_shares, ciphersuite_lst, size, fmt):
|
||||
ciphersuite_lst = ciphersuite_lst.copy()
|
||||
ciphersuite_lst.sort(key=lambda cs: data_aggregated[cs]['Aggregate'][size])
|
||||
group_lst = ciphersuite_lst
|
||||
series_lst = ['Verify all shares', 'Verify aggregated']
|
||||
min_signers = int((size * 2 + 2) / 3)
|
||||
title = f'Time comparison for Aggregate function; {min_signers}-of-{size}'
|
||||
filename = f'verify-aggregated-vs-all-shares-{size}.png'
|
||||
|
||||
def get_group_value(series_idx, series, group_idx, group):
|
||||
data = [data_all_shares, data_aggregated][series_idx]
|
||||
return data[group]['Aggregate'][size]
|
||||
|
||||
plot(title, filename, get_group_value, group_lst, series_lst, fmt, (8, 6))
|
||||
|
||||
|
||||
def generate_table(f, data, ciphersuite_lst, fn_lst, size_lst):
|
||||
for ciphersuite in ciphersuite_lst:
|
||||
print(f'### {ciphersuite}\n', file=f)
|
||||
print('|' + '|'.join([''] + fn_lst) + '|', file=f)
|
||||
print('|' + '|'.join([':---'] + ['---:'] * len(fn_lst)) + '|', file=f)
|
||||
for size in size_lst:
|
||||
min_signers = int((size * 2 + 2) / 3)
|
||||
print('|' + '|'.join([f'{min_signers}-of-{size}'] + ['{:.2f}'.format(data[ciphersuite][fn][size]) for fn in fn_lst]) + '|', file=f)
|
||||
print('', file=f)
|
||||
print('', file=f)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
ciphersuite_lst, fn_lst, size_lst, data_aggregated = load_data('benchmark-verify-aggregate.txt')
|
||||
_, _, _, data_all_shares = load_data('benchmark-verify-all-shares.txt')
|
||||
|
||||
import io
|
||||
import re
|
||||
with io.StringIO() as f:
|
||||
generate_table(f, data_aggregated, ciphersuite_lst, fn_lst, size_lst)
|
||||
f.seek(0)
|
||||
table = f.read()
|
||||
with open('performance.md') as f:
|
||||
md = f.read()
|
||||
md = re.sub('<!-- Benchmarks -->[^<]*<!-- Benchmarks -->', '<!-- Benchmarks -->\n' + table + '<!-- Benchmarks -->', md, count=1, flags=re.DOTALL)
|
||||
with open('performance.md', 'w') as f:
|
||||
f.write(md)
|
||||
|
||||
size_lst = [10, 100, 1000]
|
||||
times_by_size_and_function(data_all_shares, 'ristretto255', fn_lst, size_lst, '%.2f', 'all-shares')
|
||||
times_by_size_and_function(data_aggregated, 'ristretto255', fn_lst, size_lst, '%.2f', 'aggregated')
|
||||
|
||||
times_by_ciphersuite_and_function(data_aggregated, ciphersuite_lst, fn_lst, 10, '%.2f')
|
||||
times_by_ciphersuite_and_function(data_aggregated, ciphersuite_lst, fn_lst, 100, '%.1f')
|
||||
times_by_ciphersuite_and_function(data_aggregated, ciphersuite_lst, fn_lst, 1000, '%.0f')
|
||||
|
||||
verify_aggregated_vs_all_shares(data_aggregated, data_all_shares, ciphersuite_lst, 10, '%.2f')
|
||||
verify_aggregated_vs_all_shares(data_aggregated, data_all_shares, ciphersuite_lst, 100, '%.1f')
|
||||
verify_aggregated_vs_all_shares(data_aggregated, data_all_shares, ciphersuite_lst, 1000, '%.0f')
|
After Width: | Height: | Size: 36 KiB |
After Width: | Height: | Size: 35 KiB |
After Width: | Height: | Size: 38 KiB |
After Width: | Height: | Size: 34 KiB |
After Width: | Height: | Size: 36 KiB |
After Width: | Height: | Size: 30 KiB |
After Width: | Height: | Size: 31 KiB |
After Width: | Height: | Size: 35 KiB |