diff --git a/performance.md b/performance.md new file mode 100644 index 0000000..2c86bf7 --- /dev/null +++ b/performance.md @@ -0,0 +1,183 @@ +# FROST Performance + + +## What is FROST? + +FROST is a threshold Schnorr signature scheme +[invented](https://eprint.iacr.org/2020/852) by Chelsea Komlo (researcher at the +Zcash Foundation) and Ian Goldberg, and in the process of becoming an [IETF +RFC](https://datatracker.ietf.org/doc/draft-irtf-cfrg-frost/). Threshold +signatures allows a private key being split into shares given to multiple +participants, allowing a subgroup of them (e.g. 3 out of 5, or whatever +threshold specified at key generation) to generate a signature that can be +verified by the group public key, as if it were signed by the original unsplit +private key. It has many applications such as allowing multiple entities to +manage a cryptocurrency wallet in a safer and more resilient manner. + +Currently, the RFC is close to completion, and we also completed [a Rust +implementation](https://github.com/ZcashFoundation/frost/) of all ciphersuites +specified in the RFC, and are now doing final cleanups and improvements prior to +the first release of the crates (which will soon be audited). + + +## Benchmarking and investigating the Aggregate step + +When we presented FROST at Zcon 3, we were asked how FROST performed in larger +settings, such as a 667-of-1000 signers. (This is motivated by a mechanism +proposed by Christopher Goes for [bridging Zcash with other ecosystems using +FROST](https://forum.zcashcommunity.com/t/proposed-architecture-for-a-zcash-namada-ibc-ecosystem-ethereum-ecosystem-non-custodial-bridge-using-frost-multisignatures/42749).) +We set out to benchmark our Rust implementation, and I was a bit surprised about +one particular step, “Aggregate”. + +The FROST scheme can be split into steps. The first one is Key Generation, which +only needs to be done once, while the rest are carried out each time the group +wishes to generate a new signature. In Round 1, the participant generates +commitments which are broadcast to all other participants via a Coordinator. In +Round 2, using these commitments and their respective key shares generated +during Key Generation, they produce a signature share which is sent to the +Coordinator. Finally, the Coordinator carries out the final step, Aggregate, +which produces the final signatures from all the signatures shares received. + +The benchmark for the Ristretto255 suite looked like the following. (Benchmarks +were run on an AMD Ryzen 9 5900X 3.7GHZ, Ubuntu 22.04, Rust 1.66.0.) + +![](times-by-size-and-function-ristretto255-all-shares.png) + +(Note that Round 1 and 2 timings in this post refer to per-signer timings, while +Key Generation and Aggregate are performed by the Coordinator.) + +It was expected that the timings would increase with the larger number of +participants (with the exception of Round 1, which does not depend on that +number), but the Aggregate timings appeared too high, surpassing 400ms for the +667-of-1000 case (which may not seem much but it’s unusual for a signing +procedure). + +I intended to investigate this but I didn’t even need to. Coincidentally, while +the RFC was in the last call for feedback, Tim Ruffing [pointed +out](https://mailarchive.ietf.org/arch/msg/cfrg/QQhyjvvcoaqLslaX3gWwABqHN-s/) +that Aggregate can be sped up significantly. Originally, it was specified that +each share received from the participants should be verified (each signature +share can be verified separately to ensure it is correct) and then aggregated. +Tim’s observation is that the shares can be simply aggregated and the final +signature verified with the group public key. If the verification fails, then +it’s possible to find which participant generated an incorrect share by +verifying them one by one (if desired). This greatly speeds up the case where +all shares are correct, which should be the most common. + +This is how the Ristretto255 timings look like with that optimization +implemented: + +![](times-by-size-and-function-ristretto255-aggregated.png) + +Now the Aggregate performance is very similar to the Round 2 step, which makes +sense since they have a very similar structure. + +Here’s the Aggregate performance comparison for all ciphersuites, in three +different scenarios: + + +![](verify-aggregated-vs-all-shares-10.png) + +![](verify-aggregated-vs-all-shares-100.png) + +![](verify-aggregated-vs-all-shares-1000.png) + + +## Examining overall performance + +With the benchmark machinery in place (we used +[criterion.rs](https://github.com/bheisler/criterion.rs)) we can provide +benchmark results for all supported ciphersuites in different scenarios. These +all use the optimization described above. + +![](times-by-ciphersuite-and-function-10.png) + +![](times-by-ciphersuite-and-function-100.png) + +![](times-by-ciphersuite-and-function-1000.png) + +The same data in table format: + + +### ed448 + +| | Key Generation with Dealer | Round 1 | Round 2 | Aggregate | +| :---------- | -------------------------: | ------: | ------: | --------: | +| 2-of-3 | 1.56 | 0.51 | 0.75 | 1.39 | +| 7-of-10 | 4.65 | 0.53 | 2.36 | 2.88 | +| 67-of-100 | 46.05 | 0.52 | 21.04 | 20.37 | +| 667-of-1000 | 693.45 | 0.53 | 211.68 | 197.00 | + +### ristretto255 + +| | Key Generation with Dealer | Round 1 | Round 2 | Aggregate | +| :---------- | -------------------------: | ------: | ------: | --------: | +| 2-of-3 | 0.24 | 0.08 | 0.13 | 0.22 | +| 7-of-10 | 0.71 | 0.08 | 0.42 | 0.47 | +| 67-of-100 | 7.61 | 0.08 | 3.77 | 3.40 | +| 667-of-1000 | 179.43 | 0.08 | 38.32 | 32.54 | + +### p256 + +| | Key Generation with Dealer | Round 1 | Round 2 | Aggregate | +| :---------- | -------------------------: | ------: | ------: | --------: | +| 2-of-3 | 0.56 | 0.18 | 0.33 | 0.58 | +| 7-of-10 | 1.71 | 0.19 | 1.08 | 1.24 | +| 67-of-100 | 16.51 | 0.18 | 10.03 | 9.38 | +| 667-of-1000 | 206.85 | 0.19 | 97.49 | 90.82 | + +### secp256k1 + +| | Key Generation with Dealer | Round 1 | Round 2 | Aggregate | +| :---------- | -------------------------: | ------: | ------: | --------: | +| 2-of-3 | 0.26 | 0.09 | 0.15 | 0.25 | +| 7-of-10 | 0.78 | 0.09 | 0.48 | 0.52 | +| 67-of-100 | 7.50 | 0.09 | 4.41 | 3.82 | +| 667-of-1000 | 123.74 | 0.09 | 46.11 | 37.48 | + +### ed25519 + +| | Key Generation with Dealer | Round 1 | Round 2 | Aggregate | +| :---------- | -------------------------: | ------: | ------: | --------: | +| 2-of-3 | 0.24 | 0.08 | 0.12 | 0.22 | +| 7-of-10 | 0.73 | 0.08 | 0.39 | 0.45 | +| 67-of-100 | 7.70 | 0.08 | 3.64 | 3.28 | +| 667-of-1000 | 181.45 | 0.08 | 36.92 | 31.88 | + + + + +The time-consuming part of each step is elliptic curve point multiplication. +Here’s a breakdown: + +- Key Generation with Trusted Dealer: + + - One base point multiplication to derive the group public key from the group + private key; + - One base point multiplication per MIN_PARTICIPANTS to derive a commitment + for each polynomial coefficient; + - One base point multiplication per MAX_PARTICIPANTS to derive their + individual public keys. + +- Round 1: + + - Two base point multiplications to generate commitments to the pair of + nonces. + +- Round 2: + + - One point multiplication per NUM_PARTICIPANTS to compute the group + commitment. + +- Aggregate: + + - One point multiplication per NUM_PARTICIPANTS to compute the group + commitment. If the Coordinator is also a participant, they could reuse the + value from Round 2, but we didn’t assume that in our benchmark (and our + implementation does not support this for now); + - One base point multiplication and one general point multiplication to verify + the aggregated signature; + - Verifying all shares (i.e. in our original approach, or to find a corrupt + signer if the aggregated signature failed) additionally requires one base + point multiplication and two general point multiplications per + NUM_PARTICIPANTS to actually verify the share. diff --git a/plot.py b/plot.py new file mode 100644 index 0000000..9a3a0e2 --- /dev/null +++ b/plot.py @@ -0,0 +1,171 @@ +""" + +Generate the graphs for the FROST perfomance blog post. + +Install cargo-criterion: + +cargo install cargo-criterion + +Run the benchmarks with: + +(check out old code) + +cargo criterion --message-format=json 'FROST' | tee > benchmark-verify-all-shares.txt + +(check out new code) + +cargo criterion --message-format=json 'FROST' | tee > benchmark-verify-aggregate.txt + +And then run: + +python3 plot.py + +It will generate the figures (names are partially hardcoded in each functions) +and will insert/update the tables inside `performance.md` +""" + +import matplotlib.pyplot as plt +import numpy as np +import json + + +def load_data(filename): + ciphersuite_lst = [] + fn_lst = [] + size_lst = [] + data = {} + with open(filename, 'r') as f: + for line in f: + line_data = json.loads(line) + if line_data['reason'] == 'benchmark-complete': + ciphersuite, fn, size = line_data['id'].split('/') + ciphersuite = ciphersuite.replace('FROST Signing ', '') + size = int(size) + unit = line_data['typical']['unit'] + time = line_data['typical']['estimate'] + assert unit == 'ns' + if unit == 'ns': + time = time / 1e6 + if ciphersuite not in ciphersuite_lst: + ciphersuite_lst.append(ciphersuite) + if fn not in fn_lst: + fn_lst.append(fn) + if size in (2, 7, 67, 667): + size = {2: 3, 7: 10, 67: 100, 667: 1000}[size] + if size not in size_lst: + size_lst.append(size) + data.setdefault(ciphersuite, {}).setdefault(fn, {})[size] = time + return ciphersuite_lst, fn_lst, size_lst, data + + +def plot(title, filename, get_group_value, group_lst, series_lst, fmt, figsize): + x = np.arange(len(group_lst)) # the label locations + total_width = 0.8 + bar_width = total_width / len(series_lst) # the width of the bars + + fig, ax = plt.subplots(figsize=figsize) + + offsets = [-total_width / 2 + bar_width / 2 + (bar_width * i) for i in range(len(series_lst))] + rect_lst = [] + for series_idx, series in enumerate(series_lst): + values = [get_group_value(series_idx, series, group_idx, group) for group_idx, group in enumerate(group_lst)] + rect = ax.bar(x + offsets[series_idx], values, bar_width, label=series) + rect_lst.append(rect) + + # Add some text for labels, title and custom x-axis tick labels, etc. + ax.set_ylabel('Time (ms)') + ax.set_title(title) + ax.set_xticks(x, group_lst) + ax.legend() + + for rect in rect_lst: + ax.bar_label(rect, padding=3, fmt=fmt) + + fig.tight_layout() + + plt.savefig(filename) + plt.close() + + +def times_by_size_and_function(data, ciphersuite, fn_lst, size_lst, fmt, suffix): + group_lst = [str(int((size * 2 + 2) / 3)) + "-of-" + str(size) for size in size_lst] + series_lst = fn_lst + title = f'Times by number of signers and functions; {ciphersuite} ciphersuite' + filename = f'times-by-size-and-function-{ciphersuite}-{suffix}.png' + + def get_group_value(series_idx, series, group_idx, group): + return data[ciphersuite][series][size_lst[group_idx]] + + plot(title, filename, get_group_value, group_lst, series_lst, fmt, (8, 6)) + + +def times_by_ciphersuite_and_function(data, ciphersuite_lst, fn_lst, size, fmt): + ciphersuite_lst = ciphersuite_lst.copy() + ciphersuite_lst.sort(key=lambda cs: data[cs]['Aggregate'][size]) + group_lst = fn_lst + series_lst = ciphersuite_lst + min_signers = int((size * 2 + 2) / 3) + title = f'Times by ciphersuite and function; {min_signers}-of-{size}' + filename = f'times-by-ciphersuite-and-function-{size}.png' + + def get_group_value(series_idx, series, group_idx, group): + return data[series][group][size] + + plot(title, filename, get_group_value, group_lst, series_lst, fmt, (12, 6)) + + +def verify_aggregated_vs_all_shares(data_aggregated, data_all_shares, ciphersuite_lst, size, fmt): + ciphersuite_lst = ciphersuite_lst.copy() + ciphersuite_lst.sort(key=lambda cs: data_aggregated[cs]['Aggregate'][size]) + group_lst = ciphersuite_lst + series_lst = ['Verify all shares', 'Verify aggregated'] + min_signers = int((size * 2 + 2) / 3) + title = f'Time comparison for Aggregate function; {min_signers}-of-{size}' + filename = f'verify-aggregated-vs-all-shares-{size}.png' + + def get_group_value(series_idx, series, group_idx, group): + data = [data_all_shares, data_aggregated][series_idx] + return data[group]['Aggregate'][size] + + plot(title, filename, get_group_value, group_lst, series_lst, fmt, (8, 6)) + + +def generate_table(f, data, ciphersuite_lst, fn_lst, size_lst): + for ciphersuite in ciphersuite_lst: + print(f'### {ciphersuite}\n', file=f) + print('|' + '|'.join([''] + fn_lst) + '|', file=f) + print('|' + '|'.join([':---'] + ['---:'] * len(fn_lst)) + '|', file=f) + for size in size_lst: + min_signers = int((size * 2 + 2) / 3) + print('|' + '|'.join([f'{min_signers}-of-{size}'] + ['{:.2f}'.format(data[ciphersuite][fn][size]) for fn in fn_lst]) + '|', file=f) + print('', file=f) + print('', file=f) + + +if __name__ == '__main__': + ciphersuite_lst, fn_lst, size_lst, data_aggregated = load_data('benchmark-verify-aggregate.txt') + _, _, _, data_all_shares = load_data('benchmark-verify-all-shares.txt') + + import io + import re + with io.StringIO() as f: + generate_table(f, data_aggregated, ciphersuite_lst, fn_lst, size_lst) + f.seek(0) + table = f.read() + with open('performance.md') as f: + md = f.read() + md = re.sub('[^<]*', '\n' + table + '', md, count=1, flags=re.DOTALL) + with open('performance.md', 'w') as f: + f.write(md) + + size_lst = [10, 100, 1000] + times_by_size_and_function(data_all_shares, 'ristretto255', fn_lst, size_lst, '%.2f', 'all-shares') + times_by_size_and_function(data_aggregated, 'ristretto255', fn_lst, size_lst, '%.2f', 'aggregated') + + times_by_ciphersuite_and_function(data_aggregated, ciphersuite_lst, fn_lst, 10, '%.2f') + times_by_ciphersuite_and_function(data_aggregated, ciphersuite_lst, fn_lst, 100, '%.1f') + times_by_ciphersuite_and_function(data_aggregated, ciphersuite_lst, fn_lst, 1000, '%.0f') + + verify_aggregated_vs_all_shares(data_aggregated, data_all_shares, ciphersuite_lst, 10, '%.2f') + verify_aggregated_vs_all_shares(data_aggregated, data_all_shares, ciphersuite_lst, 100, '%.1f') + verify_aggregated_vs_all_shares(data_aggregated, data_all_shares, ciphersuite_lst, 1000, '%.0f') diff --git a/times-by-ciphersuite-and-function-10.png b/times-by-ciphersuite-and-function-10.png new file mode 100644 index 0000000..ab4020e Binary files /dev/null and b/times-by-ciphersuite-and-function-10.png differ diff --git a/times-by-ciphersuite-and-function-100.png b/times-by-ciphersuite-and-function-100.png new file mode 100644 index 0000000..9b263bd Binary files /dev/null and b/times-by-ciphersuite-and-function-100.png differ diff --git a/times-by-ciphersuite-and-function-1000.png b/times-by-ciphersuite-and-function-1000.png new file mode 100644 index 0000000..69d3689 Binary files /dev/null and b/times-by-ciphersuite-and-function-1000.png differ diff --git a/times-by-size-and-function-ristretto255-aggregated.png b/times-by-size-and-function-ristretto255-aggregated.png new file mode 100644 index 0000000..732357c Binary files /dev/null and b/times-by-size-and-function-ristretto255-aggregated.png differ diff --git a/times-by-size-and-function-ristretto255-all-shares.png b/times-by-size-and-function-ristretto255-all-shares.png new file mode 100644 index 0000000..652fa8f Binary files /dev/null and b/times-by-size-and-function-ristretto255-all-shares.png differ diff --git a/verify-aggregated-vs-all-shares-10.png b/verify-aggregated-vs-all-shares-10.png new file mode 100644 index 0000000..9335f47 Binary files /dev/null and b/verify-aggregated-vs-all-shares-10.png differ diff --git a/verify-aggregated-vs-all-shares-100.png b/verify-aggregated-vs-all-shares-100.png new file mode 100644 index 0000000..94f5e24 Binary files /dev/null and b/verify-aggregated-vs-all-shares-100.png differ diff --git a/verify-aggregated-vs-all-shares-1000.png b/verify-aggregated-vs-all-shares-1000.png new file mode 100644 index 0000000..94321cd Binary files /dev/null and b/verify-aggregated-vs-all-shares-1000.png differ