6.4 KiB

Raw Blame History

Performance Testing

Intro

Benchmarking the SDK is one of the key focuses of SDKMetrics API. The most important operation of the SDK is the ability to synchronize blocks in order to see transactions, right balances and to send & receive funds. In short, be in sync with the mainnet blockchain. Synchronization consists of several sub-operations like downloading the blocks, validating & scanning, etc.

Benchmarking the Synchronization

The initial work on benchmarking/performance testing allows us to properly measure times spent in sub-operations of a synchronization. There will be iterations and enhancements and ideally fully automated solution one day but for now we rely on manual approach.

SDKMetrics

SDKMetrics is the public interface on the SDK side allowing users to get the synchronization related metrics. It's a shared singleton and you can access it via

SDKMetrics.shared

By default the gathering of the data is turned of so anything reported inside the SDK is simply ignored. You need to call

SDKMetrics.shared.enableMetrics()

in order to allow SDKMetrics to actually collect the RAW data. There's a counterpart API for the enablement, called

SDKMetrics.shared.disableMetrics()

This method turns the SDKMetrics off from collecting and also flushes all the so far collected RAW data. Like we said, the SDK is automatically reporting sub-operation metrics. The reports are collected and held in memory, split by the operation:

enum Operation {
    case downloadBlocks                 // download of the blocks
    case validateBlocks                 // validation of the downloaded blocks
    case scanBlocks                     // scanning of the downloaded blocks
    case enhancement                    // enhancement of the transactions
    case fetchUTXOs                     // fetching the UTXOs
}

Every report is represented by a struct:

struct BlockMetricReport: Equatable {
    startHeight: BlockHeight            // start of the range to be processed
    progressHeight: BlockHeight         // latest processed height
    targetHeight: BlockHeight           // end of the range to be processed
    batchSize: Int                      // size of the batch to be processed
    startTime: TimeInterval             // when the operation started
    endTime: TimeInterval               // when the operation finished
    duration: TimeInterval              // computed property to provide duration
}

SDKMetrics holds the reports in a dictionary where keys are the operations. You can receive the data via either of the following methods.

// Get all reports for the specific Operation
SDKMetrics.shared.popBlock(operation: Operation, flush: Bool = false) -> [BlockMetricReport]?

// Get the whole dictionary of collected data
SDKMetrics.shared.popAllBlockReports(flush: Bool = false) -> [Operation : [BlockMetricReport]]

Notice flush to be set to false by default. Collection the data leaves it in memory but you can clear it out of the memory by setting flush to true. Such option is handy when you plan to start to collect a new set of data.

These two pop methods simply returns RAW data from the SDKMetrics so post-processing of the data is up to the caller. There are extension methods that help with the accumulation of reports and provide decent post-processing for typical use cases though.

SDKMetrics extension

Post-processing the array of reports per Operation or more general a dictionary of all arrays of reports may be time consuming, especially when you need to know just the times Operations have taken. For this specific needs we introduce CumulativeSummary struct:

struct CumulativeSummary: Equatable {
    downloadedBlocksReport: ReportSummary?
    validatedBlocksReport: ReportSummary?
    scannedBlocksReport: ReportSummary?
    enhancementReport: ReportSummary?
    fetchUTXOsReport: ReportSummary?
    totalSyncReport: ReportSummary?

where ReportSummary represents:

struct ReportSummary: Equatable {
    minTime: TimeInterval
    maxTime: TimeInterval
    avgTime: TimeInterval

As you can see CumulativeSummary basically holds min, max and avg times per Operation. To generate such summary you call:

// Get the cumulative summary of the collected data
SDKMetrics.shared.cumulativeSummary() -> CumulativeSummary

There is a specific use case we want to mention, imagine you want to run a performance test that calls synchronization several times and you want to measure all runs separately and process runs afterwards. You either call cumulativeSummary() and re-enable the metrics to flush the data out and post-process collected summaries on your own or you can take advantage of 2 more methods available:

// Generates `CumulativeSummary` and stores it in memory & clears out data
SDKMetrics.shared.cumulateReportsAndStartNewSet()

// Merges all cumulativeSummaries into one
SDKMetrics.shared.summarizedCumulativeReports() -> CumulativeSummary?

So typical use case of these methods is sketched in the following pseudo code

    SDKMetrics.shared.enableMetrics()
    
    for run in 1...X {
        // ensure fresh start of synchronization

        // synchronize

        // collect data for this run and start new set
        SDKMetrics.shared.cumulateReportsAndStartNewSet()
    }

    // collect final data as a merge of all runs
    let finalSummary = SDKMetrics.shared.summarizedCumulativeReports()

    SDKMetrics.shared.disableMetrics()

Printed out example of the finalSummary:

downloadedBlocksTimes: min: 0.002303004264831543 max: 0.9062199592590332 avg: 0.14520481750369074
validatedBlocksTimes: min: 0.01760399341583252 max: 0.019036054611206055 avg: 0.0178409144282341
scannedBlocksTimes: min: 0.045277953147888184 max: 0.5136369466781616 avg: 0.2530662305653095
enhancementTimes: min: 0.0 max: 0.0 avg: 0.0
fetchUTXOsTimes: min: 1.9073486328125e-06 max: 2.09808349609375e-05 avg: 3.166496753692627e-06
totalSyncTimes: min: 7.222689986228943 max: 10.718868017196655 avg: 8.997062936425209

Performance tests

We encourage you to visit Tests/PerformanceTests/ folder and check SynchronizerTests where we do exactly what is mentioned in this doc. We run synchronization for specific range of 100 blocks 5 times, measure every run separately and merge results together in the end. The SDKMetrics and SynchronizerTests lay down foundations for the future automatization of performance testing.

6.4 KiB Raw Blame History