This tutorial describes how to run benchmarking tools for ScalarDL. Benchmarking is helpful for evaluating how a system performs against a set of standards.

Benchmark workloads

  • SmallBank
  • TPC-C (New-Order and Payment transactions only)
  • YCSB (Workloads A, C, and F)

Prerequisites

  • One of the following Java Development Kits (JDKs):
  • Gradle
  • Kelpie
    • Kelpie is a framework for performing end-to-end testing, such as system benchmarking and verification. Get the latest version from Kelpie Releases, and unzip the archive file.

Note

Currently, only JDK 8 can be used when running the benchmarking tools.

Set up your environment

The benchmarking tools require the following:

  • A client to execute benchmarking
  • A target Ledger server
  • A target Auditor server (optional)

Set up the above components, and then configure the properties for client, Ledger, and Auditor (optional) according to the following getting started guides:

Note

You don’t need to download the client SDK and manually register your certificate. As described later in this tutorial, the benchmarking tools will automatically register the required certificate and contracts.

Set up the benchmarking tools

The following sections describe how to set up the benchmarking tools.

Clone the ScalarDL benchmarks repository

Open Terminal, then clone the ScalarDL benchmarks repository by running the following command:

$ git clone https://github.com/scalar-labs/scalardl-benchmarks

Then, go to the directory that contains the benchmarking files by running the following command:

$ cd scalardl-benchmarks

Build the tools

To build the benchmarking tools, run the following command:

$ ./gradlew shadowJar

Prepare a benchmarking configuration file

To run a benchmark, you must prepare a benchmarking configuration file. The configuration file requires at least the locations of the workload modules to run and the client configuration.

The following is an example configuration for running the TPC-C benchmark. The configurations under client_config should match the benchmarking environment that you previously set up.

Note

Alternatively, instead of specifying each client configuration item in the .toml file, you can use the ScalarDL client properties file. If config_file is specified (commented out below), all other configurations under client_config will be ignored.

[modules]
[modules.preprocessor]
name = "com.scalar.dl.benchmarks.tpcc.TpccLoader"
path = "./build/libs/scalardl-benchmarks-all.jar"
[modules.processor]
name = "com.scalar.dl.benchmarks.tpcc.TpccBench"
path = "./build/libs/scalardl-benchmarks-all.jar"
[modules.postprocessor]
name = "com.scalar.dl.benchmarks.tpcc.TpccReporter"
path = "./build/libs/scalardl-benchmarks-all.jar"

[client_config]
config_file = "/<PATH_TO>/client.properties"
#ledger_host = "localhost"
#auditor_host = "localhost"
#auditor_enabled = "true"
#cert_holder_id = "test_holder"
#certificate = "/<PATH_TO>/client.pem"
#private_key = "/<PATH_TO>/client-key.pem"

You can define parameters to pass to modules in the configuration file. For details, see the sample configuration files below and available parameters in Common parameters:

Run a benchmark

Select a benchmark, and follow the instructions to run the benchmark.

To run the SmallBank benchmark, run the following command, replacing <PATH_TO_KELPIE> with the path to the Kelpie directory:

$ /<PATH_TO_KELPIE>/bin/kelpie --config smallbank-benchmark-config.toml

To run the TPC-C benchmark, run the following command, replacing <PATH_TO_KELPIE> with the path to the Kelpie directory:

$ /<PATH_TO_KELPIE>/bin/kelpie --config tpcc-benchmark-config.toml

To run the YCSB benchmark, run the following command, replacing <PATH_TO_KELPIE> with the path to the Kelpie directory:

$ /<PATH_TO_KELPIE>/bin/kelpie --config ycsb-benchmark-config.toml

In addition, the following options are available:

  • --only-pre. Only registers certificates and contracts and loads the data.
  • --only-process. Only runs the benchmark.
  • --except-pre Runs a job without registering certificates and contracts and loading the data.
  • --except-process. Runs a job without running the benchmark.

Common parameters

Name Description Default
concurrency Number of threads for benchmarking. 1
run_for_sec Duration of benchmark (in seconds). 60
ramp_for_sec Duration of ramp-up time before benchmark (in seconds). 0

Workload-specific parameters

Select a workload to see its available parameters.

Name Description Default
num_accounts Number of bank accounts for benchmarking. 100000
load_concurrency Number of threads for loading. 1
load_batch_size Number of accounts in a single loading transaction. 1
Name Description Default
num_warehouses Number of warehouses (scale factor) for benchmarking. 1
rate_payment Percentage of Payment transaction. 50
load_concurrency Number of threads for loading. 1
Name Description Default
record_count Number of records for benchmarking. 1000
payload_size Payload size (in bytes) of each record. 1000
ops_per_tx Number of operations in a single transaction 2
workload Workload type (A, C, or F). A
load_concurrency Number of threads for loading. 1
load_batch_size Number of records in a single loading transaction. 1