Skip to main content
Version: 3.8

ScalarDL Benchmarking Tools

This tutorial describes how to run benchmarking tools for ScalarDL. Benchmarking is helpful for evaluating how a system performs against a set of standards.

Benchmark workloads

  • SmallBank
  • TPC-C (New-Order and Payment transactions only)
  • YCSB (Workloads A, C, and F)

Prerequisites

  • One of the following Java Development Kits (JDKs):
  • Gradle
  • Kelpie
    • Kelpie is a framework for performing end-to-end testing, such as system benchmarking and verification. Get the latest version from Kelpie Releases, and unzip the archive file.
note

Currently, only JDK 8 can be used when running the benchmarking tools.

Set up your environment

The benchmarking tools require the following:

  • A client to execute benchmarking
  • A target Ledger server
  • A target Auditor server (optional)

Set up the above components, and then configure the properties for client, Ledger, and Auditor (optional) according to the following getting started guides:

note

You don't need to download the client SDK and manually register your certificate. As described later in this tutorial, the benchmarking tools will automatically register the required certificate and contracts.

Set up the benchmarking tools

The following sections describe how to set up the benchmarking tools.

Clone the ScalarDL benchmarks repository

Open Terminal, then clone the ScalarDL benchmarks repository by running the following command:

git clone https://github.com/scalar-labs/scalardl-benchmarks

Then, go to the directory that contains the benchmarking files by running the following command:

cd scalardl-benchmarks

Build the tools

To build the benchmarking tools, run the following command:

./gradlew shadowJar

Prepare a benchmarking configuration file

To run a benchmark, you must prepare a benchmarking configuration file. The configuration file requires at least the locations of the workload modules to run and the client configuration.

The following is an example configuration for running the TPC-C benchmark. The configurations under client_config should match the benchmarking environment that you previously set up.

note

Alternatively, instead of specifying each client configuration item in the .toml file, you can use the ScalarDL client properties file. If config_file is specified (commented out below), all other configurations under client_config will be ignored.

[modules]
[modules.preprocessor]
name = "com.scalar.dl.benchmarks.tpcc.TpccLoader"
path = "./build/libs/scalardl-benchmarks-all.jar"
[modules.processor]
name = "com.scalar.dl.benchmarks.tpcc.TpccBench"
path = "./build/libs/scalardl-benchmarks-all.jar"
[modules.postprocessor]
name = "com.scalar.dl.benchmarks.tpcc.TpccReporter"
path = "./build/libs/scalardl-benchmarks-all.jar"

[client_config]
config_file = "/<PATH_TO>/client.properties"
#ledger_host = "localhost"
#auditor_host = "localhost"
#auditor_enabled = "true"
#cert_holder_id = "test_holder"
#certificate = "/<PATH_TO>/client.pem"
#private_key = "/<PATH_TO>/client-key.pem"

You can define parameters to pass to modules in the configuration file. For details, see the sample configuration files below and available parameters in Common parameters:

Run a benchmark

Select a benchmark, and follow the instructions to run the benchmark.

To run the SmallBank benchmark, run the following command, replacing <PATH_TO_KELPIE> with the path to the Kelpie directory:

/<PATH_TO_KELPIE>/bin/kelpie --config smallbank-benchmark-config.toml

In addition, the following options are available:

  • --only-pre. Only registers certificates and contracts and loads the data.
  • --only-process. Only runs the benchmark.
  • --except-pre Runs a job without registering certificates and contracts and loading the data.
  • --except-process. Runs a job without running the benchmark.

Common parameters

NameDescriptionDefault
concurrencyNumber of threads for benchmarking.1
run_for_secDuration of benchmark (in seconds).60
ramp_for_secDuration of ramp-up time before benchmark (in seconds).0

Workload-specific parameters

Select a workload to see its available parameters.

NameDescriptionDefault
num_accountsNumber of bank accounts for benchmarking.100000
load_concurrencyNumber of threads for loading.1
load_batch_sizeNumber of accounts in a single loading transaction.1