Skip to main content
Version: 3.12
Features and pricing

ScalarDL Benchmarking Tools

This tutorial describes how to run benchmarking tools for ScalarDL. Benchmarking is helpful for evaluating how a system performs against a set of standards.

Benchmark workloads​

  • SmallBank
  • TPC-C (New-Order and Payment transactions only)
  • YCSB (Workloads A, C, and F)

Prerequisites​

  • One of the following Java Development Kits (JDKs):
  • Gradle
  • Kelpie
    • Kelpie is a framework for performing end-to-end testing, such as system benchmarking and verification. Get the latest version from Kelpie Releases, and unzip the archive file.
note

Currently, only JDK 8 can be used when running the benchmarking tools.

Set up your environment​

The benchmarking tools require the following:

  • A client to execute benchmarking
  • A target Ledger server
  • A target Auditor server (optional)

Set up the above components, and then configure the properties for client, Ledger, and Auditor (optional) according to the following guides:

note

You don't need to download the client SDK and manually register your certificate. As described later in this tutorial, the benchmarking tools will automatically register the required certificate and contracts.

Set up the benchmarking tools​

The following sections describe how to set up the benchmarking tools.

Clone the ScalarDL benchmarks repository​

Open Terminal, then clone the ScalarDL benchmarks repository by running the following command:

git clone https://github.com/scalar-labs/scalardl-benchmarks

Then, go to the directory that contains the benchmarking files by running the following command:

cd scalardl-benchmarks

Change the client SDK version​

Check the build.gradle file to see if the ScalarDL Java Client SDK version is a supported version for the target ScalarDL Ledger and Auditor, based on ScalarDL compatibility with client SDKs. If it is not supported, change <VERSION> in the following part of the build.gradle file to a version like X.Y.Z (for example, 3.11.0).

dependencies {
    implementation group: 'com.google.inject', name: 'guice', version: '5.0.1'
    implementation group: 'com.scalar-labs', name: 'kelpie', version: '1.2.3'
    implementation group: 'com.scalar-labs', name: 'scalardl-java-client-sdk', version: '<VERSION>'
    implementation group: 'io.github.resilience4j', name: 'resilience4j-retry', version: '1.3.1'
}

Build the tools​

To build the benchmarking tools, run the following command:

./gradlew shadowJar

Prepare a benchmarking configuration file​

To run a benchmark, you must prepare a benchmarking configuration file. The configuration file requires at least the locations of the workload modules to run and the client configuration.

The following is an example configuration for running the TPC-C benchmark. The configurations under client_config should match the benchmarking environment that you previously set up.

note

Alternatively, instead of specifying each client configuration item in the .toml file, you can use the ScalarDL client properties file. If config_file is specified (commented out below), all other configurations under client_config will be ignored.

[modules]
[modules.preprocessor]
name = "com.scalar.dl.benchmarks.tpcc.TpccLoader"
path = "./build/libs/scalardl-benchmarks-all.jar"
[modules.processor]
name = "com.scalar.dl.benchmarks.tpcc.TpccBench"
path = "./build/libs/scalardl-benchmarks-all.jar"
[modules.postprocessor]
name = "com.scalar.dl.benchmarks.tpcc.TpccReporter"
path = "./build/libs/scalardl-benchmarks-all.jar"

[client_config]
config_file = "/<PATH_TO>/client.properties"
#ledger_host = "localhost"
#auditor_host = "localhost"
#auditor_enabled = "true"
#cert_holder_id = "test_holder"
#certificate = "/<PATH_TO>/client.pem"
#private_key = "/<PATH_TO>/client-key.pem"

You can define parameters to pass to modules in the configuration file. For details, see the sample configuration files below and available parameters in Common parameters:

Run a benchmark​

Select a benchmark, and follow the instructions to run the benchmark.

To run the SmallBank benchmark, run the following command, replacing <PATH_TO_KELPIE> with the path to the Kelpie directory:

/<PATH_TO_KELPIE>/bin/kelpie --config smallbank-benchmark-config.toml

In addition, the following options are available:

  • --only-pre. Only registers certificates and contracts and loads the data.
  • --except-pre Runs a job without registering certificates and contracts and loading the data.

You can run the benchmark several times by using the --except-pre option after the initialization is done by using the --only-pre option.

Common parameters​

concurrency​

  • Description: Number of worker threads that concurrently execute benchmark transactions against the database. This parameter controls the level of parallelism during the actual benchmark execution phase. Increasing this value simulates more concurrent client accesses and higher workload intensity.
  • Default value: 1

run_for_sec​

  • Description: Duration of the benchmark execution phase (in seconds). This parameter defines how long the benchmark will run and submit transactions to the database.
  • Default value: 60

ramp_for_sec​

  • Description: Duration of the ramp-up period before the benchmark measurement phase begins (in seconds). During this warm-up period, the system executes transactions but does not record performance metrics. This allows the system to reach a steady state before collecting benchmark results.
  • Default value: 0

Workload-specific parameters​

Select a workload to see its available parameters.

num_accounts​

  • Description: Number of bank accounts to create for the benchmark workload. This parameter determines the size of the dataset and affects the working-set size.
  • Default value: 100000

load_concurrency​

  • Description: Number of parallel threads used to load initial benchmark data into the database. This parameter controls how fast the data-loading phase completes. Increasing this value can significantly reduce data-loading time for large datasets. This is separate from the concurrency parameter used during benchmark execution.
  • Default value: 1

load_batch_size​

  • Description: Number of accounts to insert within a single transaction during the initial data-loading phase. Larger batch sizes can improve loading performance by reducing the number of transactions, but may increase the execution time of each transaction.
  • Default value: 1