# Benchmark

The Benchmark provides the ability to snapshot a collection of test results to associate with an Ensemble, and to assert future test runs provide the same results.

# Object

benchmark = hypervector.Benchmark.get("BENCHMARK_UUID")

After edits, you can ensure a local Benchmark is up-to-date with:

benchmark.refresh()

Endpoint

GET /v1/benchmark/<benchmark_uuid>

# Overview

{
  "benchmark_uuid": "9682bd54-0d0c-499b-9939-1c91ca93aca0",
  "added": "Mon, 08 Mar 2021 22:37:01 GMT",
  "ensemble_uuid": "0f3959db-9c42-4ca2-8c0f-a2064662c1f2",
  "definition_uuid": "f9bd49e3-8b2f-487d-848f-3855da2df162",
  "history": [
    {
      "assertion_uuid": "1ccfedf4-411a-4391-82d5-d97cdbe7cf6d",
      "timestamp": "Mon, 08 Mar 2021 22:37:06 GMT",
      "asserted": false
    },
    {
      "assertion_uuid": "5e0e128e-17ab-4279-b4a8-ca0bb66e1a8f",
      "timestamp": "Mon, 08 Mar 2021 22:37:04 GMT",
      "asserted": true
    }
  ]
}

# Attributes

  • benchmark_uuid: Identifier of Benchmark
  • added: Datetime when Benchmark was created
  • ensemble_uuid: Identifier of Ensemble used to create Benchmark
  • definition_uuid: Identifier of Definition used to create Benchmark
  • history: List of Assertions results made on Benchmark

# Methods

# List Benchmarks

hypervector.Benchmark.list(ensemble_uuid="ENSEMBLE_UUID")

Endpoint

GET /v1/definition/<definition_uuid>/ensemble/<ensemble_uuid>/benchmarks

Retrieves list of all Benchmarks associated with a parent Ensemble

# New Benchmark

hypervector.Benchmark.new(
    ensemble=ensemble,
    expected_output=[...]
)

Endpoint

POST /v1/definition/<definition_uuid>/ensemble/<ensemble_uuid>/benchmarks/add

Creates a new Benchmark, associated with a parent Ensemble

# Parameters

  • expected_output: List containing test output to set Benchmark with. This must be the same length as the parent Ensemble's size

# Assert Benchmark

benchmark.assert_benchmark(
    output_to_assert=[...]
)

Endpoint

POST /v1/benchmark/<benchmark_uuid>/assert

Asserts test results against an existing Benchmark

# Parameters

  • output_to_assert: List containing test output to assert against existing Benchmark (values in the expected_output parameter of the New Benchmark endpoint). This must be the same length as the parent Ensemble's size.

# Delete Benchmark

hypervector.Benchmark.delete("BENCHMARK_UUID")

Endpoint

DELETE /v1/benchmark/<benchmark_uuid>/delete

Deletes an existing Benchmark