# Detect regressions

The Benchmark resource allows you to snapshot expected behaviours from the component under test, and use this in future to ensure regressions or anomalies are detected.

# Creating a Benchmark

Any Ensemble can be used to benchmark an expected output. Given a function function_under_test:

ensemble_uuid = "YOUR_ENSEMBLE_UUID"
test_cases = hypervector.Ensemble.get(ensemble_uuid).hypervectors

results = []
for test_case in test_cases:
    result = function_under_test(test_case)
    results.append(result)
    
hypervector.Benchmark.new(
    ensemble_uuid=ensemble_uuid, 
    output=results
)

Benchmarks and security

Benchmarks store a hash of each of your test output results to examine equality for assertions, avoiding the need to store outputs directly

This means you can even hash each of your results before sending it back to Hypervector - the API is only concerned with measuring equality for each result

# Asserting against a Benchmark

Once a Benchmark has been created, you can then evaluate future versions of function_under_test with the same Ensemble to ensure it provides the same results.

ensemble_uuid = "YOUR_ENSEMBLE_UUID"
ensemble = hypervector.Ensemble.get(ensemble_uuid)

test_cases = ensemble.hypervectors
benchmark = ensemble.benchmarks[0]  # list of available Benchmarks

results = []
for test_case in test_cases:
    result = function_under_test(test_case)
    results.append(result)
    
benchmark_result = hypervector.Benchmark.assert_equal(results)
assert benchmark_result.asserted is True

If the assertion fails, Hypervector provides a diff of the Ensemble rows that do not provide an expected result. This allows for more targeted investigation into what might be causing the regression

{
  "assertion_uuid": "612b8237-cb27-4c77-92ba-a9913e41cc2a",
  "timestamp": "2021/04/21 10:01:47",
  "benchmark_uuid": "21696049-48a3-494d-a0b2-7d5e020ca691",
  "ensemble_uuid": "ab9f76e8-8d59-497b-bda7-0ce4fffb7c7f",
  "diff": {
    "indices": [241, 3839, 4912],
    "inputs": [
      [0.4170, 19.872, "True"], 
      [-1.072, 28.094, "False"],
      [-1.192, 21.197, "False"]
    ],
    "size": 5000
  }
}

The Dashboard (opens new window) also provides a history of Benchmark assertions, allowing for regressions to be tracked and resolved.

assertion