fix: flaky tests due to imprecision in floating point calculation and performance test setup

fix: flaky tests due to imprecision in floating point calculation and performance test setup #1737