How to Benchmark Testing your code in C#

As a software developer, some time we develop a peace of code even with unit testing and think that is world best code. Until we realize that it might perform very bad in specific condition or heavy load.

For instance lets say you develop a simple search mechanism to your application that preform very well when your application data is little, but at some point the application data grow at a level that your search solution break a part and can not deliver the expected results, as it might take long time to return the result.

Here comes Benchmark Testing!

What is Benchmark Testing?

Benchmark testing in software is a repeatable testing method to measure program performance and compare the results against a previous release or other approaches to find out the result of best approach or/and to ensure that the program performance is with in the standards over time. Benchmark testing include software, hardware and/or network performance all together. The goal for benchmark testing is to test all the current and future releases of an application to maintain high-quality standards.

How to create a simple Benchmark Testing?

It happens often that you are developing a code and you need to test its performance and see if you can improve your code performance or just to create a performance report of your code.

This can be achieved by calling the method you want to test x number of time and tick start and stop time. Take the difference between times.

It is also important to call method before the time starts as a warm to the main test.

The concept, Lets do it in C# with out framework

Now lets make a simple example that I have made in my C# console app.

Lets create class CodeToBeBenchmark and say we have 2 fancy method for hash calculation, no thing special. One to calculate MD5 hashing and one for SHA512 hashing of a random generated bytes. For those 2 methods we might know in our heads that MD5 hashing is faster than SHA512, but there could be other situation that you think some code is faster without having evidence for it.

public class CodeToBeBenchmark
{
    private const int N = 10000;
    private readonly byte[] _data;

    private readonly SHA512 _sha512 = SHA512.Create();
    private readonly MD5 _md5 = MD5.Create();

    public CodeToBeBenchmark()
    {
        _data = new byte[N];
        new Random(42).NextBytes(_data);
    }

    public byte[] Sha512() => _sha512.ComputeHash(_data);

    public byte[] Md5() => _md5.ComputeHash(_data);
}

Let’s now run the above methods once in our code, it goes so fast that it is almost impossible to measure the time taken to calculate the hash.

So what we can do is for example to run both method 1000 times each and measure the time it takes for each test and compare it. So for this we can do some thing like following code:

public void BenchmarkConcept()
{
    int tests = 1000;

    Console.WriteLine("| Method |     Mean |");
    Console.WriteLine("|------- |---------:|");

    Stopwatch stopwatch = new Stopwatch();
    //warm up
    Sha512();

    stopwatch.Start();
    for (int i = 0; i < tests; i++)
    {
        Sha512();
    }
    stopwatch.Stop();

    var result2 = (double)stopwatch.ElapsedMicroSeconds() / tests;
    Console.WriteLine($"| Sha512 | {result2} us|");

    //warm up
    Md5();
    stopwatch.Restart();

    stopwatch.Start();
    for (int i = 0; i < tests; i++)
    {
        Md5();
    }
    stopwatch.Stop();

    var result1 = (double)stopwatch.ElapsedMicroSeconds() / tests;

    Console.WriteLine($"| Md5    | {result1} us|");
}

And here is my output

| Method |     Mean |
|------- |---------:|
| Sha512 | 35,735 us|
| Md5    | 26,448 us|

As you can see Sha512 takes longer time to calculate than MD5.

The interesting part is not maybe how long it takes, but more the difference between the 2 methods.

As you can see in following real world example of benchmark performance test l have plotted a logarithmic graph to analyze how code behavior change on data growth. We can see very interesting results for, as you can see when the data size in file grow above 10000Kbyte the data structure Assign 3 is no longer efficient, as it is zoomed in and marked with Red compared to the Yellow and other data structure which is more efficient. The time showing in this graph are in millisecond. This was not possible to measure with out benchmark testing.

You can of course easily improve this simple benchmark code and use it to fit your benchmark test. One thing more to add, when benchmarking, it is important to document your hardware specification, like CPU type, speed, cores etc. hence you can not make Benchmark of same code on 2 different PCs and compare the same result, you will in many cases get different results. But again what is important is time different and when you put the results in plot and compare them.

I would those suggest using a well tested package or framework, hence it is validated and well tested for benchmark reason, and it has a lot of features like getting hardware information, generating reports etc. In my next example I will take a look at Benchmark DotNet package.

Benchmark DotNet

Benchmark DotNet nuget package is a future reach benchmark package for C# with a lot of feature and use best practices for benchmarking. It is easy to use as well.

If you have followed up on my previous article, I wrote 3 articles series on Search Engine. It is absolutely a good candidate for benchmark testing, hence I have developed different search methods, using list, dictionary and improve search results with Levenstein Algorithm. But before that I take Search Engine in benchmark test, lets take the previous example and make benchmark on it.

First of all I will install BenchmarkDotNet nuget in my test console project that I have created previously:

<ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.13.1" />
</ItemGroup>

I create a class BenchmarkDotNetExample like:

public class BenchmarkDotNetExample
{
    private readonly CodeToBeBenchmark _code = new CodeToBeBenchmark();

    [Benchmark]
    public byte[] Sha512() => _code.Sha512();

    [Benchmark]
    public byte[] Md5() => _code.Md5();

    public static void Run()
    {
        var summary = BenchmarkRunner.Run<BenchmarkDotNetExample>();
    }
}

The methods that I want to benchmark, I add [Benchmark] attribute on top of the methods. I have created a method Run that I will call in Main method.

In Run method I have s simple line of code BenchmarkRunner.Run<BenchmarkDotNetExample>(); that starts the benchmark test.

Before we run the code in our Console App, we need to change our code from Debug to Release mode, this is to make the code optimized while benchmarking.

And here is the results after some time of benchmark testing:

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22000
Intel Core i7-10875H CPU 2.30GHz, 1 CPU, 16 logical and 8 physical cores
.NET SDK=6.0.101
  [Host]     : .NET 6.0.1 (6.0.121.56705), X64 RyuJIT
  DefaultJob : .NET 6.0.1 (6.0.121.56705), X64 RyuJIT

| Method |     Mean |    Error |   StdDev |
|------- |---------:|---------:|---------:|
| Sha512 | 25.75 us | 0.068 us | 0.063 us |
|    Md5 | 19.43 us | 0.033 us | 0.031 us |

As you can see we get CPU information, .net version, OS and Benchmark results.

Now if you remember, we talked about Serach Engine test, for that I have create a testing method for my Search Engine Code.

private readonly SE1_done.SearchService _program1;
private readonly SE2_done.SearchService _program2;
private readonly SE3_done.SearchService _program3;
private readonly string path = $"./data/search-data-L.txt";

public BenchmarkSearchEngine()
{
    _program1 = new SE1_done.SearchService(path);
    _program2 = new SE2_done.SearchService(path);
    _program3 = new SE3_done.SearchService(path);
}

[Benchmark]
public void SearchForWordInList()
{
    _program1.FindWord("test");
}

[Benchmark]
public void SearchForWordInDict()
{
    _program2.FindWord("test");
}

public static Summary Run()
{
    return BenchmarkRunner.Run<BenchmarkSearchEngine>();
}

In this test I take only the large file, but I can make 3 tests, one for small file, medium file and large file. And I can also make search for words in start, middle and end of the each file. For for now I stick with my Large file.

I run the above benchmark test and get following results:

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22000
Intel Core i7-10875H CPU 2.30GHz, 1 CPU, 16 logical and 8 physical cores
.NET SDK=6.0.101
  [Host]     : .NET 6.0.1 (6.0.121.56705), X64 RyuJIT
  DefaultJob : .NET 6.0.1 (6.0.121.56705), X64 RyuJIT


|              Method |        ean |     Error |    StdDev |     Median |
|-------------------- |-----------:|----------:|----------:|-----------:|
| SearchForWordInList | 122,516 us |  4,874 us | 14,373 us | 113,098 us |
| SearchForWordInDict |      48 us |      2 us |      5 us |      44 us |

This will for instance quickly telling me that search for word in dictionary is much faster than list.

We are done!

Conclusion

So in this article we have learned how to cover Benchmark testing using native C# and a Benchmark DotNet library.

We covered 2 simple examples for native C# and for Benchmark DotNet nuget.

Finally we made a revisit to our search engine articles and and created 2 benchmark tests to compare search difference between list and dictionary.

Benchmark testing is not required on all projects, but it is a good way to tell you how your code perform today, and how does it perform tomorrow when you change your code.