Comparing Protobuf, JSON, BSON, XML with .NET for File streams

This post compares the Serialization to files using Protobuf, Json.NET (Newtonsoft) JSON and BSON, Servicestack Json and plain .NET Xml. The test measures the whole operation and not just serialization to a memory stream.

Code: https://github.com/damienbod/SerializationToFileTests

For Serialization, the tests use an Object class with int, double, string, Object properties and also a List of Objects. The tests can set the amount of child objects. The tests are repeated for different sizes. The results remain consistent for file of all sizes up to 100MB. The results will always be different for different hardware, operating systems be what remains the same is the relative difference between the serializers.

Test program class:

using System;
using System.IO;
using System.Text;
using SerializationToFiles.Serializers;
using FileAccess = SerializationToFiles.Serializers.FileAccess;

namespace SerializationToFiles
{
    internal class Program
    {
        private static int _counter = 1;
        private static void Main(string[] args)
        {
            const int repeatTestNTimes = 10;
            DoTestForNObjects(50, repeatTestNTimes);
            DoTestForNObjects(500, repeatTestNTimes);
            DoTestForNObjects(5000, repeatTestNTimes);
            DoTestForNObjects(50000, repeatTestNTimes);
            //Console.ReadKey();
        }

        static void DoTestForNObjects(int amount, int repeatTest)
        {
            _counter = 1;
            FileAccess.ExcelResultsReadAndWrite = new StringBuilder();
            FileAccess.ExcelResultsRead = new StringBuilder();
            FileAccess.ExcelResultsWrite = new StringBuilder();
            FileAccess.ExcelResultsSize = new StringBuilder();

            FileAccess.AmountOfChildObjects = amount;
            FileAccess.ExcelResultsReadAndWrite.Append("Protobuf, Newtonsoft Json,Newtonsoft Bson, ServiceStack Json, .Net Xml" + Environment.NewLine);
            FileAccess.ExcelResultsRead.Append("Protobuf, Newtonsoft Json,Newtonsoft Bson, ServiceStack Json, .Net Xml" + Environment.NewLine);
            FileAccess.ExcelResultsWrite.Append("Protobuf, Newtonsoft Json,Newtonsoft Bson, ServiceStack Json, .Net Xml" + Environment.NewLine);
            FileAccess.ExcelResultsSize.Append("Protobuf, Newtonsoft Json,Newtonsoft Bson, ServiceStack Json, .Net Xml" + Environment.NewLine);

            while (_counter < repeatTest)
            {
                Console.WriteLine("Test {0}", _counter++);
                FileAccessProtobuf.WriteReadProtobufFile("./fileProtobuf.txt");
                FileAccessNewtonsoftJson.WriteReadNewtonsoftFileJson("./jsonNewtonsoft.txt");
                FileAccessNewtonsoftBson.WriteReadNewtonsoftFileBson("./bsonNewtonsoft.txt");
                FileAccessServiceStackJson.WriteReadServiceStackJson("./serviceStackJson.txt");
                FileAccessDotNetXml.WriteReadDotNetXml("./dotnetXml.txt");

                FileAccess.ExcelResultsReadAndWrite.Append(Environment.NewLine);
                FileAccess.ExcelResultsRead.Append(Environment.NewLine);
                FileAccess.ExcelResultsWrite.Append(Environment.NewLine);
                FileAccess.ExcelResultsSize.Append(Environment.NewLine);
                //Console.ReadKey();
            }

            WriteCsv("Results/ExcelResultsReadAndWrite" +amount + ".txt", FileAccess.ExcelResultsReadAndWrite);
            WriteCsv("Results/ExcelResultsRead" + amount + ".txt", FileAccess.ExcelResultsRead);
            WriteCsv("Results/ExcelResultsWrite" + amount + ".txt", FileAccess.ExcelResultsWrite);
            WriteCsv("Results/ExcelResultsSize" + amount + ".txt", FileAccess.ExcelResultsSize);
        }
        private static void WriteCsv(string path, StringBuilder results)
        {
            DirectoryInfo dir = new DirectoryInfo("Results");
            if (!dir.Exists)
            {
                dir.Create();
            }  

            File.WriteAllText(path, results.ToString());
        }
    }
}

The protobuf read and write methods are shown underneath. All File serializers are implemented similar. The read does not use the result object. This is not required for the test.

private static long WriteProtobuf(string path)
{
 using (Stream file = File.Create(path))
 {
     Serializer.Serialize(file, GetTestObjects());
     file.Close();
 }

  return new FileInfo(path).Length;
}

private static void ReadProtobuf(string path)
{
   SimpleTransferProtobuf simpleTransferProtobuf;
   using (Stream file = File.OpenRead(path))
   {
      simpleTransferProtobuf = Serializer.Deserialize<SimpleTransferProtobuf>(file);
   }
}

Results:
The tests are saved to txt files in a csv format. These files were then imported into an excel and the average was calculated for each serializer.

Protobuf has the best read performance and Servicestack the worst. It is surprising to see that plain XML serialization is the second fastest in these tests.

The following diagram displays the write times in milliseconds for the same object using the different serialization techniques. ServiceStack is the fastest of the JSON types. Again protobuf is the best.

The following diagrams displays the total read and write time together. Again Protobuf is the best.

This diagram displays the size of the resulting file for the same object. Protobuf is the smallest. The Xml file is the largest. The difference between the JSON serializers and Protobuf is not so large.

Conclusion
Protobuf is by far the best of all the serializers in terms of speed and size. If you require a readable format, depending on your requirements you can choose any of the other ones. If size is not important, the plain .NET XML serializer is the best after Protobuf.

When serializing small files, the first read, write takes much longer than the following read/writes. This could be relevant when designing your system or you need to optimize for speed etc.

Links:

http://code.google.com/p/protobuf-net/

http://stackoverflow.com/questions/2892082/are-there-any-tutorials-for-protobuf-net

http://www.codeproject.com/Articles/642677/Protobuf-net-the-unofficial-manual

http://marcgravell.blogspot.ca/

http://protobuffers.codeplex.com/

https://groups.google.com/forum/#!topic/protobuf/ZiD4G6kiELY

http://code.google.com/p/protobuf-csharp-port/

http://stackoverflow.com/questions/2000933/protocol-buffers-versus-json-or-bson

Tags: BSON, IPC, JSON, JSON.NET, protobuf, Serialization, XML