C# Compact Framework - OutOfMemoryException with XmlSerializer.Serialize

Hi everyone,

I'm trying to serialize a large collection of objects (20,000) objects within the collection. I'm doing this using the following code:

XmlSerializer xs = new XmlSerializer(deserialized.GetType());
StringWriter sw;
using (sw = new StringWriter())
{
   xs.Serialize(sw, deserialized);   // OutOfMemoryException here
}

string packet = sw.ToString();
return packet;

Is there a better way of doing this, or am I doing something blatantly wrong?

From stackoverflow

• It looks like it should work, but CF does have unpredictable limitations.

  Is xml a requirement? I can't remember trying it with 20k records, but another option might be to try using a different serializer - for example, protobuf-net works on CF2. I can't guarantee it'll work, but it might be worth a shot.

  (in particular, I'm currently refactoring the code to try to work around some additional "generics" limitations within CF - but unless you have a very complex object model this shouldn't affect you).

  ---

  Example showing usage; note that this example also works OK for XmlSerializer, but protobuf-net uses only 20% of the space (or 10% of the space if you consider that characters are two bytes each in memory):

  using System;
  using System.Collections.Generic;
  using System.IO;
  using System.Xml.Serialization;
  using ProtoBuf;
  
  [Serializable, ProtoContract]
  public class Department
  {
      [ProtoMember(1)]
      public string Name { get; set; }
      [ProtoMember(2)]
      public List<Person> People { get; set; }
  }
  
  [Serializable, ProtoContract]
  public class Person
  {
      [ProtoMember(1)]
      public int Id { get; set; }
      [ProtoMember(2)]
      public string Name { get; set; }
      [ProtoMember(3)]
      public DateTime DateOfBirth { get; set; }
  }
  
  
  static class Program
  {
      [MTAThread]
      static void Main()
      {
          Department dept = new Department { Name = "foo"};
          dept.People = new List<Person>();
          Random rand = new Random(123456);
          for (int i = 0; i < 20000; i++)
          {
              Person person = new Person();
              person.Id = rand.Next(50000);
              person.DateOfBirth = DateTime.Today.AddDays(-rand.Next(2000));
              person.Name = "fixed name";
              dept.People.Add(person);
          }
  
          byte[] raw;
          using (MemoryStream ms = new MemoryStream())
          {
              Serializer.Serialize(ms, dept);
              raw = ms.ToArray(); // 473,399 bytes
          }
  
          XmlSerializer ser = new XmlSerializer(typeof(Department));
          StringWriter sw = new StringWriter();
          ser.Serialize(sw, dept);
          string s = sw.ToString(); // 2,115,693 characters
      }
  }
  

  Let me know if you want more help - I can talk about this subject all day ;-p Note that it can work just from the standard xml attributes ([XmlElement(Order=1)]) - I've used the more specific [ProtoMember(1)] etc for clarity. This also allows fine-grained control of serialization (zigzag vs twoscompliment, grouped vs length-prefixed, etc).

  GenericTypeTea : Are there any good examples of how to use ProtoBuf?

  Marc Gravell : I can add one...

  GenericTypeTea : Thank you very much!

  GenericTypeTea : Just to add, thanks to Protobuf.Net, I'm now serializing and then saving to a file 20,000 records (in my largest table structure), in 29 seconds. I can then load that file back using the deserializer in 28 seconds).

• Do you have any metrics on your application's memory consumption? I'm assuming you're running on WM, which means that each process' address space is limited to 32MB. With a large XML, it's possible that you've actually run out of memory.

• Hi there,

  Maybe you could consider persisting the individual objects (rather than persisting the collection as one big block). If so, you might want to use the NFileStorage project I created on codeplex; nfilestorage.codeplex.com (this one is not specifically made for the CF, so cannot tell if its compatible with that one)...

  Good luck, Gert-Jan

  ctacke : If you *know* your project works with the CF, then I have no problem with you offering it as a solution, but if you don't know don't post it in hopes to simply drive exposure. Test the scenario yourself, and if it works *then* let us know.