If you are familiar with the DevExpress eXpressApp Framework, than you may have already run into this issue and this will be a great tool for you. If not, the DevExpress eXpressApp Framework provides a great tool for modeling and opening up your data. Whether in-house or on a consultant level you will likely benefit from using a tool like this in order to speed up your development cycle and get better products to your customer as quick as possible.
Assuming you have used the XAF framework and a XPCollection to make an association to objects, you may eventually run into a situation where a XPCollection has too much data. Whether you have an association that is too big, or you are getting a collection of objects to process them and crunch some numbers, the collection itself does load all objects into memory and will likely bottleneck you at once point or another. To work around this, DevExpress has introduced an object called the XPCursor. The XPCursor allows you to give a persistent object a page size and still use the sorting and criteria collection.
To create an XPCursor, you would instantiate it just like you would a normal XPCollection, providing the criteria and the object type.
XPCursor xpcOrders = new XPCursor(typeof(Order),new BinaryOperator(“Date”,Today,GreaterThan));
This will return you back a cursor object which you can then set the page size on using the following piece of code:
xpcOrders.PageSize = 100;
At that point you can iterate through or hit that collection just like any other collection. The advantage to using this, is that it will page out only to the records you need. This means that if you are in a For Each loop and you are going through all the records, as soon as you get to record number 101 the collection will automatically load the next 100 records and release the first 100. If you had a record count of 100,000, loading a XPCollection would load 100,000 records as soon as you access it, whereas the XPCursor object would only load 100 records if that was your page size.