c# - A dictionary that can save its elements accessed less often to a disk -

in application use dictionary (supporting adding, removing, updating , lookup) both keys , values or can made serializable (values can possibly quite large object graphs). came point when dictionary became large holding in memory started trigger outofmemoryexception (sometimes in dictionary methods, , in other parts of code).

after attempt replace dictionary database, performance dropped down unacceptable level.

analysis of dictionary usage patterns showed smaller part of values "hot" (are accessed quite often), , rest (a larger part) "cold" (accessed or never). difficult when new value added if hot or cold, moreover, values may migrate , forth between hot , cold parts on time.

i think need implementation of dictionary able flush cold values disk on low memory event, , reload of them on demand , keep them in memory until next low memory event occurs when hot/cold status re-assessed. ideally, implementation should neatly adjust sizes of hot , cold parts , flush interval depending on memory usage profile in application maximize overall performance. because several instances of dictionary exist in application (with different key/value types), think, might need coordinate workflows.

could please suggest how implement such dictionary?

compile 64 bit, deploy on 64 bit, add memory. keep in memory.

before grown own may alternatively @ weakreference http://msdn.microsoft.com/en-us/library/ms404247.aspx. of course require rebuild objects reclaimed 1 should hope reclaimed not used much. comes caveat own guidleines state avoid using weak references automatic solution memory management problems. instead, develop effective caching policy handling application's objects.

of course can ignore guideline , work code account it.

you can implement caching policy , upon expiry save database, on fetch , cache. use sliding expiry of course since concerned keeping used.

do remember used vs heaviest trade off. losing object 10 times day takes 5 minutes restore annoy users more losing object 10000 times tool 5ms restore.

and above mentioned web cache. automatic memory management callbacks noted, depends if want lug 1 around in apps.

and...last not least, @ distributed cache. sharding can split big dictionary across few machines.