Expert: Srini Nagarajan - 5/7/2008

Question
QUESTION: i am working in c#.net. I have a text file of 10MB which contains lot of data.i want to process the data by loading it part by part so as to process it efficiently and store it in another text file.which data structure should i use to store the data to be processed part by part? or please tell me some alternative to do that.

ANSWER: hi

There is no efficient way to read the large file properly..

Have a look at the following URL gives you some idea

http://www.dotnet2themax.com/ShowContent.aspx?ID=4ee44d6c-79a9-466d-ab47-56bba52

Happy Programming!!

-Srini

---------- FOLLOW-UP ----------

QUESTION: hi Srini.thanx for the answer.i have gone through the code on on URL which u have sent.but i am not getting how it solves memory problem.if u could tell me that it will be helpfull to me.

ANSWER: hi,

sorry for the delay, i am currently in overseas trip!!

Anyways, you need enough memory on the server to load/read large file.. also try to increase the timeout, sometimes timeout also may cause this issue.

THanks

-Srini

---------- FOLLOW-UP ----------

QUESTION: hi.
The code in the URL sent by u,uses IEnumerable and iterator.So my question is that how IEnumerable is useful for reading the large file.What does it do to solve memory problem.I have enough memory to read the file,but i need time efficiency also.


Answer
hi,

May be you cane have a look at IFilter whether it can solve your problem

good article about it: http://www.codeproject.com/KB/cs/IFilter.aspx

sometime back I was looking an article there was a 3rd party .NET tool which reads the text file more efficiently, if i find that I'll send it to you.

also have you heard "BufferedStream".. Give a quick check on that, may be this can give some light for you.

Happy programming!!

-Srini