Tuesday, July 10, 2007

New File I/O in ColdFusion 8

Till now we have been using <cffile> for all kind of file operations and it does a very good job. If you want to read a file, give the file to this tag and this tag gives you the read content. If you want to write content to a file, you give the content and file name to this tag and it will do that. You want to copy/delete/move your files, this tag will do all of that. All very simple and short. But there are two particular issues which <cffile> does not address.

1. Reading/writing big files - Since <cffile> is a tag, it can only perform one-shot operations. So, to read, it has to read everything in one shot and to write, you have to provide the entire content and that means that <cffile> will have to keep the entire content in memory. It is not of much concern if the file size is just few KBs but as the size increases beyond 100 kb or when it reaches few megs, it can really hurt. It would create a memory crunch on the server and if the load is high and there are many read/write happening simultaneously with large files, it can even lead to OutOfMemory error in server. Apart from creating memory crunch, it will also slow down the server because VM would need to allocate/deallocate larger chunk of memory which would lead to larger and frequent Garbage Collection cycle. At this point, you might ask, why would I ever read or write such a big file? Well I can think of few

  • You need to process the data that comes in a flat file
  • csv parsing
  • Finding the mime type of a file like mp3, image, video etc
  • you want to create a log viewer
  • ... many more
You get the idea.. right?

2. Again since <cffile> is a tag, it is not very easy to use inside cfscript. Either you have to move out of cfscript to use this tag or you wrap this tag in a function and call that function. Though thats true with all the tags but cffile is so commonly used that this looks like a limitation.

New File I/O introduced in ColdFusion 8 addresses both these problems. New File I/O is all based on functions and hence that automatically takes care of problem 2. That means you no longer need to use cffile if you are inside cfscript. I will give more details on handling problem 2 in my next post. In this post I will mainly focus on reading/writing files in chunk using new IO .

The new I/O is based on the same philosophy that is used in other languages i.e;

  1. You first open a file
  2. Perform read/write operations on it
  3. and close the file.
Lets see each of the steps in little detail.

Step 1 : Open a file : Here is the function to open a file

FileOpen(filepath [,mode] [,charset]) -> fileobject
Both mode and charset here and optional. Mode can be "read", "readBinary", "write" or "append"

"read" mode, which is default, is used to read a text file and hence any read operation will give you text data from it. When the file is a text file, you can also optionally specify the charset of the file. So if the file contains UTF-8 or UTF-16 characters (or characters from any other charset), you need to specify it while opening the file.

"readBinary" is used to read a binary file and hence any read operation will give you the binary data i.e byte array.

"write" mode will open the file in write mode which means that if the file already exists, it will be overwritten.

"append" mode, as the name suggests, will open the file in append mode which means that any write operation on that file object will write it at the end of file.

FileOpen function returns you a handle to the native file and you need to use this handle for all further read/write operation. Of course you should keep in mind that you can not perform "read" operation on a file handle that was opened in "write" mode and vice versa.

Step 2
: Do Read/Write operations : Once you get the handle to file object, you can perform multiple read/write operations using this handle. There are several functions to do that.

2A. Read Operation :

i) FileRead(fileobj, no of character/bytes to read) : This provides you a way to read a chunk of data (say 1 kb at a time) from the file at a time. Since you only read a chink of data at a time, it does not create memory crunch on the server. Since this is read operation, file must have been opened in "read" or "readBinary" mode. Depending on which mode the file was opened, this function will return the text or binary data read. One thing to note here - If the data remaining is less than the requested size, this method will return you only the remainign data. i.e if 100 character are remaining in the file being read, and you request for 1000 characters, it will return you 100 characters only.

ii) FileReadLine(fileobject) - This reads one line from the text file. To call this method, the file must have been opened in "read" mode.

Both these read operations can be called multiple times until you reach end of the file. One the end of file has reached, any further read call will result into an "EndOfFile" error. So in order to avoid this error, you should always check whether you have reached the end of file. And the function to do that is

FileIsEOF(fileobj) : Just to be more clear, EOF here stands for "End of File". This function will return true if the end of file has been reached otherwise will return false.

Here are few examples of reading content from file
Read 1 kb binary data at a time.

<cfscript>
myfile = FileOpen("c:\temp\song.mp3", "readbinary");
while (! FileIsEOF(myfile)) { // continue the loop if the end of file has not reached
x = FileRead(myfile, 1024); // read 1 kb binary data
...// process this binary data..
}
</cfscript>
Process a text file line by line
<cfscript>
myfile = FileOpen("c:\temp\myfile.txt", "read");
while (! FileIsEOF(myfile)) { // continue the loop if the end of file has not reached
x = FileReadLine(myfile); // read a line
...// process this line..
}
</cfscript>
2B. Write operation

i)FileWrite(fileobject, content) - This will add the text or binary content to the file. The file must have been opened in "write" or "append" mode.

ii) FileWriteLine(fileobject, text) - This will add the text followed by a new line character to the file. Here again, the file must have been opened in "write" or "append" mode.

You might wonder that if both the write operations add the content to the file, whats the difference between "write" and "append" mode? The difference is only at the time of opening the file. As I said earlier, opening the file in "write" mode will overwrite the file if already existsed and put the file pointer at the the beginning of file. Whereas opening file in "append" mode will simply put the file pointer at the end of file.

Any subsequent "write" calls, irrespective of "write" or "append" mode, will append the content to the file.

Here is an example of writing content to a file. This reads one line from an input file, does some processing on it, and writes the resultant data to another file.

<cfscript>
infile = FileOpen("c:\temp\input.txt", "read");
outfile = FileOpen("C:\temp\result.txt", "write");
while (! FileIsEOF(infile)) { // continue the loop if the end of file has not reached
x = FileReadLine(infile); // read a line
data = processLine(x);
FileWriteLine(outfile, data);
}
</cfscript>

Step 3 : Close the file : Once you are done with read/write operations, you *must* close the handle to file. And the way to do that is using function
FileClose(fileobj)
What if you don't close the file object? Well, that file will remain locked by the server as long as the file is open, and no other process can modify/rename or delete that file.
You might also ask, why does not ColdFusion automatically take care of closing the file? Why should the developer be bothered about it? Well.. ColdFusion does take care of it when the file object goes out of scope and if it is not kept in any accessible scopes but you can never be certain when exactly this will happen. This might happen immediately or this might happen hours later :-).
Bottomline, you should make it a practice to call FileClose() once you are done with the file object.
Just to show its usage, I will complete the example I used in write.
<cfscript>
infile = FileOpen("c:\temp\input.txt", "read");
outfile = FileOpen("C:\temp\result.txt", "write");
while (! FileIsEOF(infile)) { // continue the loop if the end of file has not reached
x = FileReadLine(infile); // read a line
data = processLine(x);
FileWriteLine(outfile, data);
}
FileClose(infile);
FileClose(outfile);
</cfscript>

These set of functions would greatly help if you need to work with a file of more than 10 kb size.

Apart from these set of functions, ColdFusion 8 also adds a new language struct to read text files. With ColdFusion 8, you can use <cfloop> to iterate over "lines" or "characters" in a text file. This makes it very easy and convenient to do any kind of text file parsing or processing in your application. Lets take a look at the new syntax of cfloop for reading file (and I really love this syntax :-)).

New attributes in cfloop for reading file :

"file" - path of the file to read
"characters" - no of characters to read in one iteration.

  1. Reading Lines : Below is the simplest syntax to read one line at a time from the file in a loop. This would read the entire file and the loop would end when the file has been completely read. The read content will be available in the index variable specified.

    <cfloop file="c:\temp\myfile.txt" index="line">
    <cfoutput>#line#</cfoutput> <!--- or do whatever with the line --->
    </cfloop>

    With cfloop, you can also iterate over a part of the file by specifying "from" and "to" values.
    Here is an example to loop over lines between 10 and 20.

    <cfloop file="c:\temp\myfile.txt" index="line" from=10 to=20>
    <cfoutput>#line#</cfoutput> <!--- or do whatever with the line --->
    </cfloop>

    "from" and "to" both are optional attributes where "from" defaults to '1' i.e start of file and "to" defaults to the last line of the file.

    So to read first 10 lines from the file, you can use

    <cfloop file="c:\temp\myfile.txt" index="line" to="10">
    <cfoutput>#line#</cfoutput> <!--- or do whatever with the line --->
    </cfloop>

    One word of caution here - If you use "to" attribute here, its value must be less than the number of lines in the file otherwise you would get an "EndOfFile" error. For example, if I had only 5 lines in my file and value of to is 7, this would throw an error because line 6 and 7 do not exist.

  2. Reading characters : For reading characters instead of line, you need to provide the value for "characters" attribute and as many characters will be read in one iteration. The loop will automatically end when the end of file has reached. The read content will be available in the index variable specified.

    An example for that is

    <cfloop file="c:\temp\myfile.txt" index="chars" characters="1000">
    <cfset x=chars>
    <!--- do whatever with the characters --->
    </cfloop>
    One important thing to note here. In the last iteration, when the end of file has reached, index variable will only have the remaining characters. For example if I have 130 characters in the file and I run the loop to read a chunk of 20 characters, in the last iteration, index variable's value will only have last 10 characters.

This completes the first part of new File IO which mainly addresses the problem of working with larger files. However this does not mean that you can not or should not use these for smaller files. You can very much use these for all kind of files. These are very simple to use and perform really well. Go ahead and play around with it !

19 comments:

Sami said...

Very nice!

Anonymous said...

Nice overview of the new file functionality. I think this is going to be make a huge difference.

Rupesh Kumar said...

Thanks Sami and Ben.

Raymond Camden said...

I just with they had added seek. For example, MP3 headers are at the end of the file, so I'd still have to read in the entire file.

Rupesh Kumar said...

@Ray
I agree that seek would have been cool.. CF9 :-)

Regarding MP3 headers, they are not really at the end of file. In fact the headers used to identify a mp3 file are there at each frame. This is what makes it possible to split mp3 file (or extract any part of it) and they would play perfectly fine.
I am saying this because I had to extract mp3 file metadata for cfpresentation. :-)
Check out the mp3 file format details
>here
.

Raymond Camden said...

Ah - I hadn't done parsing in a long time. Well, ok, lets pretend it is some other format. ;)

So - you should see if you can getMP3Info() in CF9 as well. ;)

Rupesh Kumar said...

hehehe :-)

For MP3 Info, the code is all there. Its just a matter of exposing it :-)

Raymond Camden said...

Hmm. So I'm skimming the docs, and it mentions that yes, info on the mp3 is in every frame, but it says the ID3 info is at the end. That is what I meant - the ID3 info, which is the stuff most people are interested in. Make sense? It's too bad that we can't read backwards. That would take care of this case at least.

Raymond Camden said...

Heh, of course, that is for the earlier version. It looks like 2.3.0 may put it in front:

http://www.id3.org/id3v2.3.0#head-697d09c50ed7fa96fb66c6b0a9d93585e2652b0b

Yes, I'm confused now. ;)

Rupesh Kumar said...

Hmm.. I didn't mean ID3 info. It will be interesting to try it out with this function and see how much time it takes.
or let me write some java code to do it.

Raymond Camden said...

Good luck Rupesh. My Java skills are pretty minimal. If you get something going, it would be nice to wrap it in a CFC and publish it on RIAForge.

Anonymous said...

Can you give us an example of how this might work if you had an input type="file" type of file upload browse dialog and you wished to write the selected file to another location.

Pint said...

this still is slow, i get a GC overhead limit exceeded, meaning this method is not sufficient

DecoX said...

Woa! I can do the same in J2EE since 10 years! And I dont need to wait CF12 for the seek operation ;-)

Anonymous said...

Does cfloop file close the file at close cfloop tag or end of processing the whole template

Anonymous said...

How can we find the mime type using this

Mike said...

I did a cfloop over a file and was received a nice output of the file by using a break tag after each iteration.

I thought, "Great. Now let's increment a number after each iteration". No can do. As far as iteration over the file, it's just one iteration!

So, how do I "identify" what line to change if I wanted to change just one line in the file"?

Anonymous said...

When using cfloop to process a file, the documentation states that CF closes the file afterwards. However, I wouldn't want to rely on that. How can I ensure that the file is indeed closed after a cfloop action?

Rupesh Kumar said...

CFLoop will always close the file stream at the end of loop. You cant get a handle for the file stream if you are using the loop construct.

If you really want to do that, you can open the file stream yourself using FileOpen and then process it and close it using FileClose.