New File I/O in ColdFusion 8

Till now we have been using <cffile> for all kind of file operations and it does a very good job. If you want to read a file, give the file to this tag and this tag gives you the read content. If you want to write content to a file, you give the content and file name to this tag and it will do that. You want to copy/delete/move your files, this tag will do all of that. All very simple and short. But there are two particular issues which <cffile> does not address.

1. Reading/writing big files - Since <cffile> is a tag, it can only perform one-shot operations. So, to read, it has to read everything in one shot and to write, you have to provide the entire content and that means that <cffile> will have to keep the entire content in memory. It is not of much concern if the file size is just few KBs but as the size increases beyond 100 kb or when it reaches few megs, it can really hurt. It would create a memory crunch on the server and if the load is high and there are many read/write happening simultaneously with large files, it can even lead to OutOfMemory error in server. Apart from creating memory crunch, it will also slow down the server because VM would need to allocate/deallocate larger chunk of memory which would lead to larger and frequent Garbage Collection cycle. At this point, you might ask, why would I ever read or write such a big file? Well I can think of few

• You need to process the data that comes in a flat file
• csv parsing
• Finding the mime type of a file like mp3, image, video etc
• you want to create a log viewer
• ... many more

You get the idea.. right?

2. Again since <cffile> is a tag, it is not very easy to use inside cfscript. Either you have to move out of cfscript to use this tag or you wrap this tag in a function and call that function. Though thats true with all the tags but cffile is so commonly used that this looks like a limitation.

New File I/O introduced in ColdFusion 8 addresses both these problems. New File I/O is all based on functions and hence that automatically takes care of problem 2. That means you no longer need to use cffile if you are inside cfscript. I will give more details on handling problem 2 in my next post. In this post I will mainly focus on reading/writing files in chunk using new IO .

The new I/O is based on the same philosophy that is used in other languages i.e;

1. You first open a file
2. Perform read/write operations on it
3. and close the file.

Lets see each of the steps in little detail.

Step 1 : Open a file : Here is the function to open a file

FileOpen(filepath [,mode] [,charset]) -> fileobject

Both mode and charset here and optional. Mode can be "read", "readBinary", "write" or "append"

"read" mode, which is default, is used to read a text file and hence any read operation will give you text data from it. When the file is a text file, you can also optionally specify the charset of the file. So if the file contains UTF-8 or UTF-16 characters (or characters from any other charset), you need to specify it while opening the file.

"readBinary" is used to read a binary file and hence any read operation will give you the binary data i.e byte array.

"write" mode will open the file in write mode which means that if the file already exists, it will be overwritten.

"append" mode, as the name suggests, will open the file in append mode which means that any write operation on that file object will write it at the end of file.

FileOpen function returns you a handle to the native file and you need to use this handle for all further read/write operation. Of course you should keep in mind that you can not perform "read" operation on a file handle that was opened in "write" mode and vice versa.

Step 2 : Do Read/Write operations : Once you get the handle to file object, you can perform multiple read/write operations using this handle. There are several functions to do that.

2A. Read Operation :

i) FileRead(fileobj, no of character/bytes to read) : This provides you a way to read a chunk of data (say 1 kb at a time) from the file at a time. Since you only read a chink of data at a time, it does not create memory crunch on the server. Since this is read operation, file must have been opened in "read" or "readBinary" mode. Depending on which mode the file was opened, this function will return the text or binary data read. One thing to note here - If the data remaining is less than the requested size, this method will return you only the remainign data. i.e if 100 character are remaining in the file being read, and you request for 1000 characters, it will return you 100 characters only.

ii) FileReadLine(fileobject) - This reads one line from the text file. To call this method, the file must have been opened in "read" mode.

Both these read operations can be called multiple times until you reach end of the file. One the end of file has reached, any further read call will result into an "EndOfFile" error. So in order to avoid this error, you should always check whether you have reached the end of file. And the function to do that is

FileIsEOF(fileobj) : Just to be more clear, EOF here stands for "End of File". This function will return true if the end of file has been reached otherwise will return false.

Here are few examples of reading content from file
Read 1 kb binary data at a time.

<cfscript>
    myfile = FileOpen("c:\temp\song.mp3", "readbinary");
    while (! FileIsEOF(myfile)) { // continue the loop if the end of file has not reached
       x = FileRead(myfile, 1024); // read 1 kb binary data
       ...// process this binary data..
    }
</cfscript>

Process a text file line by line

<cfscript>
    myfile = FileOpen("c:\temp\myfile.txt", "read");
    while (! FileIsEOF(myfile)) { // continue the loop if the end of file has not reached
        x = FileReadLine(myfile); // read a line
        ...// process this line..
    }
</cfscript>

2B. Write operation

i)FileWrite(fileobject, content) - This will add the text or binary content to the file. The file must have been opened in "write" or "append" mode.

ii) FileWriteLine(fileobject, text) - This will add the text followed by a new line character to the file. Here again, the file must have been opened in "write" or "append" mode.

You might wonder that if both the write operations add the content to the file, whats the difference between "write" and "append" mode? The difference is only at the time of opening the file. As I said earlier, opening the file in "write" mode will overwrite the file if already existsed and put the file pointer at the the beginning of file. Whereas opening file in "append" mode will simply put the file pointer at the end of file.

Any subsequent "write" calls, irrespective of "write" or "append" mode, will append the content to the file.

Here is an example of writing content to a file. This reads one line from an input file, does some processing on it, and writes the resultant data to another file.

<cfscript>
    infile = FileOpen("c:\temp\input.txt", "read");
    outfile = FileOpen("C:\temp\result.txt", "write");
    while (! FileIsEOF(infile)) { // continue the loop if the end of file has not reached
        x = FileReadLine(infile); // read a line
        data = processLine(x);
        FileWriteLine(outfile, data);
    }
</cfscript>

Step 3 : Close the file : Once you are done with read/write operations, you *must* close the handle to file. And the way to do that is using function

FileClose(fileobj)

What if you don't close the file object? Well, that file will remain locked by the server as long as the file is open, and no other process can modify/rename or delete that file.
You might also ask, why does not ColdFusion automatically take care of closing the file? Why should the developer be bothered about it? Well.. ColdFusion does take care of it when the file object goes out of scope and if it is not kept in any accessible scopes but you can never be certain when exactly this will happen. This might happen immediately or this might happen hours later :-).
Bottomline, you should make it a practice to call FileClose() once you are done with the file object.
Just to show its usage, I will complete the example I used in write.

<cfscript>
    infile = FileOpen("c:\temp\input.txt", "read");
    outfile = FileOpen("C:\temp\result.txt", "write");
    while (! FileIsEOF(infile)) { // continue the loop if the end of file has not reached
        x = FileReadLine(infile); // read a line
        data = processLine(x);
        FileWriteLine(outfile, data);
    }
    FileClose(infile);
    FileClose(outfile);
</cfscript>

These set of functions would greatly help if you need to work with a file of more than 10 kb size.

Apart from these set of functions, ColdFusion 8 also adds a new language struct to read text files. With ColdFusion 8, you can use <cfloop> to iterate over "lines" or "characters" in a text file. This makes it very easy and convenient to do any kind of text file parsing or processing in your application. Lets take a look at the new syntax of cfloop for reading file (and I really love this syntax :-)).

New attributes in cfloop for reading file :

"file" - path of the file to read
"characters" - no of characters to read in one iteration.

1. Reading Lines : Below is the simplest syntax to read one line at a time from the file in a loop. This would read the entire file and the loop would end when the file has been completely read. The read content will be available in the index variable specified.
   

   
   <cfloop file="c:\temp\myfile.txt" index="line">
       <cfoutput>#line#</cfoutput> <!--- or do whatever with the line --->
   </cfloop>
   

   
   With cfloop, you can also iterate over a part of the file by specifying "from" and "to" values.
   Here is an example to loop over lines between 10 and 20.
   

   
   <cfloop file="c:\temp\myfile.txt" index="line" from=10 to=20>
       <cfoutput>#line#</cfoutput> <!--- or do whatever with the line --->
   </cfloop>
   

   
   "from" and "to" both are optional attributes where "from" defaults to '1' i.e start of file and "to" defaults to the last line of the file.
   
   So to read first 10 lines from the file, you can use
   

   
   <cfloop file="c:\temp\myfile.txt" index="line" to="10">
       <cfoutput>#line#</cfoutput> <!--- or do whatever with the line --->
   </cfloop>
   

   
   One word of caution here - If you use "to" attribute here, its value must be less than the number of lines in the file otherwise you would get an "EndOfFile" error. For example, if I had only 5 lines in my file and value of to is 7, this would throw an error because line 6 and 7 do not exist.
   


2. Reading characters : For reading characters instead of line, you need to provide the value for "characters" attribute and as many characters will be read in one iteration. The loop will automatically end when the end of file has reached. The read content will be available in the index variable specified.
   
   An example for that is
   

   
   <cfloop file="c:\temp\myfile.txt" index="chars" characters="1000">
       <cfset x=chars>
       <!--- do whatever with the characters --->
   </cfloop>
   

   One important thing to note here. In the last iteration, when the end of file has reached, index variable will only have the remaining characters. For example if I have 130 characters in the file and I run the loop to read a chunk of 20 characters, in the last iteration, index variable's value will only have last 10 characters.
   
   

This completes the first part of new File IO which mainly addresses the problem of working with larger files. However this does not mean that you can not or should not use these for smaller files. You can very much use these for all kind of files. These are very simple to use and perform really well. Go ahead and play around with it !