Google to roll out wikipedia rival - Google Knol

Google is coming out with Google Knol (Knowledge) which is exactly what Wikipedia is all about. Read on

http://googleblog.blogspot.com/2007/12/encouraging-people-to-contribute.html

http://searchengineland.com/071213-213400.php

Firefox (Adblock plus) blocks even Google adsense

I have been using my new laptop at work (my old one crashed at MAX) for two weeks and its been pretty good. Not because the laptop is something extraordinary, its because I installed the latest versions of all the softwares and plugins I use. And the most pleasant part of it is - it has been completely ads free. I did not notice it earlier but once I landed up on sys-con's site to read some article and I was pleasantly surprised to find no irritating video ad there. For a moment, I thought that may be sys-con has grown up but that was just for a moment :-). The snapshot shows it all.

IE

Firefox

Its actually a firefox plugin adblock plus that does it automatically. The most interesting thing was that it even removed google text ads. Its interesting because we have always seen Google promoting firefox and firefox as a browser closer to Google. So now a firefox plugin that blocks google adsense and in turn hurts Google revenue is certainly not going to go well with Google. It will be interesting to see how Google plays it.

So how exactly does adblock plus block the ads? Gecko engine - the engine underneath Firefox, uses a content policy which decides which content should be loaded and which should not. Adblock defines its own policy which gets added to the browser's policy. It comes with a predefined huge collection of filters and if the url to be loaded matches any of that, it does not let the browser make a request for it. For google adsense, the url contains 'pagead2' which you can see in the adblock preference below.

Its also going to hurt many bloggers as lot of them earn substantially from ads and mainly adsense.  Even my blog page contains adsense but I dont mind adblock plus as I haven't yet earned anything from it anyway. I'm loving it. :-)

Images and CFDocument performance

Sometime back one of our customer reported that using png images in cfdocument makes it very very slow. I could not replicate it with any png image but it did happen with his png image.

Today, Andy reported a similar issue but this time it was for jpeg images. In both the cases, performance hit is huge and it does not happen with all the images.

I spent a significant amount of time debugging it today and it turns out that the reason for both the issues are same - there is something special about these images and that is colorspace. All these images actually had a different colorspace than what java imaging uses. Because of this, when we need the pixel values of image to print it on pdf (by calling BufferedImage.getRGB()), it tries to convert this colorspace to RGB colorspace and that is very very costly. That is where the entire time goes. So how do you fix it? I opened all the images in an image editor and saved it again. This time it got saved in standard RGB colorspace and the time taken to create the pdf got reduced from 110 sec to 1.5 seconds. That is huge!!! Isn't it? But can you control all the images over the web? NO.. right? Read on.. there is more to this story.

A little bit of looking up on web pointed me to this Sun bug which is the exact same bug which we were hitting. Thankfully it got fixed in mustang i.e JDK1.6 which ColdFusion8 uses by default. But hey wait a second.. Didn't Andy say that he is seeing it on ColdFusion 8? why do we still see this happening when it is fixed in JDK1.6? It appears that this bug was fixed only in core JDK api but not in JAI (Java advanced imaging) codecLib that ColdFusion 8 uses. So what do we do now? You can do either of these two

1. Remove clibwrapper_jiio.jar from "lib" folder.
2. Or, set this system property to the JVM. -Dcom.sun.media.imageio.disableCodecLib=true . You can set this in [cf-install_dir]/runtime/bin/jvm.config if you are using standalone coldfusion server.

You should keep in mind that codecLib libraries are native libraries which are meant to increase the java imaging performance. So disabling it might degrade the performance of CFImage somewhere. Also keep in mind that removing this jar or disabling codecLib will not result into any loss of functionality - it just means that all image operations will be pure java.

There is another related Sun's bug which I thought might be useful to you. Image loading might get very slow if the server is running in debug mode. Your server is running in debug mode if you see something like this in your jvm.config or VM startup option.

-Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005

This bug got introduced in JDK1.6 and does not exist on 1.5. So make sure that you are not running the server in debug mode.

To summarise, you can do following to increase the performance of image loading in cfdocument.

1. If you are on JDK1.5, there is not much you can do. The only option is to change the colorspace of images. I will try to see if we can address this in ColdFusion code.
2. If you are on JDK1.6,
   
   • Disable codecLib as mentioned above.
   • Disable debug mode by removing the complete debug string mentioned above
3. In addition to this, you might want to use 'localurl' attribute for cfdocument tag. See this for more details.

Missing text in pdf created by CFDocument

I had heard about people not getting images in the generated pdf but this one was something new and spooky. Yesterday Andy Matthews posted it on cf-talk where he says that for his content, though a pdf containing 15 pages gets created but only the first page has some content and rest of the pages are blank. I ran his code on my machine and sure enough it was happening. All the content including text and image on all the pages except the first page was gone. So what was wrong? I simplified his code and here is a simple example which you can try.

<cfdocument format="pdf">
Text Outside div
<div style="width:300px;overflow:auto;">
 This is text inside the div. Will it show up?
</div><br>
</cfdocument>

In the generated pdf you will see only "Text Outside div". Spooky.. isn't it? The reason it happened was because of css style 'overflow:auto'. The rendering engine used by CFDocument underneath does not handle overflow:auto and it simply ignores the content in the div. The weird part is - it considers that content for all rendering calculations including page number and page break calculation.

Something you should watch out for if you use css styles in cfdocument. A workaround for the time being is not to use overflow:auto style. So modifying the above code like this will make it work.

<cfdocument format="pdf">
Text Outside div
<div style="width:300px;">
 This is text inside the div. Will it show up?
</div><br>
</cfdocument>

64 bit Support on ColdFusion 8 (Windows, Linux and Leopard)

ColdFusion team is working to get ColdFusion 8 working on 64 Bit platforms(Windows, Linux and Leopard). If you are interested and want to participate in the testing of early releases of 64 Bit support on ColdFusion 8 on these platforms, please enroll yourself for prerelease program here.

JRun updater 7 released

JRun 4 updater 7 got released last week and is publicly available for download. It now supports a host of new platforms, includes upgrades to existing software components and fixes for many critical bugs.

Some of the important platform support include

• JDK 1.6 support
• Support for Windows Vista.
• Mac OS X 10.4 on Intel
• 64-bit Support on Solaris-SPARC.
• IIS7 support.
• Apache 2.2.x support
• SunOne Webserver 7
• IBM JDK 1.5

Another important reason why you might want this is - a very significant performance improvement in web clustering. Check out the detailed release notes here.

The Week at Max

Wow ! What an incredible week it was. A week that saw the largest ever MAX, a week that saw the largest get together of ColdFusion community, a week that saw the best ever visibility of CF at MAX general sessions, a week that saw U2's Bono (err.. his fake??) having fun at a tech conference like MAX and a week that saw lot of cool and amazing Adobe technologies shaping up.

It also had moments of a Hollywood thriller for me. I was presenting a session on "Leveraging ColdFusion for AIR application". My first session was on Tuesday, 2nd October at 4 o'clock and at around 2 o'clock my laptop crashed and after that it just refused to boot up. Luckily I had the backup of my ppt and some demo files and hence I decided to use Sanjeev's (Pdf and compiler guy in the CF team) laptop to make the presentation. I just had one and half hour and I needed to setup everything back on his machine - install the air runtime, air sdk, cfeclipse, copy all my files etc and keep the setup ready to run whatever demos I was left with. To add to this, somehow there was no Internet available in the conference area - So where to get AIR runtime and SDK from? Sanjeev suggested that may be AIR guys might have it and so we ran across to the AIR park to get those on my flash drive. I can't tell you what those one and half hour were like :-). I barely managed to get the presentation ready in time and thankfully the presentation went well.

As if this was not enough, the adventure continued for the repeat session as well. No this time the laptop didn't crash.. but this time I could not find Sanjeev so that I could take the laptop from him. I just had half an hour left for the session and I did not have any laptop. And I didn't even have the patience for setting it all up once again on another laptop. Finally found him just 15 minutes before my session while he was chatting with BEA guys. phew..

Anyway, here is my presentation and some of the demo files if you are interested to take a look at that. I could not upload it earlier as I got a temporary laptop just yesterday. I will upload the flex part of demo in a day or two.

If you were there at MAX, you must have seen Hemant talking about ColdFusion and AIR at sneak peek. Though this was just a prototype, it surely gives an idea what can be done in CF for air apps. It would be great if you can share what exactly or what else would you like to see in ColdFusion to help you build AIR application easily. Thoughts??

CFDocument hotfix for ColdFusion 8 released

We have released a cumulative hotfix for CFDocument related issues in ColdFusion 8. Bugs being addressed in this hotfix are

• PDF/Flashpaper document generated in CF8 are nearly 70% scaled down than CFMX 7. This was making document appear smaller in CF8 as compared to CF7.
• Using cfdocumentitem in a loop inside cfdocument causes the error, "routines cannot be declared more than once".
• StackOverFlowError when a large document with table is created. A simple report with over 1500 table row causes this problem. More complex, but shorter documents have also resulted in this error. This was a pretty nasty one and it deserves a separate blog entry :-).

You can download the hotfix directly from here. You would need to extract the hotfix jar from the downloaded zip file and apply that jar.

ColdFusion 8 : Changes with File Upload

Till ColdFusion 7, it was not possible to upload a large file. The server could not handle uploading a file of probably anything more than 300-400 MB. And on top of that, if there are multiple people trying to upload large files simultaneously to the server, only thing that you can do is to pray :-)

You might wonder and I agree with you, that its a huge limitation. So why was it this way ? Well.. that was because the server had to keep the entire uploaded file in the memory. Whoa.. Why the heck does the server need to keep the entire uploaded file in memory? - For a single reason of serving the function GetHttpRequestData(). As you know, this function returns a 'struct' containing the metadata for the Http request that comes to the server. Since this struct also includes request body, and the uploaded file is part of the request, the file has to be retained in the memory in order to honor this function.

With ColdFusion 8, we have removed the limitation on file upload size. So now you can upload files of any size without causing any issue with the server. I have successfully tried with 5 GB file because that was the biggest I could get hold of but you can absolutely upload files of any size. ColdFusion nicely handles it, reads it the way it should without requiring much memory.

But hey, hold on a sec.. What about the function GetHttpRequestData()? Wasn't that the reason for this limitation in the first place? Well, the function GetHttpRequestData() works perfectly fine except in this one case - In the multipart request, i.e; request in which a file is uploaded - "content" in the returned struct will be empty. If you need to get the uploaded file content in memory, you can always read the uploaded file and keep it in memory. So that should not be much we are asking for. Are we?

If in case, you absolutely want the content from GetHttpRequestData() and file upload size is of no concern to you, we allow you to revert back to ColdFusion 7 behaviour. All you need to do for this is to set the Java system property "coldfusion.markResetForMultiPart" to "true" and you are back to the old behavior.

Unlimited aliases with Gmail and SPAM protection

Though this hack in Gmail has been known for a while, I read about it today and found it interesting enough to share it. As Amit decribes in his post, GMAIL lets you create unlimited number of aliases for your gmail id and you can then give these aliases on other websites without worrying about spams. The trick is to use '+' addressing!

As he says, gmail ignores anything between '+' and '@' in the email address. So a mail sent to paris.hilton+model@gmail.com or paris.hilton+celebrity@gmail.com will all go to paris.hilton@gmail.com. In the received mail, "To" will still point to the original id to which it was sent which makes it easy to create spam filters based on the 'To' address. If used intelligently, you also get to know which site leaks your address and creates SPAM. So, if I have to give my mail id to any website xyz.com, I can give 'mymailid+xyz@gmail.com'. If I get the spam with 'to' address as mymailid+xyz@gmail.com, I get to know that its xyz.com that had leaked the id and creates spam.  Pretty cool..

ColdFusion and WebService : "class file has wrong version"

I can't even count the number of times I have seen this error while QA folks were running webservices regressions for ColdFusion 8. Everytime, someone was running webservices tests on a J2EE configuration they used to get an error :

coldfusion.jsp.CompilationFailedException: Errors reported by Java compiler:
....
class file has wrong version 49.0, should be 48.0

I saw the same error posted in the public forum where the user had deployed ColdFusion 8 on JRun with JDK1.5 and was trying to invoke webservice. And hence this blog entry :-)

This error comes because of incompatibility between the tools.jar and the JVM version. ColdFusion 8 ships Java 1.6's tools.jar (you will find it in cfusion/lib) which is of course incompatible with any other JDK/JRE version. You will see the same behavior with ColdFusion 7 as well, if you run it on a JVM other than 1.4. So the rule of thumb is - if ColdFusion is running on a JVM version other than the shipped one, you must ensure that the tools.jar is in synch. In short, you will need to copy JDK's tools.jar to cfusion/lib.

Using CFFeed with URL sending compressed content

Today someone posted on ColdFusion forum regarding this problem where cffeed was not able to handle a particular URL and it was throwing an error. The URL which he tried was http://movies.msn.com/rss/topcelebs and it failed with an error

"Unable to read the source URL.
unknown compression method"

The reason it happens is - The URL returns the response in gzip compressed format only. So when ColdFusion sent a request to this URL and asked for uncompressed data, it could not get anything and hence it was unable to read it. A simpe workaround for this is to use cfhttp to fetch the content, write to a temporary file and then use the cffeed tag to read this file. Important thing to keep in mind here is to set an additional header in the cfhttp tag using cfhttpparam to indicate that it can accept compressed data as well.

Here is the modified code where it first tries cffeed with the URL. If that fails, then it tries to use cfhttp to fetch the content and writes to a temporary file and then uses it in cffeed.

<cfset tempDir=GetTempDirectory()>
<cfset tempFile = GetTempFile(tempDir, "myfeed")>
<cfset tempFileName = GetFileInfo(tempFile).name>
<cftry>
    <cffeed action="read" source="http://movies.msn.com/rss/topcelebs" name="feedInStruct" >
<cfcatch any>
    <cfhttp url="http://movies.msn.com/rss/topcelebs" path="#tempDir#" file="#tempFileName#">
        <cfhttpparam type="header" name="Accept-Encoding" value="compress,gzip,deflate">
    </cfhttp>
    <cffeed action="read" source="#tempFile#" name="feedInStruct" >
</cfcatch>
</cftry>
<cfif FileExists(tempFile)>
    <cfset FileDelete(tempFile)>
</cfif>

<cfdump var="#feedInStruct#">

ColdFusion 8 : IsInstanceOf

If you use lot of CFC inside your ColdFusion application, I am sure you would have come across a situation where you would need to know whether the object is an instance of a particular CFC. This is specially needed when you have components extending other component or you are passing the objects around. ColdFusion 8 introduces a new function IsInstanceOf to do exactly the same. It becomes even more useful after we have interfaces in ColdFusion. And the icing on the cake is that it works even with java objects which means that you can use this function to find out if a particular object is of a particular java class type.

Here is how the function looks.

IsInstanceOf(object, typeName)

where typeName is name of the component/Interface or fully qualified java class name.

It returns 'true' if

• The object passed is an instance of a component which is same as specified type or inherits it or implements the specified interface. Just to be clear, a component 'A' inherits a component 'B' if A or any of its super component extends 'B'. Similarly a Component 'A' implements an interface 'B' if A or any of its super component, implements interface 'B' or any of the interface that 'A' or its parents implement, extends from the specified interface.
  
• The object passed is an instance of a java class (created using cfobject or createObject for java class) which is same as specified class name or inherits the specified class name or implements the specified interface.

Here is an example

Intf.cfc

<cfinterface>
<cffunction name = "foo">
</cffunction>
</cfinterface>

Comp.cfc

<cfcomponent implements="Intf">
<cffunction name = "foo">
   <cfoutput>In method foo</cfoutput>
</cffunction>
</cfcomponent>

test.cfm

<cfset obj = CreateObject("Component", "Comp")>
<!--- Create a Java object --->
<cfset javaObj = CreateObject("java", "java.lang.StringBuffer")>

<cfoutput>object is of type Comp : #IsInstanceOf(obj, "Comp")#</cfoutput><br>
<cfoutput>object is of type Intf : #IsInstanceOf(obj, "Intf")#</cfoutput><br>
<cfoutput>java object is of type String : #IsInstanceOf(javaobj, "java.lang.String")#</cfoutput><br>
<cfoutput>java object is of type StringBuffer : #IsInstanceOf(javaobj, "java.lang.StringBuffer")#</cfoutput><br>

Printers harmful to health..

I am not kidding. There was a recent study done by Queensland University of Technology, Australia which said that printers are hazardous and can cause sever illness including cancer ! Check out the report here.

ColdFusion 8 performance whitepaper

I am glad to see the ColdFusion community so excited with CF 8 performance. We got very encouraging response regarding the performance in our pre-release forums and we are thankful to everyone who gave such invaluable feedbacks. Though people started seeing a very good performance gain in their application on ColdFusion 8, we never talked about the performance benchmark numbers until CFUnited 2007, where Ben showed the performance numbers for ColdFusion 8. No wonder this is the #1reason in Top 8 reasons why you want ColdFusion 8 :-).

Extracting performance up to this extent was not an easy task. We analyzed nearly 2.4 million lines of real life CF application code, to zero-in on the most commonly used tags and functions. Main challenge was after that - analyze the generated java code for all the tags, change the compiler to generate more optimized code, run it through profiler and optimize CF engine for each of the tags and functions and their various combinations. And this went on and on in many many iterations. Overall it was real fun :-)

I still remember the most exciting moment when I ran the load test for CFC, after I had made some code changes, and the result was freaking unbelievable. It looked too good to be true and I was literally running around with the code changes to run it on other machines and verify it . That one small change gave nearly 6x gain :-). It is not that the CF7 code was inefficient or poorly written. It was only matter of extracting juices as much as possible and putting some smart intelligence.

Check out the ColdFusion 8 performance whitepaper which talks in much detail about the performance numbers for different areas, the methodology used for benchmarking etc. Manju Kiran, who was my QA-buddy for most of the features I worked on, did a tremendous job in setting up and running the benchmark and creating the meat of this wonderful document (and of course keeping me on toes).

ListToArray in ColdFusion 8

There has been lots and lots of discussion in CF blogs and forums about ListToArray not supporting empty elements. Seeing which I had blogged about a simple way which could do the same. Thanks to Charlie Griefer for pointing out the problem in it and then thanks to Ben Nadel and Andrew Clark for pointing me in the right direction.

Though it got pretty late, I was able to sneak-in this change in ColdFusion 8. ListToArray() now takes an additional optional argument "includeEmptyElements", which if 'true' will include the empty elements of list into the array. Default is of course 'false'. It also takes care of empty elements at the end of list and multiple delimiters. Here is how the function looks

ListToArray(list, delimiter, includeEmptyElements) returns Array

Lets take a look at couple of examples to see it working

<cfset list = "a,b,,c, ,d,,">
<cfset arr = ListToArray(list, ',', true)>
<cfdump var="#arr#">

Here is how the output looks.

Here is another example.

<cfset list = "one,/$/,six">
<cfset arr = listToArray(list, ",$/",true)>
<cfdump var="#arr#">

The output for which looks like this

Though we wanted to, there was just not enough time to make similar change in all the list functions for CF 8. Something for CF 9 :-)

Enhancements to CFDocument in ColdFusion 8

As you all know, CFDocument tag is used to easily create pdf or flashpaper documents from HTML/CFML content. ColdFusion 8 has added lot of enhancements to it and in this post we will talk about those enhancements.

1. Bookmark : You can create bookmarks for each section of the pdf using "bookmark" attribute. The bookmark created is of only one level and its name is set to the documentsection's name. Here is a sample code for creating pdf with bookmarks.

<cfdocument format="PDF" bookmark="yes">
   <cfdocumentsection name="Introduction">
       <h3>Introduction</h3>
       <p>The introduction goes here.</p>
   </cfdocumentsection>
   <cfdocumentsection name="Chapter 1">
       <h3>Chapter 1: Getting Started</h3>
       <p>Chapter 1 goes here.</p>
   </cfdocumentsection>
   <cfdocumentsection name="Chapter 2">
       <h3>Chapter 2: Building Applications</h3>
       <p>Chapter 2 goes here.</p>
   </cfdocumentsection>
   <cfdocumentsection name="Conclusion">
       <h3>Conclusion</h3>
       <p>The conclusion goes here.</p>
   </cfdocumentsection>
</cfdocument>

2. Proxy Support : With ColdFusion 8, you can now provide proxy configuration to cfdocument for retrieving external content like images. This will be useful in situation where the machine hosting ColdFusion is connected to the external world via a proxy. The new attributes added for this (which are self explanatory) are listed below

• proxyHost
  
• proxyPort
  
• proxyUser
  
• proxyPort

3. Content from URL : Though this was added in 7.0.2, I think it makes sense to add here because it was not there in 7. :-) Have you ever created or wanted to create a pdf from a web page? If yes, then the new attribute "src" in cfdocument and cfdocumentsection tag makes it very easy to do this. Here is an example to do this.

<cfdocument format="pdf" src="http://www.google.com" />

4. Basic Authentication : If the CFDocument body contains a resource (e.g; image or URL) that is protected with basic authentication, ColdFusion 7 can not retrieve it and it was one of the reason for getting "red-x" for images. (See my old post on this). ColdFusion 8 addresses this by adding these two attributes to cfdocument and cfdocumentsection tag.

• authuser - user name to be used for basic authentication.
  
• authpassword - password to be used for basic authentication.

5. User Agent : There are some cases where the web server is configured to allow requests only from a certain set of browsers (User agents to be precise) to prevent spiders and bots from overloading the server. In ColdFusion 7, when CFDocument creates a URLConnection for an image, it sends a "User-Agent" header, that looks like "User-Agent:Java/1.4.2_07", in the HTTP request. If the web server does not recognize "Java" user-agent, it returns a status code of 404 (resource not found) and hence the images can not be displayed. ColdFusion 8 adds a new attribute "user-agent" to address this.

• user-agent : User agent to be used for making http connection. The default value will now be "ColdFusion". If this also does not work, you can use the same string that browsers like IE or firefox use. This attribute has been added to <cfdocument> and <cfdocumentsection> tag.

6. PDF name : When a pdf generated by cfdocument is sent to the browser and you try to save it, the browser will prompt with the cfm name for the pdf which is generally not desirable. With ColdFusion 8, you can provide the appropriate name in "saveAsName" attribute of cfdocument.

• saveAsName - Name that appears in the 'saveAs' dialog when you try to save the pdf from the browser (Generally File -> saveAs). This name will not work if you use "save" dialog of the pdf plugin.

Here is a sample code for the same.

<cfdocument format="pdf" saveAsName="mypdf">
 <p>This is a PDF document.</p>
</cfdocument>

7. Local URLs : When CFDocument body contains a relative URL, ColdFusion will resolve the relative URL to an absolute URL and will send an HTTP request for this url. A side effect of this is - Server ends up sending HTTP request even for local URL or images that are lying on the local file system which obviously hurts the performance. In ColdFusion 8, we have added a new attribute "localURL" to cfdocument tag which if enabled, will try to resolve the relative URLs as file on the local machine.

• localURL : "true" | "false" - It should be enabled if the images used in cfdocument body are on the local machine. This would make the cfdocument engine retrieve the images directly from the file system rather then asking the server for it over http.

This attribute helps reducing the load from the server so that the same web server thread can now serve user request instead of serving local images to CFDocument. This also addresses some of the "missing image" problems which I mentioned here. Here is a sample code using this attribute.

<cfdocument format="PDF" localUrl="true">
 <table>
  <tr>
     <td>bird</td>
     <td><image src="images/bird.jpg"></td>
  </tr>
  <tr>
     <td>Rose</td>
     <td><image src="images/rose.jpg"></td>
  </tr>
 </table>
</cfdocument>

8. Section Page Counts : CFDOCUMENT scope contains two new variables which give you the page counts for document section.

• TOTALSECTIONPAGECOUNT - total no of pages in the current section
  
• CURRENTSECTIONPAGENUMBER - Current page number in the current section.

9. Dynamic header and footer : CFDOCUMENT scope variables can now be used in expressions inside <cfdocumentitem> which makes it possible to have dynamic header and footers. You can now build logic for header/footer content based on the page number. Here is a sample code which prints section title if the page is even and prints the page no otherwise. Below is a code snippet which creates a dynamic header.

    <cfdocumentitem type="header">
       <cfif (cfdocument.currentpagenumber mod 2) is 0>
           <cfoutput>#sectionTitle#</cfoutput>
       <cfelse>
           <cfoutput>#cfdocument.currentpagenumber# of #cfdocument.TOTALPAGECOUNT#</cfoutput>
        </cfif>
   </cfdocumentitem>

10. We have also fixed most of the CFDocument related bugs e.g text chopping, image cropping or red-x, image scaling, "Document has no pages" etc.

New File I/O in ColdFusion 8 - Part II

In my previous post I talked about how new file I/O functions address working with large files. In this post I will talk about other file functions that have been added in ColdFusion 8. Most of these functions are equivalent to the cffile operations and we have retained the same behaviour as cffile. A side effect of that is - relative paths used in these functions are relative to the temp directory. I don't really like that and I believe that it should have been relative to the template. But since these functions were supposed to replicate cffile behaviour, we had to live with it. :-)

Here we go with the list of those new functions

FileRead(filepath, [charset]) - Similar to cffile, this function reads the entire content of a text file and returns the read content. you can also opitonally pass the charset to be used to read the text file.

FileReadBinary(filepath) - This reads the entire content of a binary file and returns the byte array.

FileWrite(filepath, textdata | binarydata, [charset]) - Writes the specified content to the file. The content can be binary as well as text. If the specified content is a text data, you can optionally specify the charset so that the data can be written properly to the file.

FileCopy(source, destination) - As the name suggests, it copies the source file to destination file. Similar to cffile, if the destination is a directory, then source will be copied to that directory otherwise source file will be copied to the destination file.

FileMove(source, destination) - Moves the file from source to destination. Here again, if the destination is a directory, then source is moved under destination directory. Otherwise source is renamed to the destination.

FileDelete(filepath) - Deletes the specified file. The only important thing to note here is that if the file is readOnly, it will not be deleted.

FileSetAttribute(filepath, attribute) - Sets the attributes on file. Applies to Windows. 'attribute' here is a comma-delimited list of attributes to set on the file. Possible attribute values are "readOnly" | "hidden" | "normal".

FileSetAccessMode(filepath, mode) - Sets the file access mode for Unix or Linux systems where the mode is octal values of UNIX chmod command assigned to owner, group, and other, respectively. To give full permission to everyone for a file, the mode should be 777.

GetFileInfo(filepath) - So far till ColdFusion 7, there was no good way to find information like size, last modified date etc about a file. Only way you could do that was to use cfdirectory tag to list the directory, get the query from it, loop over the query until you hit the desired file and then fetch the required metadata. The new function GetFileInfo in ColdFusion 8 provides an easy way to get all the meta-data about a file or directory. This returns a struct which is described below.

• name - Name of the file/directory specified. This is just the file name and not the absolute path.
• path - Full path of the file/directory.
• parent - Full path of the parent directory.
• type - "directory" if the filepath is a directory else "file".
• size - size of the file in bytes.
• lastmodified - DateTime at which this file/directory was last modified.
• canRead - "true" if this file/directory has 'read' permission. "false" Otherwise.
• canWrite - "true" if this file/directory has 'write' permission. "false" Otherwise.
• isHidden - "true" if this file/directory is hidden. "false" Otherwise.

Few more details on File handle

In my previous post I talked about file object that you get on FileOpen() which is nothing but handle to the native file. Did you ever try to dump this object? This file object provides lot of valuable information. If I run this code below

<cfset myfile = FileOpen("C:\cfunited_notes.txt")>
<cfdump var="#myfile#">
<cfloop condition="Not FileIsEOF(myfile)">
    <cfset line = FileReadLine(myfile)>
    <cfoutput>#line#</cfoutput><br>
</cfloop>
<cfset FileClose(myfile)>

this is what gets dumped.

As you can see, it gives you information like lastmodified time, mode in which the file was opened, name, path, size in bytes and status of this file object whether this is still open or closed.

This object acts very much like a struct. So you can access these data from the file object using the simple dot notation. For example, to find out the last modified time, you can use fileObj.lastmodified

That gives another useful tool in your hand. While you are writing a file incrementally, you can easily find out the size of the file written so far using fileObj.size. This will be very helpful if you want to build a logging application where log files are rotated. While you are logging the data, as soon as the file size becomes more than your certain limit, you can close the file object and start writing to a new log file.

New File I/O in ColdFusion 8

Till now we have been using <cffile> for all kind of file operations and it does a very good job. If you want to read a file, give the file to this tag and this tag gives you the read content. If you want to write content to a file, you give the content and file name to this tag and it will do that. You want to copy/delete/move your files, this tag will do all of that. All very simple and short. But there are two particular issues which <cffile> does not address.

1. Reading/writing big files - Since <cffile> is a tag, it can only perform one-shot operations. So, to read, it has to read everything in one shot and to write, you have to provide the entire content and that means that <cffile> will have to keep the entire content in memory. It is not of much concern if the file size is just few KBs but as the size increases beyond 100 kb or when it reaches few megs, it can really hurt. It would create a memory crunch on the server and if the load is high and there are many read/write happening simultaneously with large files, it can even lead to OutOfMemory error in server. Apart from creating memory crunch, it will also slow down the server because VM would need to allocate/deallocate larger chunk of memory which would lead to larger and frequent Garbage Collection cycle. At this point, you might ask, why would I ever read or write such a big file? Well I can think of few

• You need to process the data that comes in a flat file
• csv parsing
• Finding the mime type of a file like mp3, image, video etc
• you want to create a log viewer
• ... many more

You get the idea.. right?

2. Again since <cffile> is a tag, it is not very easy to use inside cfscript. Either you have to move out of cfscript to use this tag or you wrap this tag in a function and call that function. Though thats true with all the tags but cffile is so commonly used that this looks like a limitation.

New File I/O introduced in ColdFusion 8 addresses both these problems. New File I/O is all based on functions and hence that automatically takes care of problem 2. That means you no longer need to use cffile if you are inside cfscript. I will give more details on handling problem 2 in my next post. In this post I will mainly focus on reading/writing files in chunk using new IO .

The new I/O is based on the same philosophy that is used in other languages i.e;

1. You first open a file
2. Perform read/write operations on it
3. and close the file.

Lets see each of the steps in little detail.

Step 1 : Open a file : Here is the function to open a file

FileOpen(filepath [,mode] [,charset]) -> fileobject

Both mode and charset here and optional. Mode can be "read", "readBinary", "write" or "append"

"read" mode, which is default, is used to read a text file and hence any read operation will give you text data from it. When the file is a text file, you can also optionally specify the charset of the file. So if the file contains UTF-8 or UTF-16 characters (or characters from any other charset), you need to specify it while opening the file.

"readBinary" is used to read a binary file and hence any read operation will give you the binary data i.e byte array.

"write" mode will open the file in write mode which means that if the file already exists, it will be overwritten.

"append" mode, as the name suggests, will open the file in append mode which means that any write operation on that file object will write it at the end of file.

FileOpen function returns you a handle to the native file and you need to use this handle for all further read/write operation. Of course you should keep in mind that you can not perform "read" operation on a file handle that was opened in "write" mode and vice versa.

Step 2 : Do Read/Write operations : Once you get the handle to file object, you can perform multiple read/write operations using this handle. There are several functions to do that.

2A. Read Operation :

i) FileRead(fileobj, no of character/bytes to read) : This provides you a way to read a chunk of data (say 1 kb at a time) from the file at a time. Since you only read a chink of data at a time, it does not create memory crunch on the server. Since this is read operation, file must have been opened in "read" or "readBinary" mode. Depending on which mode the file was opened, this function will return the text or binary data read. One thing to note here - If the data remaining is less than the requested size, this method will return you only the remainign data. i.e if 100 character are remaining in the file being read, and you request for 1000 characters, it will return you 100 characters only.

ii) FileReadLine(fileobject) - This reads one line from the text file. To call this method, the file must have been opened in "read" mode.

Both these read operations can be called multiple times until you reach end of the file. One the end of file has reached, any further read call will result into an "EndOfFile" error. So in order to avoid this error, you should always check whether you have reached the end of file. And the function to do that is

FileIsEOF(fileobj) : Just to be more clear, EOF here stands for "End of File". This function will return true if the end of file has been reached otherwise will return false.

Here are few examples of reading content from file
Read 1 kb binary data at a time.

<cfscript>
    myfile = FileOpen("c:\temp\song.mp3", "readbinary");
    while (! FileIsEOF(myfile)) { // continue the loop if the end of file has not reached
       x = FileRead(myfile, 1024); // read 1 kb binary data
       ...// process this binary data..
    }
</cfscript>

Process a text file line by line

<cfscript>
    myfile = FileOpen("c:\temp\myfile.txt", "read");
    while (! FileIsEOF(myfile)) { // continue the loop if the end of file has not reached
        x = FileReadLine(myfile); // read a line
        ...// process this line..
    }
</cfscript>

2B. Write operation

i)FileWrite(fileobject, content) - This will add the text or binary content to the file. The file must have been opened in "write" or "append" mode.

ii) FileWriteLine(fileobject, text) - This will add the text followed by a new line character to the file. Here again, the file must have been opened in "write" or "append" mode.

You might wonder that if both the write operations add the content to the file, whats the difference between "write" and "append" mode? The difference is only at the time of opening the file. As I said earlier, opening the file in "write" mode will overwrite the file if already existsed and put the file pointer at the the beginning of file. Whereas opening file in "append" mode will simply put the file pointer at the end of file.

Any subsequent "write" calls, irrespective of "write" or "append" mode, will append the content to the file.

Here is an example of writing content to a file. This reads one line from an input file, does some processing on it, and writes the resultant data to another file.

<cfscript>
    infile = FileOpen("c:\temp\input.txt", "read");
    outfile = FileOpen("C:\temp\result.txt", "write");
    while (! FileIsEOF(infile)) { // continue the loop if the end of file has not reached
        x = FileReadLine(infile); // read a line
        data = processLine(x);
        FileWriteLine(outfile, data);
    }
</cfscript>

Step 3 : Close the file : Once you are done with read/write operations, you *must* close the handle to file. And the way to do that is using function

FileClose(fileobj)

What if you don't close the file object? Well, that file will remain locked by the server as long as the file is open, and no other process can modify/rename or delete that file.
You might also ask, why does not ColdFusion automatically take care of closing the file? Why should the developer be bothered about it? Well.. ColdFusion does take care of it when the file object goes out of scope and if it is not kept in any accessible scopes but you can never be certain when exactly this will happen. This might happen immediately or this might happen hours later :-).
Bottomline, you should make it a practice to call FileClose() once you are done with the file object.
Just to show its usage, I will complete the example I used in write.

<cfscript>
    infile = FileOpen("c:\temp\input.txt", "read");
    outfile = FileOpen("C:\temp\result.txt", "write");
    while (! FileIsEOF(infile)) { // continue the loop if the end of file has not reached
        x = FileReadLine(infile); // read a line
        data = processLine(x);
        FileWriteLine(outfile, data);
    }
    FileClose(infile);
    FileClose(outfile);
</cfscript>

These set of functions would greatly help if you need to work with a file of more than 10 kb size.

Apart from these set of functions, ColdFusion 8 also adds a new language struct to read text files. With ColdFusion 8, you can use <cfloop> to iterate over "lines" or "characters" in a text file. This makes it very easy and convenient to do any kind of text file parsing or processing in your application. Lets take a look at the new syntax of cfloop for reading file (and I really love this syntax :-)).

New attributes in cfloop for reading file :

"file" - path of the file to read
"characters" - no of characters to read in one iteration.

1. Reading Lines : Below is the simplest syntax to read one line at a time from the file in a loop. This would read the entire file and the loop would end when the file has been completely read. The read content will be available in the index variable specified.
   

   
   <cfloop file="c:\temp\myfile.txt" index="line">
       <cfoutput>#line#</cfoutput> <!--- or do whatever with the line --->
   </cfloop>
   

   
   With cfloop, you can also iterate over a part of the file by specifying "from" and "to" values.
   Here is an example to loop over lines between 10 and 20.
   

   
   <cfloop file="c:\temp\myfile.txt" index="line" from=10 to=20>
       <cfoutput>#line#</cfoutput> <!--- or do whatever with the line --->
   </cfloop>
   

   
   "from" and "to" both are optional attributes where "from" defaults to '1' i.e start of file and "to" defaults to the last line of the file.
   
   So to read first 10 lines from the file, you can use
   

   
   <cfloop file="c:\temp\myfile.txt" index="line" to="10">
       <cfoutput>#line#</cfoutput> <!--- or do whatever with the line --->
   </cfloop>
   

   
   One word of caution here - If you use "to" attribute here, its value must be less than the number of lines in the file otherwise you would get an "EndOfFile" error. For example, if I had only 5 lines in my file and value of to is 7, this would throw an error because line 6 and 7 do not exist.
   


2. Reading characters : For reading characters instead of line, you need to provide the value for "characters" attribute and as many characters will be read in one iteration. The loop will automatically end when the end of file has reached. The read content will be available in the index variable specified.
   
   An example for that is
   

   
   <cfloop file="c:\temp\myfile.txt" index="chars" characters="1000">
       <cfset x=chars>
       <!--- do whatever with the characters --->
   </cfloop>
   

   One important thing to note here. In the last iteration, when the end of file has reached, index variable will only have the remaining characters. For example if I have 130 characters in the file and I run the loop to read a chunk of 20 characters, in the last iteration, index variable's value will only have last 10 characters.
   
   

This completes the first part of new File IO which mainly addresses the problem of working with larger files. However this does not mean that you can not or should not use these for smaller files. You can very much use these for all kind of files. These are very simple to use and perform really well. Go ahead and play around with it !

CFUnited '07 experience

This was my first CFUnited conference and I was amazed with the energy and passion of ColdFusion community. I have been to other non-CF conferences earlier and I can say that I had never seen such passionate and loyal developer community. The interaction that I had with folks there was really awesome and much valuable. The excitement and buzz around ColdFusion 8 was tremendous and all of us in the team are really excited with the response. I am sure ColdFusion 8 will be the best ColdFusion release so far.

All the sessions around CF8 including my session on "CFML Enhancements in ColdFusion 8" were full packed sessions and were very well received. Thanks a lot guys if you attended my session :-). Here are the slides for my session in case you missed it.

At CFunited 07

I am posting this from Bethesda Marriott where I am attending CFUnited. I would love to meet you guys if you are here. You can catch me near our booth or at the end of my session where I will be speaking on "Language Enhancements in ColdFusion 8" on Thursday morning 8.30 - 9.30. Apart from tons of feature, we have done lots and lots of enhancements in Scorpio which should help you write better and faster application. I have selected some of the exciting enhancements for this session which I am sure you would love.
See you there !

New CFDocument hotfix released !

We have released ColdFusion MX 7.02 - CFDocument Cumulative Hot Fix that fixes lot of CFDocument related issues.
Some of the important issues that we fixed in this hotfix are

1. Text getting chopped / clipped
2. Image zoom or image cropped
3. Header/Footer overwriting the text in the page body.

If you use cfdocument, it is a must have hotfix for you.

You can download the hotfix here but do not forget to read the instructions to apply this hotfix. Enjoy !

Finding Image Type for a file

This is a follow up on my previous post on URLConnection APIs
We will use this API to find out the image type of a given file. Here is the udf.

<cffunction name="getMimeType">
    <cfargument name="filepath">

    <cfset var urlConn = createObject("java", "java.net.URLConnection")>
    <cfset var fileobj = fileopen(filepath, "readbinary")>
    <!--- just read the first 20 characters of the file as thats sufficient --->

    <cfset var bytes = fileread(fileobj, 20)>
    <cfset var istream = createObject("java", "java.io.ByteArrayInputStream").init(bytes)>
    <cfset fileobj.close()>
    <cfreturn urlConn.guessContentTypeFromStream(istream)>
</cffunction>

<cffunction name="GetImageType">
    <cfargument name="filepath">
    <cfset var mimetype = getMimeType(filepath)>
    <cfset var imagetype="">
    <cfif not isDefined("mimetype")>
        <cfthrow message="Not an Image file">
    </cfif>
    <cfswitch expression="#mimetype#">
        <cfcase Value="image/gif">
            <cfset imagetype="gif">
        </cfcase>
        <cfcase Value="image/x-bitmap">
            <cfset imagetype="bmp">
        </cfcase>
        <cfcase Value="image/png">
            <cfset imagetype="png">
        </cfcase>
        <cfcase Value="image/jpeg">
            <cfset imagetype="jpeg">
        </cfcase>
        <cfcase Value="image/jpg">
            <cfset imagetype="jpeg">
        </cfcase>
        <cfdefaultcase>
            <cfthrow message="Not an Image file">
        </cfdefaultcase>
    </cfswitch>
    <cfreturn imagetype>
</cffunction>

If I run the code below

<cfset filepath = "C:\temp\test.jpg">
<cfoutput>#GetImageType(filepath)#</cfoutput>

It nicely prints out "jpeg".

You should note that I used the new File IO function added in ColdFusion 8 using which I can read as many no of bytes from the file as I want. No more reading the entire file into memory.

More extensive post on File IO functions coming next !

Finding mime type of any file

Hemant recently told me about this cool API in java.net.URLConnection which can tell you the mime type for common file types.

guessContentTypeFromName(String name) - that simply checks the extension of the file name specified and gets the mime type from a static map it maintains. This can be very useful in many scenarios. The map it uses is quite extensive and contains almost all the mime types.

But this has a problem. What if some one has renamed a gif to jpg? This method will say that the file is a jpg image whereas it was a gif.

Thankfully there is another method in the same class which can address this problem.


guessContentTypeFromStream(InputStream stream) - which reads the stream, takes a look at the initial bits of the file and uses that to determine the file type.
This method does a check for the following mime types :-

- application/java-vm
- application/x-java-serialized-object
- text/html
- application/xml
- image/gif
- image/x-bitmap
- image/x-pixmap
- image/png
- image/jpeg
- image/jpg
- image/vnd.fpx
- audio/basic
- audio/x-wav

Though this method leaves out many of other common types such as mp3, it is still very useful. This makes it so easy to find out the type of an image.

ListToArray with empty elements - Part II

Yesterday I had posted about how to convert List to Array that would include empty elements. Charlie rightly pointed out a problem with that which is - it does not consider the empty elements at the end of the list.
So, a list "a,b,,c,d,," when converted to array will have only 5 elements instead of 7.

Incidently Charlie had also talked about the same some time back here and he mentions another problem with this approach. The array that is returned after split method is read only and you can not modify this array.
In order to address both these, I had to write this UDF which I guess many people would have written anyways..

<cfscript>
   function list2Array(list)
   {
       var endCommaCount = 0;
       var i = 0; var c = "";
       var stringForSplit = ""; var retList=""; var arr = "";

       for(i=len(list); i > 0; i--)
       {
           c = list.charAt(i-1);
           if(c == ',')
               endCommaCount++;
           if( c != ',' && c != ' ')
               break;
       }
       retlist = ArrayNew(1);

       if(i != 0)
       {
           stringForSplit = left(list, i);
           arr = stringForSplit.split("\s*,\s*");

           for(i = 1; i <= ArrayLen(arr); i++)
               ArrayAppend(retList, arr[i]);
       }

       for(i = 0; i < endCommaCount; i++)
           ArrayAppend(retList, "");

       return retList;
   }
</cfscript>

UPDATE : Thanks to Andrew Clark for the split tip which makes it more slim and elegant. The code above changes like

<cfscript>
   function list2Array(list)
   {
       var i = 0; 
       var retlist = ArrayNew(1);
       var arr = list.split(",", -1);
       for(i = 1; i <= ArrayLen(arr); i++)
           ArrayAppend(retList, arr[i]);
       return retList;
   }
</cfscript>

Lets test this udf and see if it works..

<cfset cflist = "a,b,,c,,d, ,">
<cfset cfarr1 = ListToArray(cflist)>
<cfset ArrayAppend(cfarr1, "END")>
<cfset cfarr2 = List2Array(cflist)>
<cfset ArrayAppend(cfarr2, "END")>

With CF Function : <cfdump var="#cfarr1#" format="text">
With UDF : <cfdump var="#cfarr2#" format="text">

And this produces this result.
With CF Function :

array - Top 6 of 6 rows

1) a
2) b
3) c
4) d
5) 
6) END

With UDF :

array - Top 9 of 9 rows

1) a
2) b
3) [empty string]
4) c
5) [empty string]
6) d
7) [empty string]
8) [empty string]
9) END

One thing to note here is that I am using new cfscript operators added in ColdFusion 8 which is so cool!!!

ListToArray with empty elements

Recently there was a request on our forum to support empty elements in ListToArray function for Scorpio i.e ColdFusion 8. it might not be done in Scorpio since it is too late to add a new function. However it can be very easily achieved in cfm itself. Here is the code to do it.

<cfset cflist="a,b,,c,,,d">
<cfset cfarr1 = ListToArray(cflist)>

<cfset cfarr2 = cflist.split(",")>
<cfset cfarr2 = cflist.split(",", -1)>

<cfdump var="#cfarr1#">
<cfdump var="#cfarr2#">

<cfscript>
for(i=1; i <= ArrayLen(cfarr2); i++)
{
 writeoutput("Element #i# : #cfarr2[i]# <br>");
}
</cfscript>

As you can see, 3rd, 5th and 6th element in this cfarr2 are empty.

ColdFusion 8 beta has arrived

I think I am the last person to say that ColdFusion 8 i.e Scorpio beta is out. Yayyy !!! :)

And that means I am back on blogsphere after a long hiatus. Seriosuly, what a fabulous journey it has been! It is a result of nearly one and half year of hard work, huge amount of research, so many design discussions, fights over nitties-gritties of features, many spec review meetings (which many a times reminded me of the college time ragging or soap box for elections at IIT kgp), many beer parties (because some one was beer boy coz he broke the build) and eating rubber chicken few times (becoz the build was broken again !!) etc etc etc.. -- all to ensure that there is enough sting in the Scorpio. :)

Everyone of us in the team is really proud of the stuff that we have built and I am sure people gonna love it.

And we are not done yet! Ashwin's cartoon says it all !!!

Use Java 5 feature in Jdk1.4

I have been getting the itch for a long tome to use Java 5 features like generics, new concurrency (java.util.concurrent), autoboxing, "for each" loop and many more. But sadly I can not do it at work because ColdFusion has to be supported on all application servers. Since many of the application servers are still on J2EE 1.4 and not all of them are compatible with Java 5, we have to write JDK1.4 compliant code. Which means that all those wonderful features of Java5 can not be used for some more time to come. Or so I thought until yesterday when I read Brian Goetz's (of "Java concurrency in practice" fame) article which says that we can use some of the Java features in JDK1.4 also.
At first I was kind of puzzled as in how could that be possible? Did they had these features and they just did not mention about it in JDK1.4 or probably they just sneaked it in 1.4 in last moment but not talk about it because of some reasons.

It dawned to me when I went halfway through the article. It looks like most of these new syntaxes did not need any change in the bytecode instruction set which means that the compiler would generate the same bytecodes or instruction set for it as it would have for equivalent code without using these features. So it should be possible to use a class created by java5 compiler in 1.4. But don't we get UnsupportedClassVersionError when we try to do that?

Here is the catch. Due to some reasons they had to change the class version and that is why this UnsupportedClassVersionError. However these guys added a flag to java compiler (which ofcourse is undocumented) that can allow this. You need to invoke javac with "-source 1.5 -target jsr14" option and this will generate 1.4 compatible bytecodes for java5 syntax. This means I can happily write my code using generics, autoboxing, for..each loop, compile it with java5 compiler with this flag and be assured that this can run on 1.4 VM also.

That leaves one more set of features i.e; new concurreny, which I really really like in java5 and want to use. Thankfully some guys at Distibuted Computing Lab at Emory University have backported java5 concurrency (java.uti.concurrent) features that can be used in java 1.3+. I have infact used it and it is awesome !

So most of the juicy stuffs that were added in java5 can run on 1.4. Off to checking it out little more. :)

java.net article on ColdFusion and JavaEE

I found this interesting article by Kola Oyedeji on java.net. He talks about integration between ColdFusion and JavaEE and how this could be of interest to JSP developers.
We in our team here have discussed it so many times that why can not ColdFusion be used by J2EE/JSP developers or enterprises that standardize on J2EE. ColdFusion is a pure J2EE solution, runs on all the major applications servers, is so feature rich, so productive that it should be a no brainer for everyone to adapt this. And to this when I add the features that we are adding in Scorpio, it makes even more sense. I don't think there is any other platform which can come even close to CF in terms of features and productivity.
At MAX, I had a conversation with one of our customers and he said something very interesting. His client wanted a pure J2EE solution and his competitors were asking for nearly 6 months to implement this. So this guy did everything in CF, bundled it as an ear (that can be deployed on application server) and shipped in flat one month !! And his client never ever thought that this was a CF application. :D

Good to see this kind of article on a java website.

Accessing password protected URL

How do you access a password protected URL (requiring basic authentication) programmatically? well, there are two ways

1. If you know that the URL is protected : This is the simple one. Since you know the URL is protected, add the required authorization header in the request and you are done. What if you are not making the socket connection but instead using java.net.URL or java.net.URLConnection to retrieve the content of url? You can use setRequestProperty() method of URLConnection to add any request header you want. So the code would look like
   
   

   
   URL url = new URL(someurlstring)
   URLConnection conn = url.openConnection();
   String encoding = new sun.misc.BASE64Encoder().encode("username:password".getBytes());
   conn.setRequestProperty ("Authorization", "Basic " + encoding);
   InputStream in = conn.getInputStream()
   ...
   

   
   


2. What if you dont know in advance whether the URL is protected or not? You can not arbitrarily add the authorization heder for all the URLs. The answer to this is "java.net.Authenticator". You need to install an instance of java.net.Authenticator using setDefault() method of Authenticator. Whenever URLConnection sees that the url is protected, it will use this installed authenticator to get the username and password and set the required authorization header automatically.
   Here is how the code snippet for doing this
   
   

   
   Authenticator.setDefault(new MyAuthenticator());
   ....
   
   public class MyAuthenticator extends Authenticator
   {
    protected PasswordAuthentication getPasswordAuthentication()
    {
        return new PasswordAuthentication ("username", "password");
    }
   }
   

   
   This particular technique is particularly useful for people who need a workaround for retrieving password protected images in cfdocument. Here is what you need to do.
   
   
   
   • Use the source below to create Authenicator class
     

     
     import java.net.*;
     public class MyAuthenticator extends Authenticator
     {
         private String user;
         private String passwd;
     
         public MyAuthenticator(String user, String passwd)
         {
             this.user = user;
             this.passwd = passwd;
         }
     
         protected PasswordAuthentication getPasswordAuthentication()
         {
             return new PasswordAuthentication(user, passwd.toCharArray());
         }
     }
     

     
     Compile this and put MyAuthenticator.class in wwwroot/WEB-INF/classes.
     
   
   
   • Put the following code in your application.cfm or application.cfc
     

     
     <cfif not IsDefined("Application.authenticator")>
           <!--- Do initializations --->
           <cfset authenticator= createObject("java", "java.net.Authenticator")>
           <cfset myauthenticator = createObject("java", "MyAuthenticator").init("username", "password")>
           <cfset authenticator.setDefault(myauthenticator)>
           <cfset Application.authenticator=myauthenticator>
     </cfif>
     

     
     replace "username" and "password" with actual username and password.
     
   
   
   • give the request for the cfdocument page for which you were getting the red-x for image. The images should appear this time.
     
   
   

A page in CFSwitch-CFCase' diary

Undoubtedly I am one of the most powerful programming construct of ColdFusion language. And definitely more powerful than all my cousins in other programming languages. All my cousins and even my brother (read java switch-case) work only on integers but I work on almost every kind of objects be it string or any numeric data or even date. Since I can work on almost all datatypes, I make cf developers life so much simpler. I was created this way because the world I was going to be in was UnTyped - where every one was equal and where there was no discrimination between datatypes and I really thank God (ColdFusion Architects) for creating such a wonderful world.
Recently I heard someone talking about me that I am not as fast as my cousins. Some one even talked about ignoring me and taking help of kiddo cfif-cfelse. I dont want to say anything against anyone coz I know "with great power comes great responsibility err..cost".

Its sheer hardwork that makes me so much more powerful than all my cousins. This is what I do when any object comes to me. First I try to see if it is numeric. I do that because that is what all my cousins are used to and I have to remain as fast as them in that case. If it is not numeric, then I check if it can be date. If it is not even date, I try converting it in String. Once I arrive at the data, I use my own data structure to match it with appropriate CASE. So when the data is string, i will take some more time as compared to what i will take when data is numeric. Is that so bad? My cousins dont even do that!

Sometimes I compare myself to a busy lawyer who likes working on many CASEs. I dont like to work just for 2-3 cases. I prefer my grandson cfif-cfelse take care of those small no of cases.
I hope someday people will read these pages and if even after reading this they think that I am too slow and I should be ignored I only want to say "God, forgive them, for they dont know what they are doing!".

Performance Tips : ColdFusion List

how many of you have written/seen code like this?

<cfset mylist="jan,feb,mar,apr,may,jun,jul,sep,oct,nov,dec">
<cfloop from="1" to=#ListLen(mylist)# index="i">
 <cfset month = ListGetAt(mylist, i)>
 <!--- do something with this month --->
 <cfoutput>#month#</cfoutput>
</cfloop>

While there is nothing wrong with it syntactically or functionally, performance wise it is very poor. Why? ColdFusion list is nothing but String (delimited by delimiter). ColdFusion does not have any way to build any intelligence to keep it in any other datastructure because you can use it like a normal string also. So what happens when you call any List function on this string? We parse the string using the delimiter and get the delimited tokens and process that.
Now lets take ListGetAt(list, index) function. It will keep parsing and getting the token unless it reaches the required index. Imagine doing in a loop. We will be parsing the same string again and again and traversing from the beginning everytime till we reach the next loop index. So, in the Nth iteration, it will start from beginning and tokenize N times. Thus by the time you have completed the loop, you have parsed/tokenized the string N*(N+1)/2 times. Isn't that too costly? Lesson - Never ever use ListGetAt() in a loop. Either iterate using OR convert the list into array using ListToArray() and iterate over it. Using cfloop is the most optimized way to do this.

Even if you are not iterating over list but you need to call ListGetAt() many times, it is better to convert it to array and then search the index in that.

Same thing applies to search functions like ListFind, ListContains etc. If you need to call these multiple times on the same list, you will be better off converting the list to array and searching in that.

If you need to append many items to the list, then also you will get a better performance by converting the list to array and doing all appends on that.

This does not mean you should not use list at all or you should always convert the list to array and work on that. If the number of operations that you are doing on the list is less, you should stick to list because converting the list to array is also costly. If you are inserting an element in the middle of list, list will be better than array in most cases.

ColdFusion Array : pass by reference or Value.

Ben rightly pointed out in my last post that since ColdFusion array are always passed by value, the second technique can not be used if you want to build the array over multiple method calls. In each of the function call, ColdFusion will create a copy of the array passed and that cost (cost of creating a new instance and copying old array to new one) might even exceed the cost of appending string.

ColdFusion array is one unique data structure in ColdFusion. Unique in the sense that this is the only data structure that is passed by value and not by reference. I do feel it is a limitation sometimes but thats legacy and can not be changed. (You would not want us to break your code. Would you? ;) )

There does exist one hack if you absolutely need to pass the array by reference. Here is a code snippet that uses pass by reference.

<cfset x = ArrayNew(1)>
<cfloop from=1 to=5 index=i>
 <cfset Arrayappend(x,"something")>
</cfloop>

<cfset x = CreateObject("java", "java.util.ArrayList").init(x)>
<cfset foo(x)>
<cfset foo(x)>
<cfset x[8] = "after the method call">
<cfset x[9] = "end of method call">
<cfset foo(x)>
<cfdump var="#x#">

<cffunction name="foo">
 <cfargument name="arr">
 <cfset ArrayAppend(arr, "from function")>
</cffunction>

So what did we do here? We created an ArrayList from the ColdFusion array. Since ColdFusion Array is an implementation of java.util.List, almost all Array functions work on all implementations of java.util.List. And this list implementation will not be passed by value but will be passed by reference. Thats the power of using java in ColdFusion !

The Mystery of "Too many open files"

Last week we saw an interesting problem while running the regression tests on Linux. Immediately after the tests were started, the VM started throwing error "FileNotFoundException : Too many open files". Ofcourse the files were there but the VM was trying to say that there are too many file descriptors open which were not closed. It was hitting the open file limit set by the OS. This was kind of absurd because we always close all the files that we open. And moreover we had never seen this problem before. So we started suspecting 1.6 VM on which we were running it.

Immediately after this, it got worse. The exception changed into "SocketException: Too many open files" and all the socket connection started getting rejected. So merely after serving 100 requests, the server was down to its knees.

A quick google search suggested to increase the open file descriptor limit on the machine.

"ulimit -aH" that gives the max limit for number of open files returned 1024.

I added every possible way to increase it. Here are few

1. In /etc/security/limits.conf
* soft nofile 1024
* hard nofile 65535
2. Increase ulimit by "ulimit -n 65535"
3. echo 65535 > /proc/sys/fs/file-max
4. In /etc/sysctl.conf
fs.file-max=65535

Increasing file descriptor limit did not help much either. Even after increasing this limit, we were still getting this error.And then Sanjeev (another brilliant CF engineer with an amazing sense of humour) cracked it !!
Just before we started getting these errors, there was another error which I had overlooked assuming it was test problem which infact was the clue.
The error was something like
coldfusion.jsp.CompilationFailedException: Errors reported by Java compiler: error: error reading /opt/coldfusionscorpio/lib/./././././././././././././././././././././././././././././
././././././././././././././././././././././././././././././././././././././././././.
/././././././././././././././././././././././././././././././././././././././././././
././././././././././././././././././././././././././././././././././././././././././.
/././././././././././././././././././././././././././././././././././././././././././
././././././././././././././././././././././././././././././././././././././././././.
/././././././././././././././././././././././././././././././././././././././././././
././././././././././././././././././././././././././././././././././././././././././.
/././././././././././././././././././././././././././././././././././././././././././
././././././././././././././././././././././././././././././././././././././././././.
/././././././././././././././././././././././././././././././././././././././././././
././././././././././././././././././././././././././././././././././....
at coldfusion.jsp.JavaCompiler.compileClass(JavaCompiler.java:138)

Sanjeev ran it through the debugger and he nailed the culprit. It was something which no one had even suspected. It was javac. :)
We had hit upon a Sun's bug in javac where if the classpath contains a jar which has a manifest and manifest contains classpath entry with relative paths of other jars as well as path to itself, javac goes in an infinite loop. (Sun's bug no 6400872, 6446657, 6456960, 6485027, 6206485)

I dont have the source for javac to pin point what exactly it did, but probably it kept opening all the entries in the classpath and because of the stack overflow it could not close those files - hence reaching the max limit of open file descriptors. This was a third party jar that had this manifest entry. Once we found the problem, the solution was simple - just remove the classpath entry from the manifest! Whew !!!

String Concatenation optimization

String concatenation is one of the most common, but, a pretty expensive operation. It can hit the performance severly if not used correctly. The performance goes down drastically if you append strings using '&' OR ListAppend() in a loop. I have seen application performance improving by 50-100% just by optimizing String concatenation (though that depends on how much concatenation is used in the app). So what do you about it?
The simplest and the most optimized way to do these append operations is using java's StringBuffer. (I am sure you must be aware of it but still.. :)) .
The code would look like

<cfset sb = createObject("java", "java.lang.StringBuffer")>
<cfloop from=1 to=100 index=i>
  <cfset sb.append("something")>
  <cfset sb.append(i)>
</cfloop>
<cfset result=sb.toString()>

Sometimes I feel that we should have a datastructure like this in ColdFusion directly but again I think whats wrong with using StringBuffer? Its like any other function which we would create. Isn't it so?

If you are a puristic and don't want to use any java API inside your CF app, there is another simple way to do the same thing. It uses ColdFusion Array to do the same thing what StringBuffer does. Instead of appending the string in the buffer, you can append to the array using ArrayAppend() and then once you are done and want to get the string back, use ArrayToList() with empty string ("") as delimiter. The code would look like

<cfset arr = ArrayNew(1)>
<cfloop from=1 to=100 index=i>
  <cfset ArrayAppend(arr, "something")>
  <cfset ArrayAppend(arr, i)>
</cfloop>
<cfset result=ArrayToList(arr,"")>

This would give a much better performance as compared to concatenation using '&' or using ListAppend() but will have lower performance as compared to StringBuffer. That is because of the overhead of Array object creation and array append operation. ArrayToList() will anyway create the string buffer and append the strings

You should use '&' or ListAppend() only when there are only 2-3 strings to be concatenated. Otherwise always use either of the two techniques above.

Optimizations with literals

Look at these two pieces of code carefully. Is there any difference between these two apart from the fact that the second one is shorter?

<cfset x = "sun,mercury,venus,earth,mars,jupiter,saturn,uranus,pluto,neptune">
<cfset y = ListSort(x,"text")>

and

<cfset y = ListSort("sun,mercury,venus,earth,mars,jupiter,saturn,uranus,pluto,neptune","text")>

If you think there is not much, read on.

There is a huge difference between these two piece of code - in terms of performance. The second one will have much better performance as compared to the first one. How?? Because the ListSort() method in the second case will not even be executed in the page request. Still scratching your head?

It is because of the intelligence that is built into CFML compiler (really superb code written by Edwin Smith). During compilation, it analyzes all the code and wherever there is a literal or functions executing literals, it tries to optimize it. In the second piece here, it sees that ListSort method is being called on a literal and it can be done statically. So compiler will execute this call during compilation itself and set the sorted value on 'y'. During the page execution, only thing that will get executed will be an assignment. Smart.. isn't it? Even java compiler, which does quite a many compile time optimization for java source files, does not have this intelligence of executing calls at compile time :)

It is not only about 'ListSort'. This is true for most of those CF functions which can work on a literal and can return a literal.

Extend CF native Objects - Harnessing Java

Since Coldfusion native objects are java objects, you can harness the java APIs to extend the functionality of these objects. In this post we will take CF Array and see how we can use these APIs to get some cool functionalities from them.

ColdFusion array is actually an implementation of java list (java.util.List). So all the list methods are actually available for Array.
CF provides most of the list functionality using Array functions but there are few things possible with java list which you can not do directly with CF functions.

1. Merge : Lets say you create two arrays and you want to merge these two arrays to create one bigger array. There is no CF function to do this.
However You can call List.addAll() methods to do it.

Here is how it would look.

I am creating array this way just because it is easier and I don't have to write whole lot of code :)

<cfset y = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
<cfset z = ListToArray("dean,manju,jason,tim")>
<cfset y.addAll(z)>
<cfdump var="#y#">

2. Merge in middle : Lets say you want to add the second array somewhere in the middle of first array say after 4 elements. The code would look like

<cfset y = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
<cfset z = ListToArray("dean,manju,jason,tim")>
<cfset y.addAll(4, z)>
<cfdump var="#y#">

3. Search : I have heard people complaining that there is no find method in Array. Actually you had it all the time. Just that it was hidden :)
You can use List.Contains() or List.indexOf() methods to achieve that. Here is the code.

<cfset y = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
<cfoutput>Contains Hemant: #y.contains("hemant")#</cfoutput>
<cfoutput>Index of damon : #y.indexof("damon")#</cfoutput>

Please note that the java index starts at 0 where as CF index starts at 1. So the index here will be 2. You must also note that since java is case sensitive, this search will also be case sensitive. To build case insensitiveness, you will have to make the list as well as the search in same case - either uppercase it or lowercase it.

4. Search whole array : You can also search if all the elements of one array are present in another array using containsAll() method of java list.

<cfset y = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
<cfset z = ListToArray("ram,prank,rupesh")>

<cfoutput> y Contains z: #y.containsAll(z)#</cfoutput>

5. Equality check : You can find out if two arrays are same or not using list's equals method.

<cfset y = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
<cfset z = ListToArray("dean, manju,jason,tim")>
<cfset x = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
<cfoutput>x equals y : #x.equals(y)#</cfoutput>
<cfoutput>x equals z : #x.equals(z)#</cfoutput>

6. RemoveAll : You can remove all the elements of one array from the second array using removeAll()

<cfset y = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
before removal <cfdump var="#y#">

<cfset z = ListToArray("ram,prank,tom")>
<cfset y.removeAll(z)>
After removal <cfdump var="#y#">

Go ahead and play around with it!