Accessing password protected URL

How do you access a password protected URL (requiring basic authentication) programmatically? well, there are two ways

1. If you know that the URL is protected : This is the simple one. Since you know the URL is protected, add the required authorization header in the request and you are done. What if you are not making the socket connection but instead using java.net.URL or java.net.URLConnection to retrieve the content of url? You can use setRequestProperty() method of URLConnection to add any request header you want. So the code would look like
   
   

   
   URL url = new URL(someurlstring)
   URLConnection conn = url.openConnection();
   String encoding = new sun.misc.BASE64Encoder().encode("username:password".getBytes());
   conn.setRequestProperty ("Authorization", "Basic " + encoding);
   InputStream in = conn.getInputStream()
   ...
   

   
   


2. What if you dont know in advance whether the URL is protected or not? You can not arbitrarily add the authorization heder for all the URLs. The answer to this is "java.net.Authenticator". You need to install an instance of java.net.Authenticator using setDefault() method of Authenticator. Whenever URLConnection sees that the url is protected, it will use this installed authenticator to get the username and password and set the required authorization header automatically.
   Here is how the code snippet for doing this
   
   

   
   Authenticator.setDefault(new MyAuthenticator());
   ....
   
   public class MyAuthenticator extends Authenticator
   {
    protected PasswordAuthentication getPasswordAuthentication()
    {
        return new PasswordAuthentication ("username", "password");
    }
   }
   

   
   This particular technique is particularly useful for people who need a workaround for retrieving password protected images in cfdocument. Here is what you need to do.
   
   
   
   • Use the source below to create Authenicator class
     

     
     import java.net.*;
     public class MyAuthenticator extends Authenticator
     {
         private String user;
         private String passwd;
     
         public MyAuthenticator(String user, String passwd)
         {
             this.user = user;
             this.passwd = passwd;
         }
     
         protected PasswordAuthentication getPasswordAuthentication()
         {
             return new PasswordAuthentication(user, passwd.toCharArray());
         }
     }
     

     
     Compile this and put MyAuthenticator.class in wwwroot/WEB-INF/classes.
     
   
   
   • Put the following code in your application.cfm or application.cfc
     

     
     <cfif not IsDefined("Application.authenticator")>
           <!--- Do initializations --->
           <cfset authenticator= createObject("java", "java.net.Authenticator")>
           <cfset myauthenticator = createObject("java", "MyAuthenticator").init("username", "password")>
           <cfset authenticator.setDefault(myauthenticator)>
           <cfset Application.authenticator=myauthenticator>
     </cfif>
     

     
     replace "username" and "password" with actual username and password.
     
   
   
   • give the request for the cfdocument page for which you were getting the red-x for image. The images should appear this time.
     
   
   

A page in CFSwitch-CFCase' diary

Undoubtedly I am one of the most powerful programming construct of ColdFusion language. And definitely more powerful than all my cousins in other programming languages. All my cousins and even my brother (read java switch-case) work only on integers but I work on almost every kind of objects be it string or any numeric data or even date. Since I can work on almost all datatypes, I make cf developers life so much simpler. I was created this way because the world I was going to be in was UnTyped - where every one was equal and where there was no discrimination between datatypes and I really thank God (ColdFusion Architects) for creating such a wonderful world.
Recently I heard someone talking about me that I am not as fast as my cousins. Some one even talked about ignoring me and taking help of kiddo cfif-cfelse. I dont want to say anything against anyone coz I know "with great power comes great responsibility err..cost".

Its sheer hardwork that makes me so much more powerful than all my cousins. This is what I do when any object comes to me. First I try to see if it is numeric. I do that because that is what all my cousins are used to and I have to remain as fast as them in that case. If it is not numeric, then I check if it can be date. If it is not even date, I try converting it in String. Once I arrive at the data, I use my own data structure to match it with appropriate CASE. So when the data is string, i will take some more time as compared to what i will take when data is numeric. Is that so bad? My cousins dont even do that!

Sometimes I compare myself to a busy lawyer who likes working on many CASEs. I dont like to work just for 2-3 cases. I prefer my grandson cfif-cfelse take care of those small no of cases.
I hope someday people will read these pages and if even after reading this they think that I am too slow and I should be ignored I only want to say "God, forgive them, for they dont know what they are doing!".

Performance Tips : ColdFusion List

how many of you have written/seen code like this?

<cfset mylist="jan,feb,mar,apr,may,jun,jul,sep,oct,nov,dec">
<cfloop from="1" to=#ListLen(mylist)# index="i">
 <cfset month = ListGetAt(mylist, i)>
 <!--- do something with this month --->
 <cfoutput>#month#</cfoutput>
</cfloop>

While there is nothing wrong with it syntactically or functionally, performance wise it is very poor. Why? ColdFusion list is nothing but String (delimited by delimiter). ColdFusion does not have any way to build any intelligence to keep it in any other datastructure because you can use it like a normal string also. So what happens when you call any List function on this string? We parse the string using the delimiter and get the delimited tokens and process that.
Now lets take ListGetAt(list, index) function. It will keep parsing and getting the token unless it reaches the required index. Imagine doing in a loop. We will be parsing the same string again and again and traversing from the beginning everytime till we reach the next loop index. So, in the Nth iteration, it will start from beginning and tokenize N times. Thus by the time you have completed the loop, you have parsed/tokenized the string N*(N+1)/2 times. Isn't that too costly? Lesson - Never ever use ListGetAt() in a loop. Either iterate using OR convert the list into array using ListToArray() and iterate over it. Using cfloop is the most optimized way to do this.

Even if you are not iterating over list but you need to call ListGetAt() many times, it is better to convert it to array and then search the index in that.

Same thing applies to search functions like ListFind, ListContains etc. If you need to call these multiple times on the same list, you will be better off converting the list to array and searching in that.

If you need to append many items to the list, then also you will get a better performance by converting the list to array and doing all appends on that.

This does not mean you should not use list at all or you should always convert the list to array and work on that. If the number of operations that you are doing on the list is less, you should stick to list because converting the list to array is also costly. If you are inserting an element in the middle of list, list will be better than array in most cases.

ColdFusion Array : pass by reference or Value.

Ben rightly pointed out in my last post that since ColdFusion array are always passed by value, the second technique can not be used if you want to build the array over multiple method calls. In each of the function call, ColdFusion will create a copy of the array passed and that cost (cost of creating a new instance and copying old array to new one) might even exceed the cost of appending string.

ColdFusion array is one unique data structure in ColdFusion. Unique in the sense that this is the only data structure that is passed by value and not by reference. I do feel it is a limitation sometimes but thats legacy and can not be changed. (You would not want us to break your code. Would you? ;) )

There does exist one hack if you absolutely need to pass the array by reference. Here is a code snippet that uses pass by reference.

<cfset x = ArrayNew(1)>
<cfloop from=1 to=5 index=i>
 <cfset Arrayappend(x,"something")>
</cfloop>

<cfset x = CreateObject("java", "java.util.ArrayList").init(x)>
<cfset foo(x)>
<cfset foo(x)>
<cfset x[8] = "after the method call">
<cfset x[9] = "end of method call">
<cfset foo(x)>
<cfdump var="#x#">

<cffunction name="foo">
 <cfargument name="arr">
 <cfset ArrayAppend(arr, "from function")>
</cffunction>

So what did we do here? We created an ArrayList from the ColdFusion array. Since ColdFusion Array is an implementation of java.util.List, almost all Array functions work on all implementations of java.util.List. And this list implementation will not be passed by value but will be passed by reference. Thats the power of using java in ColdFusion !

The Mystery of "Too many open files"

Last week we saw an interesting problem while running the regression tests on Linux. Immediately after the tests were started, the VM started throwing error "FileNotFoundException : Too many open files". Ofcourse the files were there but the VM was trying to say that there are too many file descriptors open which were not closed. It was hitting the open file limit set by the OS. This was kind of absurd because we always close all the files that we open. And moreover we had never seen this problem before. So we started suspecting 1.6 VM on which we were running it.

Immediately after this, it got worse. The exception changed into "SocketException: Too many open files" and all the socket connection started getting rejected. So merely after serving 100 requests, the server was down to its knees.

A quick google search suggested to increase the open file descriptor limit on the machine.

"ulimit -aH" that gives the max limit for number of open files returned 1024.

I added every possible way to increase it. Here are few

1. In /etc/security/limits.conf
* soft nofile 1024
* hard nofile 65535
2. Increase ulimit by "ulimit -n 65535"
3. echo 65535 > /proc/sys/fs/file-max
4. In /etc/sysctl.conf
fs.file-max=65535

Increasing file descriptor limit did not help much either. Even after increasing this limit, we were still getting this error.And then Sanjeev (another brilliant CF engineer with an amazing sense of humour) cracked it !!
Just before we started getting these errors, there was another error which I had overlooked assuming it was test problem which infact was the clue.
The error was something like
coldfusion.jsp.CompilationFailedException: Errors reported by Java compiler: error: error reading /opt/coldfusionscorpio/lib/./././././././././././././././././././././././././././././
././././././././././././././././././././././././././././././././././././././././././.
/././././././././././././././././././././././././././././././././././././././././././
././././././././././././././././././././././././././././././././././././././././././.
/././././././././././././././././././././././././././././././././././././././././././
././././././././././././././././././././././././././././././././././././././././././.
/././././././././././././././././././././././././././././././././././././././././././
././././././././././././././././././././././././././././././././././././././././././.
/././././././././././././././././././././././././././././././././././././././././././
././././././././././././././././././././././././././././././././././././././././././.
/././././././././././././././././././././././././././././././././././././././././././
././././././././././././././././././././././././././././././././././....
at coldfusion.jsp.JavaCompiler.compileClass(JavaCompiler.java:138)

Sanjeev ran it through the debugger and he nailed the culprit. It was something which no one had even suspected. It was javac. :)
We had hit upon a Sun's bug in javac where if the classpath contains a jar which has a manifest and manifest contains classpath entry with relative paths of other jars as well as path to itself, javac goes in an infinite loop. (Sun's bug no 6400872, 6446657, 6456960, 6485027, 6206485)

I dont have the source for javac to pin point what exactly it did, but probably it kept opening all the entries in the classpath and because of the stack overflow it could not close those files - hence reaching the max limit of open file descriptors. This was a third party jar that had this manifest entry. Once we found the problem, the solution was simple - just remove the classpath entry from the manifest! Whew !!!

String Concatenation optimization

String concatenation is one of the most common, but, a pretty expensive operation. It can hit the performance severly if not used correctly. The performance goes down drastically if you append strings using '&' OR ListAppend() in a loop. I have seen application performance improving by 50-100% just by optimizing String concatenation (though that depends on how much concatenation is used in the app). So what do you about it?
The simplest and the most optimized way to do these append operations is using java's StringBuffer. (I am sure you must be aware of it but still.. :)) .
The code would look like

<cfset sb = createObject("java", "java.lang.StringBuffer")>
<cfloop from=1 to=100 index=i>
  <cfset sb.append("something")>
  <cfset sb.append(i)>
</cfloop>
<cfset result=sb.toString()>

Sometimes I feel that we should have a datastructure like this in ColdFusion directly but again I think whats wrong with using StringBuffer? Its like any other function which we would create. Isn't it so?

If you are a puristic and don't want to use any java API inside your CF app, there is another simple way to do the same thing. It uses ColdFusion Array to do the same thing what StringBuffer does. Instead of appending the string in the buffer, you can append to the array using ArrayAppend() and then once you are done and want to get the string back, use ArrayToList() with empty string ("") as delimiter. The code would look like

<cfset arr = ArrayNew(1)>
<cfloop from=1 to=100 index=i>
  <cfset ArrayAppend(arr, "something")>
  <cfset ArrayAppend(arr, i)>
</cfloop>
<cfset result=ArrayToList(arr,"")>

This would give a much better performance as compared to concatenation using '&' or using ListAppend() but will have lower performance as compared to StringBuffer. That is because of the overhead of Array object creation and array append operation. ArrayToList() will anyway create the string buffer and append the strings

You should use '&' or ListAppend() only when there are only 2-3 strings to be concatenated. Otherwise always use either of the two techniques above.

Optimizations with literals

Look at these two pieces of code carefully. Is there any difference between these two apart from the fact that the second one is shorter?

<cfset x = "sun,mercury,venus,earth,mars,jupiter,saturn,uranus,pluto,neptune">
<cfset y = ListSort(x,"text")>

and

<cfset y = ListSort("sun,mercury,venus,earth,mars,jupiter,saturn,uranus,pluto,neptune","text")>

If you think there is not much, read on.

There is a huge difference between these two piece of code - in terms of performance. The second one will have much better performance as compared to the first one. How?? Because the ListSort() method in the second case will not even be executed in the page request. Still scratching your head?

It is because of the intelligence that is built into CFML compiler (really superb code written by Edwin Smith). During compilation, it analyzes all the code and wherever there is a literal or functions executing literals, it tries to optimize it. In the second piece here, it sees that ListSort method is being called on a literal and it can be done statically. So compiler will execute this call during compilation itself and set the sorted value on 'y'. During the page execution, only thing that will get executed will be an assignment. Smart.. isn't it? Even java compiler, which does quite a many compile time optimization for java source files, does not have this intelligence of executing calls at compile time :)

It is not only about 'ListSort'. This is true for most of those CF functions which can work on a literal and can return a literal.

Extend CF native Objects - Harnessing Java

Since Coldfusion native objects are java objects, you can harness the java APIs to extend the functionality of these objects. In this post we will take CF Array and see how we can use these APIs to get some cool functionalities from them.

ColdFusion array is actually an implementation of java list (java.util.List). So all the list methods are actually available for Array.
CF provides most of the list functionality using Array functions but there are few things possible with java list which you can not do directly with CF functions.

1. Merge : Lets say you create two arrays and you want to merge these two arrays to create one bigger array. There is no CF function to do this.
However You can call List.addAll() methods to do it.

Here is how it would look.

I am creating array this way just because it is easier and I don't have to write whole lot of code :)

<cfset y = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
<cfset z = ListToArray("dean,manju,jason,tim")>
<cfset y.addAll(z)>
<cfdump var="#y#">

2. Merge in middle : Lets say you want to add the second array somewhere in the middle of first array say after 4 elements. The code would look like

<cfset y = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
<cfset z = ListToArray("dean,manju,jason,tim")>
<cfset y.addAll(4, z)>
<cfdump var="#y#">

3. Search : I have heard people complaining that there is no find method in Array. Actually you had it all the time. Just that it was hidden :)
You can use List.Contains() or List.indexOf() methods to achieve that. Here is the code.

<cfset y = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
<cfoutput>Contains Hemant: #y.contains("hemant")#</cfoutput>
<cfoutput>Index of damon : #y.indexof("damon")#</cfoutput>

Please note that the java index starts at 0 where as CF index starts at 1. So the index here will be 2. You must also note that since java is case sensitive, this search will also be case sensitive. To build case insensitiveness, you will have to make the list as well as the search in same case - either uppercase it or lowercase it.

4. Search whole array : You can also search if all the elements of one array are present in another array using containsAll() method of java list.

<cfset y = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
<cfset z = ListToArray("ram,prank,rupesh")>

<cfoutput> y Contains z: #y.containsAll(z)#</cfoutput>

5. Equality check : You can find out if two arrays are same or not using list's equals method.

<cfset y = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
<cfset z = ListToArray("dean, manju,jason,tim")>
<cfset x = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
<cfoutput>x equals y : #x.equals(y)#</cfoutput>
<cfoutput>x equals z : #x.equals(z)#</cfoutput>

6. RemoveAll : You can remove all the elements of one array from the second array using removeAll()

<cfset y = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
before removal <cfdump var="#y#">

<cfset z = ListToArray("ram,prank,tom")>
<cfset y.removeAll(z)>
After removal <cfdump var="#y#">

Go ahead and play around with it!

Scorpion Queen !

When Tim asked for spiffy Scorpio logo, a few CF engineers (Sanjeev, Chandan, Jayesh, Sandeep and Vamsee. special mention - Hemant & Praveen.) got together to create her. Of course they didnt use Photoshop. We are very much used to Whiteboard :)



Did you notice the cool scorpio tatoo she is wearing. And No, she would not execute <CFLAPDANCE /> !!!

Connecting to URL from behind a proxy server

I needed a way to set the proxy information on URL/URLConnection and I could not find any good way. One simple way that java recommends is to set the information as system property. These properties are "http.proxyHost" and "http.proxyPort".

So it can either be set as jvm arguments like

-DproxySet=true -Dhttp.proxyHost=proxyIP -DproxyPort=port

or set them in the code using System.setProperty()

However since this is a system property, it gets set on the VM itself and hence it is not dynamic. In ColdFusion, since tags like 'cfhttp' keep them dynamic, I wanted a similar behaviour. After looking around for a while, I noticed that this capability was added in Java 1.5 aka Tiger release. (Is it only me? I keep hitting things which I feel is lacking in java API and then I find them added in 1.5 :) )

In Java 1.5, you can use

URLConnection conn = url.openConnection(proxy); // added in 1.5

where proxy is an object of java.net.Proxy. Pretty neat.


However this was not a solution for me as we are still developing on JDK1.4 (need to consider all the application servers that we have to support). I stumbled upon an interesting article by Daniel Horn who faced the exact same problem. And guess what he did? Since you send an HTTP request to the proxy and then proxy sends out the actual request, he created the URL object by passing proxyHost and proxyPort as IP and port and then he gave the target url string as 'file' argument. This is what he did.

String actualUrl = "http://www.adobe.com";
URL url = new URL("http", proxyHost, proxyPort, actualUrl);
URLConnection conn = url.openConnection();
..
..

And it works ! Brilliant !! I wonder why this is not documented in the java API.

A workaround for cfdocument missing images

This is with reference to the post Missing images in CFDocument. There are some cases when the images are locally on the machine running ColdFusion but even then cfdocument is not able to show the images. The reasons could be
1) ColdFusion is behind a firewall because of which it is not able to send any HTTP request (even though to itself).
2) The images are under a protected directory which needs authentication. Since cfdocument can not send authentication information currently, it is not able to fetch the image.
3) ColdFusion is using HTTPS and it is not configured properly to trust itself. So cfdocument can not send a https request to itself.
4) Any other reason which is preventing CFDocument from sending request to the local server.

If the images are on local machine, it is possible to use the file url for images (or CSS,javascripts, etc). CFDocument in that case will not send requests for the images over HTTP and fetch the image directly from the file system. Here is a simple way to use the file url.

<cfdocument format="pdf">
 <cfoutput>
   Some html content
   <br>
   <img src=#localUrl("img1.gif")#><br>
   <img src=#localUrl("images/img.jpg")#>
 </cfoutput>
</cfdocument>

<cffunction name="localUrl" >
  <cfargument name="file" />
  <cfset var fpath = ExpandPath(file)>
  <cfset var f="">
  <cfset f = createObject("java", "java.io.File")>
  <cfset f.init(fpath)>
  <cfreturn f.toUrl().toString()>
</cffunction>

basically here I have an UDF which converts any path to local URL and then I am using that UDF in 'src' attribute of image. This can be used to fetch images, css or any other similar contents from the local machine. You should note the <cfoutput> right under cfdocumet tag that allows the evaluation of UDF before it goes to cfdocument body.. This workaround is applicable only when the these contents are present on the same machine as ColdFusion.

This workaround has another advantage too. Normally when CFDocument body has any images, it fetches those images by sending HTTP request to the local server which is served by web threads. This has its own overhead. In a way, CFDocument uses server resource for getting something which is available locally on the server. This resource can instead be used to serve actual client http requests. Converting the image path to local urls will not go through HTTP and thus should have a better performance.

Update to CFThread POC tags

Damon posted an update to the CFThread proof of concept tag that was published some time back. I wanted to do it for a long time but was busy in other Scorpio features and had to keep delaying this. Neverthless better late than never :) This update includes "thread safety" while retaining the old syntax in the original post. Thanks to Dan Switzer, Derek, Mike and all others who provided the valuable feedback on it !

Whats there in the update

• Thread very much acts like a function and some time little more than that.
• Any attribute can be passed to cfthread. These attributes can be accessed using 'attributes.<varname>'. These attributes are passed by value and hence they are completely thread safe.
• Any variable unless defined with a scope prefix goes in thread local scope. So this is slightly different from function. Like function, variable defined in var scope will also go in thread local scope. So in following snippet, x, y, z and a all will be in thread local scope. However since b is directly used inside thread without defining it, it will use it from the page scope.
  

<cfset x = 10>
<cfset b = 20>
<cfthread name="t1">
 <cfset var y = 10>
 <cfset x = 20>
 <cfset z = y*x>
 ...
 <cfset a = z*Variables.x>
 <cfset a = a/b>
</cfthread>

• All other scope variables will be accessible using the appropriate prefix like "Variables", "request", "Server" etc.

• Threads have an another scope called thread scope in which only the owner thread can write but all other can read. This scope can be accessed using 'thread' prefix by owner thread or using the thread name by other thread or main page thread. This scope will be useful when the owner thread wants to put some data in it which needs to accessed by other thread. The same thing could have been achieved by all the threads writing to the page scope in its own variable but that needs some discipline from developers and is error prone. Having a separate thread scope which other can read makes it threadsafe and easier for developers.
  

• If thread name is dynamic, it will be everyone's question how to access that thread data from another thread or main page. It can be easily done using Variables[threadname].xxx. See the example below.

• Threads can continue even after the main page is done.

Below is a sample cf code that uses cfthread. (modified version of Dan's example. Thanks Dan !!).

<cfset CRLF = CHR(13) & CHR(10)>
<cfset sDirectory = expandPath(".") & "\tmp" />

<cfsavecontent variable="sContent">XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX </cfsavecontent>
<cfloop index="loopcounter" from="1" to="50">
<cfset threadname = "thread_" & #loopcounter#>
<cfthread name="#threadname#" filename="file#loopcounter#.txt" counter="#loopcounter#">
<cfset sOutput = "Content written by thread " & #attributes.counter#>
<cfloop index="i" from="1" to="100">
   <cfset sOutput = sOutput & sContent>
</cfloop>
<cfset sOutput>
<cfset dest="#sDirectory#\tmp_#attributes.filename#">
<cffile action="write" file="#dest#" output="#sOutput#" />
<cfset thread.msg="Message from thread "&#attributes.counter#>
</cfthread>

</cfloop>
<cfloop index="loopcounter" from="1" to="50">
<cfset threadname = "thread_" & #loopcounter#>
<cfjoin thread="#threadname#"/>
<cfoutput><br>#threadname# output : #Variables[threadname].msg#</cfoutput>

</cfloop>
<br> Work Complete!<br>

Feel free to play around with this tag and send feedbacks if you have any. Once again, as Damon said, this is not a CF feature and hence it is unsupported. On whether it will make it to the Scorpio or not, I can not guarantee anything but stay tuned ;)
Enjoy !

Handling J2EE session with cookies disabled

Someone recently reported that when cookies are disabled and J2EE session is enabled, his sessions are not maintained in case of POST request. As per that, CF or rather the app server always creates a new session everytime. His code looked like


<form method="post" action="test.cfm?#session.urltoken#">
...
<input type="submit" value="Submit" >
</form>


can you see whats wrong with above code?

As per the Servlet spec of J2EE, when cookies are disabled, session is maintained by url rewriting and that is done by appending ';jsessionid=' to the URI. Note the semicolon ';' before 'jsessionid'.

In the above code, it is appending session.urltoken which looks like 'CFID=1600&CFTOKEN=59663989&jsessionid=2830a9edcf6f794ff481'. Therefore the url becomes "test.cfm?CFID=1600&CFTOKEN=59663989&jsessionid=2830a9edcf6f794ff481" whereas it should been like "test.cfm;jsessionid=2830a9edcf6f794ff481?CFID=1600&CFTOKEN=59663989". Since jsessionId is not correctly specified, server does not get this and hence creates a new session.

So how do you handle it? One way is to get the sessionId and urltoken from the session and create the url as expected (which is some effort on developer part). Alternatively, you can use a rather simple approach of using URLSessionFormat(url) which will do the exact thing which is required here. URLSessionFormat() appends the necessary information if cookies are disabled. If they are enabled, it does not do anything. Therefore it might be a better idea to always use this function for any GET or POST url.

The above code should actually have been


<cfset myurl=URLSessionFormat("test.cfm")>
<form method="post" action="#myurl#">
...
<input type="submit" value="Submit" >
</form>


JRun Threadpool settings

The other day I was looking at JRun's Threadpool implementation and it really took me SOME time to understand that piece of code. It is one of those codes which are not meant to be understood by others :D. It got me really confused about 'Active Handler Threads', 'min Handler Threads' and 'max Handler threads'. It was very much different from what I had assumed. Too much of 'creating runnables', 'swapping runnables', 'destroying runnables'.. phew.. I think this is the exact reason why Doug Lea and team had to introduce a standard implementation of ThreadPool in 'Tiger' release. Its so simple, neat and elegant I wonder why wasn't it introduced earlier in JDK.

Anyways, enough of cribbing. After my enlightenment of JRun's or CFMX's threadpool, I thought it would be nice to share it with you all. So what are these thread counts? (I am sure most of you would have it figured out. Neverthless.. :) )

Min handler threads - It is the number of web threads that will be spawned initially and will be waiting for HTTP requests. Which effectively means that it is the no of threads which will be waiting on serversocket.accept(). Thus it controls the no of requests that will be accepted concurrently. It is ideally the minimum concurrent users that you expect on the server. As soon as a thread gets a client request (i.e comes out of serversocket.accept()), it enters into a throttle before processing the request. This is where active handler threads come into the picture. Before the thread starts processing the request, it spawns another thread, if required, which can listen to the incoming requests.

Active handler threads - This decides how many requests would be concurrently processed. The throttle we talked about above, allows a maximum of "active handler" threads to continue and rest of threads wait until another thread exits the throttle. This along with "min handler thread" controls the throughput of the server. The value of active handler threads must be between min Handler count and maxHandler Count.

Max handler threads - This is maximum number of threads that can be created in the pool. This includes the threads queued in the throttle + threads processing the requests + threads waiting on the server socket. Once the server reaches the max Handler thread count, server will start denying the request throwing "Server Busy Error". You can see the "Server Busy" error even without server reaching the "max handler" limit if the thread in the throttle queue timeout. So if you see this error, dont start increasing the max handler thread count. You might need to tune all the three counts.

Having both MinHandler and activeHandler counts help JRun in addressing any sudden spike in the load. Lets say your minHandler count is 20 and activeHandler count is 40 and suddenly you have 40 concurrent requests, all of them will be served without any queuing and delay. When the load eases down on the server, it will let the extra threads die and bring the thread count down to minhandler count i.e 20.

The next question that naturally comes to the mind is what should be the appropriate values of these for my server? Having too less value for it would mean that you are not utilizing the potential of the server well and requests start queuing up even though server can handle it. Having a too high value for these would mean too many context switches and the server performance will deteriorate. (Too many context switches means CPU is busy scheduling the threads rather than executing them and thus hurts the performance) So what should be the appropriate value? well.. there can not be one answer or a formula to compute these values. It depends on your application, the traffic that your application expects, memory and processors of the machine on which you are going to run it etc etc.

By default the values in "ColdFusion standalone" are

- Min Handler Thread - 1
- Active Handler Thread - 8
- max Handler thread - 1000

min and active counts here are fine for a development machine but definitely not for a production machine. And in my opinion the value of 'max handler' is bit high even for a production machine. Creating a large no of threads does not necessarily increases the throughput of your server. It can actually lower it down because of the high no of context switches VM will have to make. Moreover, it might not be possible to create 1000's of threads because of OS limitations. On many of the OS, you will get an OutOfMemory Error because the VM will not be able to spawn so many native threads for you. I think max handler count in the range of 300-400 should be good enough.

Regarding tuning these counts, there are huge no of articles around which will tell you how to go about it. Since notion of these counts exist on all the application/web servers, articles need not be CF or JRun specific. In brief, you would need to run some kind of load tests with different values of minHandler and active handler counts, note the throughput and plot a graph. This graph should help you arrive at the appropriate value for these settings.

Weird Error while JRun/CF Startup

Today suddenly one of the test machine here started to give one weird error. It had Coldfusion standalone and the CF server would not start up. I checked up all the logs and there was nothing in it. In fact JRun process itself didnt start up.
When I tried to run it from the console using "jrun -start coldfusion", I saw the following error laughing at me.

Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: java.security.AccessControlException: access denied (java.util.PropertyPermission jrun.home read)
 at java.security.AccessControlContext.checkPermission(Unknown Source)
 at java.security.AccessController.checkPermission(Unknown Source)
 at java.lang.SecurityManager.checkPermission(Unknown Source)
 at java.lang.SecurityManager.checkPropertyAccess(Unknown Source)
 at java.lang.System.getProperty(Unknown Source)
 at jrunx.kernel.JRun.(JRun.java:52)

It was spooky as I could see that the policy file has all permission and the JVM was complaining that the JRun process does not have the permission to read jrun.home property.
I was going bonkers before I realized that policy file was not getting picked up in the first place.
Finally it turned out that some test setup had messed around with the jvm.config file and misplaced some quotes. java.args in this file looked like

java.args=-server -XX:MaxPermSize=128m -Xmx512m -Dsun.io.useCanonCaches=false
-Dcoldfusion.classPath={application.home}/../lib/updates,{application.home}/../lib/,{application.home}/../gateway/lib/,{application.home}/../wwwroot/WEB-INF/cfform/jars
-Dcoldfusion.libPath={application.home}/../lib
-Dcoldfusion.rootDir={application.home}/../ -Djava.security.manager
-Djava.security.policy="C:/CF/lib/coldfusion.policy"
-Djava.security.auth.policy="C:/CF/lib/neo_jaas.policy"

where it should have been like

java.args=-server -XX:MaxPermSize=128m -Xmx512m -Dsun.io.useCanonCaches=false
-Dcoldfusion.classPath={application.home}/../lib/updates,{application.home}/../lib/,{application.home}/../gateway/lib/,{application.home}/../wwwroot/WEB-INF/cfform/jars
-Dcoldfusion.libPath={application.home}/../lib
-Dcoldfusion.rootDir={application.home}/../ -Djava.security.manager
"-Djava.security.policy=C:/CF/lib/coldfusion.policy"
"-Djava.security.auth.policy=C:/CF/lib/neo_jaas.policy"

So in case you are touching jvm.config file (Ideally you should not), take extra care about the quotes.

Hotfix for CFDocument thread hang in CF7.0.1 standard.

There was a problem in CF7.0.1 standard edition where if you hit any error in a page using CFDocument, any other page using cfdocument will hang. This was a threading issue we had for standard edition and was logged as bug 61378.

This was fixed in the hotfix 1 and is available at
http://www.adobe.com/support/coldfusion/ts/documents/aae43964/chf7010001.jar

However I would recommend you to use hotfix 2 at
http://www.adobe.com/support/coldfusion/ts/documents/aae43964/chf7010002.jar
See the details of this hotfix at http://www.adobe.com/cfusion/knowledgebase/index.cfm?id=aae43964

Scale to fit for CFDocument

I have seen couple of posts on the forum in which people wanted to know a way to fit the document in a page. I am wondering about the use case if this is really required. Could you please let me know the use cases and how are you working around it currently. You need to build a case to get this attribute in the tag :)

Host name verification in HTTPS

As I mentioned in my blog on Missing images in CFDocument, in case you are using HTTPS, you must ensure that the certicate host name matches the host name in the URL. Lets say your certificate is issued to "www.mysite.com" then the request URL must have the host as "www.mysite.com". It can not be accessed using 'localhost', '127.0.0.1', that machine's IP address or machine's name.
Till JDK1.3, Sun's SSL implementation never used to verify the host name of the certificate. Since JDK1.4, it now verifies the hostname to prevent URL spoofing (When I request for some URL, some other guy in between intercepts and sends his own certificate and I will remain under the impression that I was getting the certificate of the requested server and hence a threat).

In case you want to access the URL using localhost or IP address or using machine's name, there is a workaround possible but that would invlove wrting some java code.
HttpsURLConnection that is used to make the connection, uses an interface HostnameVerifier to verify the host name of the certificate. A default implementation is used by default. You can provide your own implementation of this interface and set it on HttpsURLConnection. That will give the control of host name verification in your hand and you can verifiy it the way you want.

Unfortunately, this interface and HttpsURLConnection are present in both javax.net.ssl and com.sun.net.ssl package. So depending on which SSL packages are being used, you will have to implement appropriate interface and you will have to set this on appropriate HttpsURLConnection. To be sure, let your implementation class implement both the interface and set it on both the HttpsURLConnection by calling the static method

setDefaultHostnameVerifier(HostnameVerifier)

A simplistic implementation which disables any host name verification could be like

    class MyHostnameVerifier implements com.sun.net.ssl.HostnameVerifier, javax.net.ssl.HostnameVerifier{
        public boolean verify(String urlHostName, String certHostName){
            return true;
        }

        public boolean verify(String urlHost, SSLSession sslSession){
            return true;
        }
    }

set this verifier to both HttpsURLConnection at appropriate place.

    MyHostnameVerifier verifier = new MyHostnameVerifier();
    javax.net.ssl.HttpsURLConnection.setDefaultHostnameVerifier(verifier);
    com.sun.net.ssl.HttpsURLConnection.setDefaultHostnameVerifier(verifier);

Thread dumps...

Many a times our customers complain of Coldfusion Server(CF) hanging, becoming non-responsive, taking 100% cpu or some request taking too much time. In order to understand the reasons why this happens, only way is to analyze the "Thread dumps". Not many of our CF users are java geeks and therefore when we ask them to send a thread dump, they have a very little clue about it. I hope they wouldn't be clueless anymore ;)

What exactly is "Thread dump"
"Thread dump" basically gives you information on what each of the thread in the VM is doing at any given point of time. This makes "Thread dump" an excellent debugging tool. It can tell you the states of each of the thread in the VM, where exactly each thread is in the execution path at that point, which thread is waiting and where is it waiting and lot more. It also shows you the stack of each thread and can help you track the execution path.

How do I take a Thread dump?
For any Java VM, if you are running it from a command prompt, you can get the thread dump by pressing Ctrl+Break (for windows) or Ctrl+\ (for unix machines) on the terminal running the server.
In case you are running the VM in background or as services, you can send a "SIGQUIT" signal to the process to get the thread dump. To send this signal to VM on unix machine, you can use "kill -SIGQUIT [pid]" where 'pid' is the process id of the server process. This will send the signal to the VM to dump the thread stack on the error stream (On Sun JVM) or to a new file (IBM's JVM).

You can also use the Stacktrace tool to get the thread dump. It is a nice java webstart application that you can run on the machine running the server. All you need is to specify the "process id" and it will nicely show the threaddump in its own window.

To be more specific for CF, if the standalone CF server or CF on J2EE server has a terminal, use Ctrl+Break or Ctrl+\ depending on the platform. If its running as background process or as services, use Stacktrace tool I mentioned earlier.

Here is how the thread dump would look.

Full thread dump Java HotSpot(TM) Server VM (1.5.0_04-b05 mixed mode):

"web-4" prio=5 tid=0x28999418 nid=0x1460 runnable [0x2fbaf000..0x2fbafd18]
       at java.io.WinNTFileSystem.canonicalize0(Native Method)
       at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:374)
       at java.io.File.getCanonicalPath(File.java:531)
       at java.io.FilePermission$1.run(FilePermission.java:218)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.io.FilePermission.init(FilePermission.java:212)
       at java.io.FilePermission.(FilePermission.java:264)
       at java.lang.SecurityManager.checkRead(SecurityManager.java:871)
       at java.io.File.exists(File.java:700)
       at jrunx.resource.FileResource.exists(FileResource.java:98)
       at jrunx.resource.FileResource.getURL(FileResource.java:140)
       at jrun.servlet.JRunServletContext.getResource(JRunServletContext.java:192)
       at jrun.servlet.file.FileServlet.service(FileServlet.java:148)
       at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:91)
       at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
       at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:257)
       at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:541)
       at jrun.servlet.http.WebService.invokeRunnable(WebService.java:172)
       at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:318)
       at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:426)
       at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:264)
       at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

"web-3" prio=5 tid=0x2814e470 nid=0xd70 in Object.wait() [0x2faaf000..0x2faafd98]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x064b5a90> (a jrunx.scheduler.JSemaphore)
       at jrunx.scheduler.JSemaphore.acquire(JSemaphore.java:74)
       - locked <0x064b5a90> (a jrunx.scheduler.JSemaphore)
       at jrun.servlet.network.NetworkService.accept(NetworkService.java:348)
       at jrun.servlet.http.WebService.createRunnable(WebService.java:104)
       at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable(ThreadPool.java:285)
       at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable(ThreadPool.java:347)
       at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable(ThreadPool.java:239)
       at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"web-2" prio=5 tid=0x271954c0 nid=0x170c in Object.wait() [0x2ef6e000..0x2ef6fa18]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x02af8eb8> (a java.awt.image.PixelGrabber)
       at java.awt.image.PixelGrabber.grabPixels(PixelGrabber.java:254)
       - locked <0x02af8eb8> (a java.awt.image.PixelGrabber)
       at java.awt.image.PixelGrabber.grabPixels(PixelGrabber.java:209)
       at com.lowagie.text.Image.getInstance(Unknown Source)
       at com.lowagie.text.Image.getInstance(Unknown Source)
       at com.lowagie.text.pdf.PdfGraphics2D.addAltText(Unknown Source)
       at ice.pilots.html4.MacromediaCSSExtension.addonImageProcessing(MacromediaCSSExtension.java:186)
       at ice.pilots.html4.ObjectBox.paintChunk(OEAB)
       at ice.pilots.html4.InlineBox.paintChildren(OEAB)
       at ice.pilots.html4.InlineBox.paintChunk(OEAB)
       at ice.pilots.html4.InlineBox.paintChildren(OEAB)
       at ice.pilots.html4.BlockBox.paint(OEAB)
       at ice.pilots.html4.OutlinePainter.drawBox(OEAB)
       at ice.pilots.html4.BlockBox.paint(OEAB)
       at ice.pilots.html4.OutlinePainter.drawBox(OEAB)
       at ice.pilots.html4.BlockBox.paint(OEAB)
       at ice.pilots.html4.CSSLayout.paint(OEAB)
       - locked <0x02cebb10> (a java.lang.Object)
       at ice.pilots.html4.ThePrinter.printPage(Unknown Source)
       at coldfusion.document.DocumentSection.process(DocumentSection.java:230)
       at coldfusion.document.DocumentSection.print(DocumentSection.java:108)
       at coldfusion.document.DocumentExporter.export(DocumentExporter.java:237)
       at coldfusion.document.DocumentFrame.exportContent(DocumentFrame.java:118)
       at coldfusion.document.DocumentProcessor.processContent(DocumentProcessor.java:130)
       at coldfusion.document.DocumentProcessor.ProcessContent(DocumentProcessor.java:59)
       at coldfusion.tagext.lang.DocumentTag.processContent(DocumentTag.java:1218)
       at coldfusion.tagext.lang.DocumentTag.access$100(DocumentTag.java:84)
       at coldfusion.tagext.lang.DocumentTag$3.run(DocumentTag.java:1179)
       at java.security.AccessController.doPrivileged(Native Method)
       at coldfusion.tagext.lang.DocumentTag.doAfterBody(DocumentTag.java:1174)
       at cfcmyk2ecfm937582361.runPage(E:\CFusionMX7\wwwroot\testfolder\doctest\cmyk.cfm:3)
       at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:152)
       at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:349)
       at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
       at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:209)
       at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:51)
       at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
       at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:69)
       at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
       at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
       at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
       at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
       at coldfusion.filter.RequestThrottleFilter.invoke(RequestThrottleFilter.java:115)
       at coldfusion.CfmServlet.service(CfmServlet.java:107)
       at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:78)
       at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:91)
       at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
       at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:257)
       at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:541)
       at jrun.servlet.http.WebService.invokeRunnable(WebService.java:172)
       at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:318)
       at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:426)
       at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:264)
       at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
.
.
.

If you anayze this thread dump, you can see that "web-2" thread is handling request for "cmyk.cfm" and is waiting for the images to be loaded i.e (actually at Image.getInstance()). At this point, cfdocument has sent another http request for the image file that it has to render and that request is handled by another thread "web-4" which is running and is currently checking if the image file requested exists on the file system. Isn't it very powerful ?

You can checkout a very nice article Thread dump and stack traces written by Rajiv (my ex-colleague and mentor in Pramati) which I feel is a must read for Java developers.

Missing images in CFDocument

I have seen huge no of postings on CF forum where CFDocument users have complained about seeing red-cross (or red-x ) for images . In this post, I would like to list the reasons why they happen and how they could be resolved .

Before we proceed with different cases of red-x, lets see in brief, how images are rendered in CFDocument.

During the processing of cfdocument tag, CF engine interprets/executes the content inside the cfdocument tag, creates html content out of it and renders it in the memory. While rendering this html content, if any image tag is found, a separate HTTP request is made to retrieve this image content. A separate HTTP request is necessary because image in the generated html can be local as well as remote. For Java geeks out there, we use URLConnection.getContent() to retrieve the image data. A red-x means that CFDocument was not able to retrieve the image.

Now lets see different scenarios one by one

1. Image name has space in it. In CFMX 7.0, you can get red-x for images if the image file name has any space in it. For example if the image file name is "my picture.jpg", only a red-x will appear in the pdf/flashpaper. It happens because the url created for the image is not encoded. A workaround for this is to either use encoded url for the images i.e replace 'space' in the name with '%20' OR dont have spaces in the file name at all :). This bug has been fixed in Merrimack (Coldfusion 7.0.1). So if you are still on 7.0, upgrade :)
   
   
2. If your server is behind firewall. As we mentioned earlier, CF server needs to send an HTTP request for the images. If the firewall prevents any outgoing connection from the server, CF will not be able to retrieve them and will show a red-x in place of them. You will need to setup your firewall in such a way that server can send an HTTP request to itself.
   
   
3. If your server is behind a proxy. If Coldfusion server is connected to the external world using a proxy, then also CFDocument will not be able to load the images. This is because currently there is no way you can specify proxy configuration for CFDocument tag.
   Current solution to solve this is to define the following system properties for the JVM. You can specify these in "runtime/bin/jvm.config" if you are using standlone or on JRun server.
   
   -DproxySet=true -DproxyHost=[hostname] -DproxyPort=[port] -Dhttp.proxyHost=[hostname] -Dhttp.proxyPort=[port]
   
   
4. If you are using HTTPS and your images do not appear in the pdf/flashpaper, you must ensure the following
• CF server's certificate is trusted. In other words, certificate of the CA who issued the certificate for you, must be present in the trusted certificate store (runtime/lib/trustStore). You can use keytool to list/view/import/.. certificate in the certificate store.
  If CF is using a self signed certificate, CF's certificate must be present in the trustStore.
• The certificate is valid and has not expired.
• Host name of the server must match the host name to which the certificate was issued.
  
  
5. If the resources on your webserver are protected using some kind of authentication like basic authentication or digest authentication, cfdocument can not retrieve those resources. That is because you can not provide any authentication information to cfdocument tag currently. This means that cfdocument can not retrieve images if it is protected using authentication and you will see a red-x. One solution for this is to replace all image urls with "file" urls.
   See this entry for more details on this workaround. The other solution is to write little java code to set a 'java.net.Authenticator'. I will post a separate entry for this.


6. You get red-x for images and you have verified that its none of the above mentioned cases. Time to check the web server now. We have seen some cases where the web server is configured to allow requests only from a certain set of browsers (User agents to be precise) perhaps to prevent spiders and bots from overloading the server. When CFDocument creates a URLConnection for the image, it sends a "User-Agent" header, that looks like "User-Agent:Java/1.4.2_07", in the HTTP request. If the web server does not recognize "Java" user-agent, it returns a status code of 404 (resource not found) and hence the images can not be displayed. Solution for this case is to either change the configration for the web server or set your own user agent using the following system property on the JVM.
   

       -Dhttp.useragent="ColdFusion"
   

   You can give any name here as userAgent in place of "ColdFusion".
   

Hope this helps people in resolving isues related to missing images in CFDocument. In case you are getting a red-x even after verifying all the cases mentioned above, please let me know.

Related Entry :

• Workaround for cfdocument missing images
• Accessing password protected URL
• New CFDocument Hotfix released
  
• Enhancement to CFDocument in CF8