Coldfused?

ColdFusion ORM : Using DB Views instead of Table

2009-09-15T02:34:00.001-07:00

Posted at http://www.rupeshk.org/blog/index.php/2009/09/coldfusion-orm-using-db-views-instead-of-table/

ColdFusion ORM : What is "N+1 Select problem"

2009-09-14T01:59:00.001-07:00

Posted at http://www.rupeshk.org/blog/index.php/2009/09/coldfusion-orm-what-is-n1-select-problem/

ColdFusion ORM : Performance tuning - Lazy loading

2009-09-11T06:36:00.001-07:00

Another post in this series posted at http://www.rupeshk.org/blog/index.php/2009/09/coldfusion-orm-performance-tuning-lazy-loading/

ColdFusion ORM : Performance tuning – Fetching Strategy

2009-09-10T04:39:00.001-07:00

Posted at http://www.rupeshk.org/blog/index.php/2009/09/coldfusion-orm-performance-tuning-fetching-strategy/

ColdFusion ORM : Troubleshooting - 'Lazy' does not work

2009-07-31T07:08:00.001-07:00

Few days back Manju logged a bug in CF-ORM saying 'lazy' does not work for many-to-one relation and that too on non-Windows machine. At first, I simply rejected the bug because a) ORM can not have anything to do with OS and therefore, if it works on Windows, it works on all the plaform and b) I know it works :-). But he did not agree and I had to take a look at that machine. And apparently he was right - lazy was not working ! The related entity was in-fact getting loaded immediately. (Question for you - how will you know that lazy is working or not?)

Even after seeing this, I did not believe it and asked him to replicate this on another system and he successfully showed that to me on one another system. And he agreed that it works fine on most of the configurations. The problem exists only on a few of the systems.

This got me thinking - Why would a relation get loaded immediately even after it is marked lazy? The only answer would be - if some one is accessing that lazy field and calling some method on it. I checked his code which was loading the entities to see if there could be any case, where the field would get loaded and unfortunately there was none.

And then suddenly it hit me - what if "memory tracking" is swithched on? That would access each of the field of each object recursively to compute the size of an object and that can definitely cause this. I immediately checked the server monitor and the "memory tracking" was right there staring at me in "red"! It was indeed enabled. I asked Manju to check the other system as well (where lazy was not working) and the memory tracking was enabled there as well.

So the lesson - If the 'memory tracking' is enabled on the server, the relationship will no longer remain lazy. And btw, you should enable "Memory tracking" on the server only if you need to track the memory for some troubleshooting. Memory tracking is really really expensive in terms of performance.

Another reason why it might not work for you could be - if you are sending the object to flex. Currently, during serialization, the related objects also get sent irrespective of the lazy attribute set on the relationship. We are still working on it and hopefully by the time we release, this will be fixed.

ColdFusion ORM : How to log SQL

2009-07-27T07:52:00.001-07:00

When you use ORM for developing an application, SQLs are generated and executed by the underlying ORM engine (i.e Hibernate for ColdFusion ORM). However, for both troubleshooting and performance optimization, it is crucial to monitor what queries are getting generated. It can help you find out if there is any error in mapping that you have provided as well as it can help you decide various tuning strategies.

ColdFusion can log the SQLs generated by ORM either onto the console or a file. At the same time it leaves enough hook for you to log it anywhere you want.

ColdFusion ORM provides two ways to log the SQLs.

Using application setting to log to console : This is a quick and simple way to log the sql to console. This is enabled by setting "logsql" in ormsettings.
```
   <cfset this.ormsettings.logsql="true">
```
This setting is self sufficient and it will log all the sqls executed by hibernate to the console (or a file where the server output goes). However this is not a very flexible option. The sqls are always written to the console and it will be combined with any other output that goes to console. Also this option will not show the DDL queries used for creating or updating tables. It only logs the SQL for the entity operations.
Using log4J.properties: Hibernate uses log4j for its logging and you can completely control its logging (including SQL) by modifying the log4j.properties. log4j.properties is present under <cf_home>/lib directory. Please note that you don't need to do any application specific setting for this.

I will go in details about using log4j.properties for SQL logging. Here is a snippet from log4j.properties file that is shipped with ColdFusion.

###--------------- Hibernate Log Settings ------ 
### Set Hibernate log 
log4j.logger.org.hibernate=ERROR, HIBERNATECONSOLE 
 
### log just the SQL 
#log4j.logger.org.hibernate.SQL=DEBUG, HIBERNATECONSOLE 
#log4j.additivity.org.hibernate.SQL=false 
### Also log the parameter binding to the prepared statements. 
#log4j.logger.org.hibernate.type=DEBUG 
### log schema export/update ### 
log4j.logger.org.hibernate.tool.hbm2ddl=DEBUG, HIBERNATECONSOLE 
### log cache activity ### 
log4j.logger.org.hibernate.cache=ERROR, HIBERNATECONSOLE 

# HibernateConsole is set to be a ColsoleAppender for Hibernate message  using a PatternLayout. 
log4j.appender.HIBERNATECONSOLE=org.apache.log4j.ConsoleAppender 
log4j.appender.HIBERNATECONSOLE.layout=org.apache.log4j.PatternLayout 
log4j.appender.HIBERNATECONSOLE.layout.ConversionPattern=%d{MM/dd HH:mm:ss} [%t] HIBERNATE %-5p - %m%n%n 
#---------------------------------------------

First we will see the relevant settings for SQL logging:

log4j.logger.org.hibernate.SQL : Defines whether the SQL executed for entity operations will be logged and where it will be logged. The second value for this i.e 'HIBERNATECONSOLE' is a appender that controls where the SQLs will be logged. In the above example HIBERNATECONSOLE is a 'console' appender which means it will log the sql to console.
log4j.logger.org.hibernate.type : Defines whether parameter values used for parametrized query will be logged.
log4j.logger.org.hibernate.tool.hbm2ddl : Defines whether DDL sql statements will be logged.

To enable the SQL logging for console, we just need to uncomment the settings mentioned above. Here is how the hibernate log settings in log4j.properties file would look like

###--------------- Hibernate Log Settings ------ 
### Set Hibernate log 
log4j.logger.org.hibernate=ERROR, HIBERNATECONSOLE 
 
### log just the SQL 
log4j.logger.org.hibernate.SQL=DEBUG, HIBERNATECONSOLE 
log4j.additivity.org.hibernate.SQL=false 
### Also log the parameter binding to the prepared statements. 
#log4j.logger.org.hibernate.type=DEBUG 
### log schema export/update ### 
log4j.logger.org.hibernate.tool.hbm2ddl=DEBUG, HIBERNATECONSOLE 
### log cache activity ### 
log4j.logger.org.hibernate.cache=ERROR, HIBERNATECONSOLE 

# HibernateConsole is set to be a ColsoleAppender for Hibernate message  using a PatternLayout. 
log4j.appender.HIBERNATECONSOLE=org.apache.log4j.ConsoleAppender 
log4j.appender.HIBERNATECONSOLE.layout=org.apache.log4j.PatternLayout 
log4j.appender.HIBERNATECONSOLE.layout.ConversionPattern=%d{MM/dd HH:mm:ss} [%t] HIBERNATE %-5p - %m%n%n 
#---------------------------------------------

Here is the complete log4j.properties for logging SQL for console. Ofcourse after changing this you need to restart the server. If you need to log the parameter values used for queries, you need to uncomment '#log4j.logger.org.hibernate.type=DEBUG' as well.

What if you want to log the SQL to a file and not to console? That is pretty easy. You just need to change the 'Appender' used here (HIBERNATECONSOLE) to point to a 'FileAppender' instead of a ConsoleAppender. Here is how the configuration for HIBERNATECONSOLE should look like after you point it to a File Appender.

log4j.appender.HIBERNATECONSOLE=org.apache.log4j.FileAppender
log4j.appender.HIBERNATECONSOLE.File=../hibernatesql.log
log4j.appender.HIBERNATECONSOLE.Append=true
log4j.appender.HIBERNATECONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.HIBERNATECONSOLE.layout.ConversionPattern=%d{MM/dd HH:mm:ss} [%t] HIBERNATE %-5p - %m%n%n

For standalone ColdFusion installation, the file 'hibernatesql.log' will be created in the /logs directory.You can also specify a full path of the file for property 'log4j.appender.HIBERNATECONSOLE.File' and the log will be written to that.

That was easy. Isn't it? What if you want a rolling log file where you dont want the log file size to grow infinitely. That is fairly easy too. All you need to do is to use an appropriate appender. The appender definition for that will look like

log4j.appender.HIBERNATECONSOLE=org.apache.log4j.RollingFileAppender
log4j.appender.HIBERNATECONSOLE.File=../hibernatesql.log
log4j.appender.HIBERNATECONSOLE.Append=true
log4j.appender.HIBERNATECONSOLE.MaxFileSize=500KB
log4j.appender.HIBERNATECONSOLE.MaxBackupIndex=3
log4j.appender.HIBERNATECONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.HIBERNATECONSOLE.layout.ConversionPattern=%d{MM/dd HH:mm:ss} [%t] HIBERNATE %-5p - %m%n%n

Here is the complete log4j.properties for logging SQL to a file that will be rolled automatically once it reaches 500KB.

Now that you have seen how easy it is to change one 'Appender' to another, you can pretty much log it anywhich way you want. Here are some of the interesting 'Appender's that come with log4j which you can easily use.

DailyRollingFileAppender, ExternallyRolledFileAppender, JMSAppender, SMTPAppender, SocketAppender, SyslogAppender

See log4J for more details on log4j settings.

ColdFusion 9 : Implicit/Generated CFC Methods

2009-07-18T00:40:00.001-07:00

How many times did you write the accessor methods (better known as getters and setters) for the fields of your CFC? And how many times did you feel that it was really a mundane job and wished there was a better way to do this? With ColdFusion 9, we have done just that! You no longer need to write those accessors. ColdFusion will automatically generate the accessor methods for you in the object. All you need to do is to define the properties using cfproperty.

Consider this CFC Person.cfc

Component Person{
    property firstname;
    property lastname;
    property age;
    property city;
    property state;
}

When an object is created for this CFC, you can directly call the setters and getters for the properties as



    person = new Person();
    person.setFirstName("Brad");
    person.setLastName("Pitt");
    person.setAge(46);

    writeOutput("Name : #person.getFirstName()# #person.getLastName()#");
    writeOutput("Age : #person.getAge()#");

The implicit setter/getter uses the 'variable' scope, which is a private scope for a object, for storing the data. i.e the setter method puts the data in the variable scope and the getter method gets the data from the variable scope.

What if you want to override the setter/getter method? Ofcourse you can do that. You just need to define those methods in your CFC and ColdFusion will call your method instead of calling the implicit one.

Hmm.. What if you dont want ColdFusion to generate getter or setter or both methods for a particular property? You can disable that by adding attribute getter="false" and setter="false" on the property.

So if you define a property

property name="city" getter="false" setter="false";

ColdFusion will not generate getCity() and setCity() methods for this CFC.

The implicit methods can also do some nice validation if you have given the 'type' for properties. Consider the same Person.cfc with type specified where we have made age as a numeric type.

Component Person{
    property string firstname;
    property string lastname;
    property numeric age;
    property string city;
    property string state;
}

ColdFusion will do the datatype validation whenever the setter method is called for any property. So if you call



    person = new Person();
    person.setFirstName("Brad");
    person.setLastName("Pitt");
    person.setAge("LA");

It will throw a nice error saying "The value cannot be converted to a numeric".

Whats more interesting is that you can do a whole lot of validation using cfproperty. We have added two more attributes on cfproperty named 'validate' and 'validateparams'. These attributes allow you do even more advanced validations of the property data when the setter method is called for a property. (This is similar to cfparam or cfform validation).

The possible values for validate attributes are

string
boolean
integer
numeric
date
time
creditcard: A 13-16 digit number conforming to the mod10 algorithm.
email: A valid e-mail address.
eurodate: A date-time value. Any date part must be in the format dd/mm/yy. The format can use /, -, or . characters as delimiters.
regex: Matches input against pattern specified in validateparams.
ssn: A U.S. social security number.
telephone: A standard U.S. telephone number.
UUID: A Home Universally Unique Identifier, formatted 'XXXXXXXX-XXXX-XXXX-XXXXXXXXXXXXXXX', where 'X' is a hexadecimal number.
guid: A Universally Unique Identifier of the form "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" where 'X' is a hexadecimal number
zipcode: U.S., 5- or 9-digit format ZIP codes

The 'validateparams' attribute available with <cfproperty> takes the parameters required by the validator specified in the 'validate' attribute. This needs to be specified in the implicit struct notation.

min: Minimum value if 'validate' is integer/numeric
max: Maximum value if the 'validate' is integer/numeric
minLength: Minimum length of the string if the 'validate' is string
maxLength: Maximum length of the string if the 'validate' is string
pattern: regex expression if the validator specified in 'validate' attribute is regex

Consider the Person again where we will add few more properties with some validation

Component Person{
    property string firstname;
    property string lastname;
    property numeric age;
    property string city;
    /**
     * @validate string
     * @validateparams {minLength=2, maxLength=2}
     */
    property string state;
    /**
     * @validate zipcode
     */
    property numeric zip;
    /**
     * @validate telephone
     */
    proeprty phone; 
}

Now when you create an object of Person and call setState(state), before setting the data for state in variable scope, ColdFusion will validate that state value provided is of type 'string' and its length is 2. Similarly setZip() will validate that the input data is a valid zip code and setPhone() will validate that the input data is a valid telephone number.

For String type properties, you can also do regular expression validation allows you to have all sort of validation on a property of string type.

And to top it all, all of these nice goodies are available to you at a much lesser cost than regular UDFs. Yes, you read it right. Implicit methods perform much better than regular UDFs that you write in CFC. To confirm that, lets use this CFC where we will use implicit methods for 'firstName' and write our own accessors for 'lastName'.

component 
{
    property String firstName;
    lastName = "";
	
    public void function setLastName(string name)
    {
        lastName = name;
    } 
	
    public string function getLastName()
    {
        return lastName;
    }
}

Both implicit methods and UDFs are doing exactly the same thing i.e settings and getting the data in/from variable scope. Now lets run this comparison test


    person = new Person();
    t = getTickCount();
    for(i = 0; i < 100000; i++)
    {
        person.setfirstName("Brad");
        fname = person.getFirstName();
    }
    writeOutput("Total Time in implicit methods " & (getTickCount() - t));
    t = getTickCount();
    for(i = 0; i < 100000; i++)
    {
        person.setLastName("Pitt");
        lname = person.getLastName();
    }
    writeOutput("Total Time in UDF methods " & (getTickCount() - t));

Here is the output

Total Time in implicit methods 63
Total Time in UDF methods 485

Thats almost eight times faster than regular UDF. Isn't that sweet? You get those auto-generated methods, whole lot of auto-magical validations and to top that, incredible performance. I am sure writing accessors for your CFC fields will become a thing of past now !

A demo for ORM code generator in ColdFusion builder

2009-07-17T21:38:00.001-07:00

Evelin created this nice demo that shows how to setup this "ORM code generator" extension and how to use it. Check out the demo here.

ColdFusion Builder : ORM code generator

2009-07-17T05:20:00.001-07:00

If you have started using ORM (or you want to start) in ColdFusion 9, there is a nice plugin in ColdFusion builder that can help you quickly generate all the ORM code for you. All you do is select the tables from the RDS view, define the relationships, if there is any, and all the code including the service layer will be generated. Evelin talks about it here. Oh btw, if you did not notice, thats the ColdFusion Builder team blog.

You can download the plugin from here and download the instruction from here. Have Fun playing with it !!!

ColdFusion ORM and CFC Performance

2009-07-16T09:32:00.001-07:00

Since the day we started thinking about ColdFusion ORM, people have been raising concerns over it because of the CFC performance. We have heard people saying that ORM will be unusable because of poor CFC performance and it is a stupid idea to implement CFC based ORM. And today I heard Hal Helms and Brian kotek talking about it where Hal says that he shudders at the thought of hibernate integration in CF because of poor performance.

It is true that CFC performance is poor as compared to POJO and it will always be. ColdFusion CFC has lot of cost involved because of the dyanmic nature of it and additionally, it has reflection cost involved. I had shared some of the reasons with few of the community members and I thought it will be interesting to share it witha wider audience. Here are the main reasons :

Unlike java where the class has a fully qualified name and is loaded only once in the classloader, when you create CFC, CF needs to search for the CFC file on the disk, compute the class name for it and then load it. And that happens on each object creation unless you have switched on trusted cache .
CFC is compiled at runtime, loaded and constructed using reflection. I am sure you will be aware that reflection is much slower (25 times easily) than direct invocation in java.
CFC has two different scopes - this and variables scope which is a struct. So every object creation involves two additional struct creation cost.
CFC creation also involves running the default constructor i.e any code outside cffunction and that happens for each object creation. That involves the complete setup for method invocation i.e setting up localscope, superscope etc (another two structs here), which is also significant
CFC allows UDFs to be added/removed from the object at runtime and hence methods are added to each object on object creation and that too in both the scopes (adding each UDFs in both the scopes also has significant cost). Thats the reason why object creation becomes costlier as you add methods in the CFC. Higher the number of methods, higher the object creation cost.

We have done a lot of performance improvements regarding CFC creation and invocation in ColdFusion 9 which I hope you would have noticed. But even then it can not match POJO performance and thats an unrealistic target.

So given that, lets come to the question - How much does CFC creation performance matters when it comes to ORM. In reality, not much. We have done a number of performance comparison between CFC and POJO with hibernate keeping everything else absolutely same. And all the time, CFC performance was same or better than POJO persistence. It might sound unbelievable and in fact I also could not believe it myself when I saw the performance numbers first. But that is the truth. And here is why it happens - When it comes to persistence, the time taken in DB operation and the persistence layer is much more than the time taken in object creation. And that cost is same for POJO and CFC. So the only difference in cost between CFC and POJO with respect to ORM is object creation cost and cost of populating/fetching the properties data. For instantiating POJO object and for using properties accessors, Hibernate has to use reflection which is always costly and that removes this cost difference of CFC and POJO. And for few of the runs, CFC was able to beat POJO as well and explanation for that is the optimizations that we have done in using accessors for properties.

I would be curious to know about your experience regarding CF-ORM performance and CFC performance in general. How has your experience been so far regarding CF-ORM peformance?

ColdFusion ORM : The basics

2009-07-16T06:49:00.001-07:00

Before starting on the advanced topics, I thought it will be better to build some ground and hence I decided to do a post on the ORM basics. In this post, we will build a simple example to get a taste of ColdFusion ORM (CF-ORM) and during that we will also understand some of the basic concepts.

ORM is object relational mapping and in ColdFusion, objects are created using CFC. CFCs that needs to be persisted are called persistent CFC and that is marked by setting 'persistent' attribute to true on the component. We also need to define what persistent fields will be there in a persistent CFC and that is defined using 'cfproperty'. A field/property is marked persistent by setting persistent attribute to true on the cfproperty. By default, if the CFC is persistent, all its properties are considered as persistent unless you mark a property non-persistent. So typically 'persistent' attribute on the property is used only when you need to make that property non-persistent.

Each persistent CFC in ColdFusion application maps to a table in the database and each property in the persistent CFC maps to a column in the table (not exactly true but we will come to that later.. For the time being lets keep it that way). We will use the cfartgallery datasource for this example which has Artists and Art tables.

The first thing you need to do is - enable ORM for the application and define a datasource to be used (What is an ORM without a datasource?). ColdFusion ORM uses Application.cfc to define all the ORM specific settings. (If you haven't started using Application.cfc for your application, its time to start using it!)

Application.cfc


Component {
   this.name="ArtGallery";
   this.ormenabled="true";
   this.datasource="cfartgallery";
}

Note that the datasource setting defined here makes it the default datasource of your application which can be used by tags like cfquery, cfinsert, cfupdate, cfdbinfo. The same default datasource will be used by ORM as well.

There are a whole bunch of ORM related configuration that you can do in application.cfc which you can refer here.

Now that the application is configured, let us build the object and define the persistence information on it. To start with, we will first define the Artist.cfc

Artists.cfc

/**
* @persistent
*/
Component {
   property name="artistid" generator="increment";
   property firstname;
   property lastname;  
   property address;
   property city;
   property state;
   property postalcode;
   property email;
   property phone;        
   property fax;
   property thepassword;
}

This is the most simplistic definition of the component where we have defined only the component and its properties names. Since the table for this CFC already exists in the database, we have not added any table specific information in this and we will let ORM infer all the information from the database. The only additional setting that we have added here is the 'generator' attribute which is used to auto-generate the primary key.

After the components are defined, the first request to this application (i.e a page in this application) will make CF-ORM do all the setup necessary (basically generation of hibernate configuration, mapping files, building the session factory etc). Once the setup is done, you are all set to work with the entities.

We will first list all the artists and here is what you need to do for that

listAll.cfm

<cfscript>
  artists = EntityLoad("Artists");
  writedump(artists);
</cfscript>

To load a particular Artists with its ID, here is what you do
list.cfm


<cfscript>
  artist = EntityLoadByPK("Artists", 1);
  writedump(artist);
</cfscript>

There are several flavors of EntityLoad functions details of which can be read here

Let us now see how to perform insert and update on it.

save.cfm


<cfscript>
  // Insert a new artist
  artist = new Artists();
  artist.setfirstname("Leonardo");
  artist.setlastname("Da vinci");
  artist.setcity("Paris");
  EntitySave(artist);
  writeOutput(artist.getartistid());// Update an artist
  artist = EntityLoadByPK("Artists", 2);
  artist.setcity("NewYork"); // artist is automatically updated.
</cfscript>

As we see in the above example, EntitySave is used to insert/update an object in the table. There are some important things to note here

EntitySave is an intelligent function which automatically finds if a new row needs to be inserted for the given object or whether an existing row needs to be updated.
We called EntitySave for the insert here but not for update but even then artist '2' gets updated. So how did that happen? Actually what happens here is when you load an artist object, it becomes associated with the hibernate session which keeps track of any changes in the object and automatically saves it when the session is flushed. We will talk about more about hibernate session in a later post. For the time being lets just say that Hibernate Session is a short-lived object that represents a conversation between the application and the persistence layer and also acts as the first level of cache.
We did not write any setter or getter method for artist's properties in Artists.cfc but we are calling them here. That works because ColdFusion 9 automatically generates accessor methods for any property written in a CFC. More details on generated methods in a later post.
At line no 6, we called entitySave, but if you check the database, the row is not inserted yet. So when does that happen? Hibernate batches up all the operations till the end of the request or to be exact till hibernate session is flushed. ColdFusion ORM starts up a session when the first ORM method is called in the request and is automatically flushed when the request ends. The batching is done for performance reason so that hibernate executes the sql with the final state of the objects. It will be a huge performance bottleneck if ORM keeps executing sql for each changes in the object.

To delete an Artist, you need to call EntityDelete() passing the object to be deleted.

delete.cfm


   artist = EntityLoadByPK("Artists", 15, true);
   EntityDelete(artist);

Relationship

So far we have seen how to perform CRUD for a single entity. But in any application, there will be entities which are associated and ORM must load the associated object as well when loading a particular entity. For our example, an Art will have an Artist and when loading the art object, it should also load the associated artist. So lets build the model first after which we will see how to work with the association.

In cfartgallery, the table Artists has a one-to-many relationship with Art table, which are joined using the foreign key column ARTISTSID. This means that each artist has created multiple arts and each art is created by one artist. To represent this in the object model, each ARTIST object would contain an array of ART objects. Each ART object will also contain a reference to its ARTIST object thereby forming a bidirectional relation.

To achieve this, we will need to add an extra property 'arts' to 'Artists' that contains an array of ART objects for that Artist. The modified Artists.cfc would look like

/**
* @persistent
*/
Component {
   property name="artistid" generator="increment";
   property firstname;
   property lastname;  
   property address;
   property city;
   property state;
   property postalcode;
   property email;
   property phone;        
   property fax;
   property thepassword;
   property name="arts" fieldtype="one-to-many" fkcolumn="artistid" cfc="Art" cascade="all" inverse="true";
}

Here is the Art.cfc


/**
* @persistent
*/
Component {
   property name="artid" generator="increment";
   property artname;
   property price;
   property largeimage;
   property mediaid;
   property issold;
   property name="artist" fieldtype="many-to-one" fkcolumn="artistid" cfc="Artists" ;
}

Notice the artist property above which is of many-to-one type. Also notice that both the property use the same value for fkcolumn attribute i.e 'artistid' of Art table that references artistID pk of Artist table.

Since we have added a new persistent CFC (Art.cfc) after the application was loaded, we need to re-initialize the ORM for this application so that mappings for Art.cfc also gets generated. This can be done by calling ORMReload() method. There are some nice ways to do this but for the time being lets keep it simple by putting this in a separate page which we will call to reload ORM.

initializeORM.cfm

<cfset ormReload()>

If you load and dump Artist (using listAll.cfm), you should also see the associated art objects for artists.

Now let us create a new Art and associate it with an existing Artist.
artCreate.cfm


<cfscript>
   artist = EntityLoad("artists", 1 ,"true");
   art = new Art();
   art.setartname("landscape");
   art.setPrice(1500);
   art.setissold(false);
   art.setartist(artist);
   artist.addArts(art);
   EntitySave(art);
</cfscript>

If you notice line 7-8 above, we associate artist to art by calling art.setArtist(artist) as well as art to artist by calling artist.addArts(art). Hibernate needs us to do this in order to set up the bidirectional relation properly. Since it is a bidirectional relation, you must also decide which side will set the relation in the database. i.e which side of the relation will set the fkcolumn in the table. This is controlled by the "inverse" attribute of proeprty, which if set to true, tells hibernate that this is a inverse of the other relation and this side of relation should be ignored for persistance. If you don't set inverse to true, Hibernate will unnecessarily fire two sqls for the same association.

So there you have it. We have seen how you can use ORM to perform the basic CRUD operations on entities. For more details, you can refer to the ORM doc and Hibernate docs.

ColdFusion ORM - An Evolution in building datacentric application

2009-07-13T22:51:00.001-07:00

As we all know, ColdFusion was born as DBML - a script that was very similar to HTML but had certain tags that could perform database operation and it revolutionized the way web applications were built. Tags like cfquery, cfinsert, cfupdate made it very easy to perform database operations while building web application. Even after 14 years of CFML, (which we incidently completed day before yesterday), cfquery is *the* most popular and the most commonly used tag. However there are few downsides to it

Database vendor dependency - The queries that are written are DB specific. If you ever need to switch the database or if you need to make the application database agnostic, it becomes too much of pain. You will end up changing the entire application code or lot of conditional code to handle queries for different databases.
Very tight coupling with the database - The queries have very tight coupling with the database. This means that if there is any change in the table name or column name, you will end up with changes all over your application code.
Repititive and hence low on productivity - For any CRUD (Create, read, update and delete) operation, you need to write queries which are more or less same for all the tables. You must have felt that you are repeating the same code everywhere in the application. There is too much of repititiveness which reduces productivity.
Error prone - How many times did you write insert/update queries and then you had to struggle to debug because the order of column and its values were different? And how many times did you get frustrated because there was a datatype mismatch between the value you had specified and the one expected? How many times did you write a query that was prone to sql injection and you were reminded to use query param? I am sure all of this has happened to all of us and its completely natural given the way queries are written. They are error prone.

ColdFusion 9 introduces a new way to build datacentric application using ORM, that handles all the downsides we saw above. It is not a replacement to cfquery/cfinsert/cfupdate tags but its a different approach altogether. It allows you to build your application using objects where you focus on the business logic and all the grunt work of persistence is taken care automatically (In simple words you dont write SQL queries generally).

ORM (Object relational mapping) is a well known strategy/technique to map relational data to the object model. In an object model, business objects are not aware of the database structure. Objects have properties and references to other objects whereas Databases consist of tables with columns that are related to other tables using foreign key. ORM provides a bridge between the relational database and the object model allowing you to access and update data entirely using the object model of an application.

ORM is not new to ColdFusion either - Transfer and Reactor being the most popular ones. ColdFusion ORM (or CF-ORM) is much more extensive and sophisticated ORM solution than these ORMs. ColdFusion ORM is built on top of Hibernate - the best java ORM engine out there, which is quite powerful and extensive. ColdFusion ORM makes it very easy to use persistence with objects and at the same time allows you to leverage the full power of Hibernate.

ColdFusion ORM provides features such as

Defining the persistence mapping and managing the object's persistence using very simple methods.
Database vendor independence
Loose coupling between database and application - The application becomes very adaptive to changes as the configuration is central and changes will be required only there.
Auto-generation of schema - ORM lets you auto-generate the schema for your application. So you never need to bother about keeping your object model and the database schema in sync. It can automatically happen.
Productivity and manageability - Since ORM takes care of all persistence grunt work, you focus on your application logic and thus your application becomes much more cleaner and manageable. It also makes you lot more productive as application can be built much more faster.
Concurrency - ORM inherently takes care of concurrency control. So you dont need to worry much about how things would work when multiple database operations happen in parallel web requests.
Performance - ORM provides lot of performance optimizations that can make your application run faster. The optimizations include two levels of caching (including pluggable distributed cache), lazy loading, various settings to tune the sql queries generated by ORM.
Secure - Since queries will be executed by ORM, issues like SQL injection no longer exist and thus your application becomes secure.
Inbuilt pagination
SQL like query (HQL) for a finer control of the data to be loaded.
...

In this series of posts, I will be talking lot more about ORM which I hope you would like. In case you have not downloaded ColdFusion 9 beta, go download it and start playing with it.

Moving my blog to a new address

2009-07-13T22:44:00.001-07:00

With the release of ColdFusion 9 beta, I am also moving my blog from blogger to a new address (http://www.rupeshk.org). I have moved all the content including comments from blogger to the new site. The new blog is still in beta as I still need to do lot of cleanup.

I will keep posting the entries to both the blogs for some time till I feel that the traffic has moved to the new blog. So please put all your comments on the new blog so that its all at one place :-)

If you are subscribing to my feed or if you have linked to this blog, you can change that to use the new feed. See you over at http://www.rupeshk.org

ColdFusion 9 and ColdFusion builder beta available

2009-07-12T22:48:00.001-07:00

I am sure by the time you will be reading this, the blogosphere will be buzzing with this news :-). We just (13th July 12:01 AM EST) went live with the beta of ColdFusion 9 and ColdFusion builder. All of us in the team had been working really hard for last one and half years to put in features, that will take RAD to the next level, features that will make your life easy, features that you are going to love. ColdFusion has always been known for "making hard things easy" and I am sure this release will make even harder things easy :-)

Ben talked about the new features in ColdFusion 9 and ColdFusion beta here :

http://www.adobe.com/devnet/coldfusion/articles/coldfusion9_whatsnew.html

http://www.adobe.com/devnet/coldfusion/articles/cfbuilder_whatsnew.html

Here are the links for public beta

ColdFusion 9 : http://labs.adobe.com/technologies/coldfusion9/

ColdFusion Builder : http://labs.adobe.com/technologies/coldfusionbuilder/

I also wanted to highlight an excellent devnet article written by Mark Mandel introducing ColdFusion ORM. I am sure ColdFusion ORM is going to completely change the way ColdFusion applications are built and is going to make you much more productive. Hey, I am not boasting because its my baby, but I genuinely feel this :-)

Devnet Article on ColdFusion Performance tuning

2009-06-13T06:38:00.001-07:00

In case you haven't seen it yet, there is a new devnet atricle that was published this week on "Performance tuning for ColdFusion Application". Its an excellent article written by Kunal from the ColdFusion team. It gives a great insight into how to go about tuning your JVM, how to tune the VM's Garbage collector settings and how to tune different settings in ColdFusion server. It also gives some tips on best practices for writing ColdFusion application from a performance perspective. Overall an excellent article.

However, there is one mistake in the fourth page where it suggests to use File functions instead of File tag which I would like to correct. Actually it should have suggested to use new File functions that were introduced in ColdFusion 8 instead of older File functions or cfile tag.

ColdFusion Livedocs on AIR

2008-07-01T01:45:00.001-07:00

I was doing a code review for Vinu today (dont ask me which feature it was :-)) and we needed to check the reference doc for a tag. And instead of opening the local doc page, he opened this cool application that had livedocs and it was blazing fast (I know Livedocs sucks ! and yes I work for Adobe ;-)).

I was like 'whoa!! thats a cool app.. Did you write it? Why dont you release it for public??' And then he told me that some one from the community had written it and its already available for everyone. It has been developed by Brian Love and it is available here. It has very small size application, very lightweight and extremely fast.

So in case you haven't downloaded it, download it now!! I am sure you would love it.

New 'EvalAtPrint' attribute in CFDocumentItem

2008-06-30T06:28:00.001-07:00

A new evalAtPrint attribute has been added to cfdocumentitem tag in ColdFusion 801 which is applicable when the item type is header or footer. The default value of this attribute is false.

Note: The cfdocumentitem behavior with the default evalAtPrint value is identical to the behavior in ColdFusion MX 7, but differs from the behavior of the ColdFusion 8 release.

When the evalAtPrint attribute value is false, the default value, ColdFusion evaluates the cfdocumentitem tag body at the same time when it is processing the cfdocument tag. This is the behavior identical to CFMX7. In this case you can use CFDocument variables (TotalPageCount, CurrentPageNumber, TotalSectionpageCount and CurrentSectionpageNumber) only inside cfoutput and not in any expression because the page numbers are not known unless the cfdocument body content is evaluated and layed out.

In order to use CFDocument variables in expression inside CFDocumentItem tagbody, you should set evalAtPrint to true as shown in the following example.

<cfdocument format="pdf">
   <cfdocumentitem type="header" evalAtPrint="true">
       <cfif (CFDocument.currentPageNumber % 2 eq 0)>
           <cfoutput> Page is even</cfoutput>
       <cfelse>
           <cfoutput> Page is odd</cfoutput>
       </cfif>
   </cfdocumentitem >
   Document body…
</cfdocument>

In this case, ColdFusion evaluates the cfdocumentitem tag body when it prints the header or footer, not when it evaluates the cfdocument tag body. This ensures that the cfdocument tag body is already evaluated and layed out and page number information is available when cfdocumentitem tag is being evaluated

A side effect of this is, if a variable is used inside both cfdocument and cfdocumentitem, cfdocumentitem tag will only see the final value of that variable. In the following example, cfdocumentitem will always get 11 as value of ‘I’ and hence all the headers will print ‘Chapter 11’.


<cfdocument format="pdf" >
<cfloop from=1 to=10 index="i">
   <cfdocumentsection>
   <cfdocumentitem type="header" >
       <cfoutput>Chapter #i#</cfoutput>
   </cfdocumentitem >
DocumentSection body
   </cfdocumentsection>
</cfloop>
</cfdocument>

To handle this, you can now pass the variable into the cfdocumentitem tag as a custom attribute, as shown in the following example:


<cfdocument format="pdf" >
   <cfloop from=1 to=10 index="i">
       <cfdocumentsection>
           <cfdocumentitem type="header" evalAtPrint="true" no = #i#>
               <cfoutput>Chapter #attributes.no#</cfoutput> 
               <cfif (CFDocument.currentPageNumber % 2 eq 0)>
                   <cfoutput> Page is even</cfoutput>
               <cfelse>
                   <cfoutput> Page is odd</cfoutput>
               </cfif>
           </cfdocumentitem >
           Document Body.
       </cfdocumentsection>
   </cfloop>
</cfdocument>

CFUnited Presentation - All about CFThread

2008-06-30T00:26:00.001-07:00

I did a presentation on "All about CFThread" at CFUnited this year and I think it went fairly well. It was a repeat session - once on friday 21st June and another on Saturday 22nd June and both the sessions had pretty good number of people attending it. Thanks a lot to all those attended. As promised, I have uploaded my presentation and the sample code up here.

Please feel free to ask me questions if you have any.

Back from hiatus

2008-06-27T06:37:00.001-07:00

I took a long long break from blogosphere and now that I am rejuvenated after CFUnited, I am back. :-) Thinking about it, it was a really long break - my last post was on Feb 18th which makes it 4 months. So what did I do in those last 4 months? Well lot of things ;-)

I GOT MARRIED :-) So thats the biggest thing that happened. I took a one month long vacation to get married, went to Mauritius for my honeymoon and we had a really great time. Here is one of our pics. In case you want to see more of these, you can head over to Flickr. :D

There was a more than one month long Cricket series called "Indian Premier League" and I watched each and every match ;-)
All of us in the ColdFusion team were quite busy (and still are) in research, planning and implentation for the Centaur. In other words, I was working for you guys.

Now that I am back, you should see some regular posts from me. The next one goes in a while.

Reading MP3 meta-data from ColdFusion

2008-02-18T05:33:00.001-08:00

On friday I posted about how to read the meta-data for Swf (flash) file from ColdFusion and on the same note, I think it will be cool to read the meta-data for MP3 file as well. Here is the specification for MP3 header and it is fairly straight forward to read the header using the spec.

ColdFusion 8 knows how to read MP3 file and it can provide a set of meta-data for it. Though this functionality is not exposed via any tag or function, you can still use it. The meta-data it provides are

Bit Rate : in kbps
Frequency : in Hz
Play duration : in second
Version : integer
Copy Right : true/false
Channel mode : integer

Here is an example code.

<cfset mp3File = createObject("java", "coldfusion.util.MP3File").init("C:\music.mp3")>
<cfoutput>mp3File.getBitRate() : #mp3File.getBitRate()# kbps</cfoutput><br>
<cfoutput>mp3File.getFrequency() : #mp3File.getFrequency()# Hz</cfoutput><br>
<cfoutput>mp3File.getVersion() : #mp3File.getVersion()#</cfoutput><br>
<cfoutput>mp3File.getDuration() : #mp3File.getDuration()# Sec</cfoutput><br>
<cfoutput>mp3File.isCopyRighted() : #mp3File.isCopyRighted()#</cfoutput><br>
<cfoutput>mp3File.getChannelMode() : #mp3File.getChannelMode()#</cfoutput><br>

Though you can use the above mentioned class (coldfusion.util.MP3File) without any problem, you must keep it in mind that it is unsupported and *might* change in future. Just a disclaimer !!! :-)

This class does not get you the ID3V1 or ID3V2 meta-data, but it should be easy for you to do it. For ID3V1, all you need to do is to read the last 128 bytes of the mp3 file and you have all the information required. Reading ID3V2 tag is little more involved and there are a number of open source java libraries available (common one being jidlib and jaudiotagger) which can be used.

Reading Flash (Swf) metadata from ColdFusion

2008-02-15T09:34:00.001-08:00

Wouldn't it be cool if you could read the meta-data of Swf file in ColdFusion? The most common use case I can think of is to find the dimension of swf movie so that you can set the dimension for the same in object tag while you are embedding the flash movie in your page. And the good news is that you can do that currently in ColdFusion with just 2-3 lines of code!! All you need to do is to create a TagDecoder and decoder the header with it. Here is the complete code.

<cfscript>
  fis = createObject("java", "java.io.FileInputStream").init("C:\test.swf"); // Create the FileInputStream
  decoder = createObject("java", "macromedia.swf.TagDecoder").init(fis); // create TagDecoder
  header = decoder.decodeHeader(); // Decode the header.
  fis.close();
  rect = header.size;
  WriteOutput("
    Is Compressed : #header.compressed#<br>
    Frame Count : #header.framecount#<br>
    Frame rate : #header.rate#<br>
    Version : #header.version#<br>
    Height : #rect.getHeight()#<br>
    Width : #rect.getWidth()#");
</cfscript>

The height and width value that you see here would be in twips which is defined as 1/20 of a point or 1/1440 inch. This means, for a 72 dpi screeen, you will have to divide it by 20 to get the height and width in pixels.

Adobe ColdFusion Survey

2008-01-14T10:55:00.001-08:00

As promised earlier, here they come. As part of the research for Centaur (next major version of ColdFusion aka ColdFusion 9), we need your feedback on ColdFusion 8 features, what enhancements do you need in them, what new features you want in the new release etc etc.

Adobe ColdFusion Survey

We would also like your inputs on platform and vendor support for upcoming ColdFusion release.

Platform and Vendor Support

Please do take some time to take this survey as this would really help us shape the new features in the upcoming version. THIS is YOUR chance to get your favorite features in !

DNS lookup Caching in ColdFusion/Java

2008-01-10T11:22:00.001-08:00

Two days back, I got an interesting question from our support team. The Customer in this case was using CFLdap which connected to a particular LDAP server and things were working fine. The problem came when they replaced this LDAP server with another LDAP server and assigned the same dns name to the new ldap server. Ideally any connection made henceforth should have worked with the new ldap server but actually that did not happen. ColdFusion started throwing error and it did not work until the ColdFusion server is restarted. Ever seen something similar?

It is obvious that it happend because the IP address was being cached for the host name as a result of which ColdFusion was still trying to connect to the old IP address even though the host name now pointed to a different IP address. This caching also applies for all other network protocol tags such as CFHTTP, CFFTP, CFFEED etc and is not limited to CFLDAP. It is actually the JVM that does this caching. When JVM is requested to resolve a host name, it does the host name to IP address resolution and caches the result. This result will be used for all subsequent lookups. It also caches the negative results. By that I mean, if the dns reolution fails, it caches the failed result for a certain period so that any lookup for that hostname in that period will not result into another resolution on network and will immediately return the failed result.

For more detail on this caching, check out the Javadoc for InetAddress class.

As per this doc, there are two parameters that control this caching

networkaddress.cache.ttl (default: -1)
Indicates the caching policy for successful name lookups from the name service. The value is specified as as integer to indicate the number of seconds to cache the successful lookup.
A value of -1 indicates "cache forever".

networkaddress.cache.negative.ttl (default: 10)
Indicates the caching policy for un-successful name lookups from the name service. The value is specified as as integer to indicate the number of seconds to cache the failure for un-successful lookups.
A value of 0 indicates "never cache". A value of -1 indicates "cache forever".

Now where do you specify this settying? You can specify this setting in <Java_home>/jre/lib/security/java.security file. For standalone ColdFusion server it will be in <ColdFusion _dir>/runtime/jre/lib/security/java.security file.

As you see, by default networkaddress.cache.ttl caches the result for ever and hence it is configured for best performance. Any change to this mean drop in performance. If you don't want to cache the resolved IP address for ever, as is the case here, you would need to change networkaddress.cache.ttl value to 60 seconds or 300 seconds or any value you feel suitable. You would not want to set it to 0 as that would mean "never cache" the result which might affect the performance significantly.

In which case you would want to change the value for networkaddress.cache.negative.ttl? That would be mostly in case when you want to cache the negative result for a longer time and in turn improving the performance. For example, if you are trying to connect to a hostname which can not be resolved to any ip address, and that happens very frequently, each of the call (as long as they are not in the same 10 sec window) would become very slow. Increasing this value would increase the performance but again you would not want to cache the negative result for ever.

After you change this setting, you will have to restart the ColdFusion server for this change to take effect.

ColdFusion 8 finalist among Dr. Dobb's Jolt Product Excellence Awards

2008-01-07T08:50:00.001-08:00

Good to see ColdFusion 8 as a finalist in Web Development tools category among Dr. Dobb's 18th Annual Jolt Product Excellence Awards. Here is the entire story and here is the list of finalists.

Adobe ColdFusion IDE Survey

2008-01-07T08:07:00.001-08:00

As part of our research on CF9 (Centaur), the ColdFusion team has come up with a survey to find out what features do you use and would like to have in an IDE for Coldfusion. Please take few minutes to take this quick survey.

http://www.surveymonkey.com/s.aspx?sm=321RrO9_2fWaP_2bdYMnmF9CuQ_3d_3d

Check out this space in few days for another survey to tell us what you would want to see in CF 9 !

Encrypted CFML Templates

2008-01-02T08:49:00.001-08:00

Coming back from the 10 days shutdown period that we had at Adobe, I am catching up with the blogs and I read this interesting post by Ray where he mentioned that some one got his cfml templates encrypted and then lost the originals. Hmmm.. that is a situation, I would never want to be in. In case you have not figured it out yet, it is not possible to get back the original source. Let me explain.

Why do we need to encrypt cfml templates? To protect the cfml code and your intellectual properties (IP) in case you want to distribute your CF application. Right? In pre-CFMX7 era, ColdFusion used to have "cfencode" utility which used to encrypt the template which you can then distribute to anyone. ColdFusion used to know how to decrypt that and then how to execute that. However, this was not a good solution as people came up with utility to decode this encrypted file. And THAT means that your code and IP was not protected at all. This had another disadvantage - template execution was slower as it meant decrypting the template first and then compiling/executing it.

With CFMX7, this changed. ColdFusion came up with sourceless deployment to make it nearly impossible to decode it back to the original source. When you use 'cfcompile' with deploy option, your cfml template is compiled to java byte code (class) but the file in which this byte code is written still retains the same template name. To give an example, if you run cfcompile on "hello.cfm", the output would still be "hello.cfm" but it will actually be java class file. To confirm this, just open one of such encrypted file in a hex editor and you would see the first 8 bytes will be CAFEBABE - the magic number for java class files. This file can then be distributed to your customers. For execution, ColdFusion works smart - when request comes for this template, it sees that it is not a cfml template but a class file, skips parsing and compiling the template, and directly proceeds with the execution.

As you see, there is no encryption and no key involved here. The encrypted file is simply a java class file and I have not heard of any utility that can decompile this class file back to cfm source. Though there are many Java decompilers available which can convert the class file to approximate java source file, it will be a huge huge task to write a decompiler which can generate cfm code for class file. One needs to know how CF does the code generation, know a great deal about bytecode instructions and apply lots and lots of heuristic to get the approximate cfm source. This is definitely not an easy thing to do even if some one breaks open the CF engine to see how it generates the java code from cfm code.

The bottom line is - in case you use sourceless deployment, ALWAYS backup your original source.

Google to roll out wikipedia rival - Google Knol

2007-12-19T11:39:00.001-08:00

Google is coming out with Google Knol (Knowledge) which is exactly what Wikipedia is all about. Read on

http://googleblog.blogspot.com/2007/12/encouraging-people-to-contribute.html

http://searchengineland.com/071213-213400.php

Firefox (Adblock plus) blocks even Google adsense

2007-12-14T14:29:00.001-08:00

I have been using my new laptop at work (my old one crashed at MAX) for two weeks and its been pretty good. Not because the laptop is something extraordinary, its because I installed the latest versions of all the softwares and plugins I use. And the most pleasant part of it is - it has been completely ads free. I did not notice it earlier but once I landed up on sys-con's site to read some article and I was pleasantly surprised to find no irritating video ad there. For a moment, I thought that may be sys-con has grown up but that was just for a moment :-). The snapshot shows it all.

Firefox

Its actually a firefox plugin adblock plus that does it automatically. The most interesting thing was that it even removed google text ads. Its interesting because we have always seen Google promoting firefox and firefox as a browser closer to Google. So now a firefox plugin that blocks google adsense and in turn hurts Google revenue is certainly not going to go well with Google. It will be interesting to see how Google plays it.

So how exactly does adblock plus block the ads? Gecko engine - the engine underneath Firefox, uses a content policy which decides which content should be loaded and which should not. Adblock defines its own policy which gets added to the browser's policy. It comes with a predefined huge collection of filters and if the url to be loaded matches any of that, it does not let the browser make a request for it. For google adsense, the url contains 'pagead2' which you can see in the adblock preference below.

Its also going to hurt many bloggers as lot of them earn substantially from ads and mainly adsense. Even my blog page contains adsense but I dont mind adblock plus as I haven't yet earned anything from it anyway. I'm loving it. :-)

Images and CFDocument performance

2007-12-14T12:31:00.001-08:00

Sometime back one of our customer reported that using png images in cfdocument makes it very very slow. I could not replicate it with any png image but it did happen with his png image.

Today, Andy reported a similar issue but this time it was for jpeg images. In both the cases, performance hit is huge and it does not happen with all the images.

I spent a significant amount of time debugging it today and it turns out that the reason for both the issues are same - there is something special about these images and that is colorspace. All these images actually had a different colorspace than what java imaging uses. Because of this, when we need the pixel values of image to print it on pdf (by calling BufferedImage.getRGB()), it tries to convert this colorspace to RGB colorspace and that is very very costly. That is where the entire time goes. So how do you fix it? I opened all the images in an image editor and saved it again. This time it got saved in standard RGB colorspace and the time taken to create the pdf got reduced from 110 sec to 1.5 seconds. That is huge!!! Isn't it? But can you control all the images over the web? NO.. right? Read on.. there is more to this story.

A little bit of looking up on web pointed me to this Sun bug which is the exact same bug which we were hitting. Thankfully it got fixed in mustang i.e JDK1.6 which ColdFusion8 uses by default. But hey wait a second.. Didn't Andy say that he is seeing it on ColdFusion 8? why do we still see this happening when it is fixed in JDK1.6? It appears that this bug was fixed only in core JDK api but not in JAI (Java advanced imaging) codecLib that ColdFusion 8 uses. So what do we do now? You can do either of these two

Remove clibwrapper_jiio.jar from "lib" folder.
Or, set this system property to the JVM. -Dcom.sun.media.imageio.disableCodecLib=true . You can set this in [cf-install_dir]/runtime/bin/jvm.config if you are using standalone coldfusion server.

You should keep in mind that codecLib libraries are native libraries which are meant to increase the java imaging performance. So disabling it might degrade the performance of CFImage somewhere. Also keep in mind that removing this jar or disabling codecLib will not result into any loss of functionality - it just means that all image operations will be pure java.

There is another related Sun's bug which I thought might be useful to you. Image loading might get very slow if the server is running in debug mode. Your server is running in debug mode if you see something like this in your jvm.config or VM startup option.

-Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005

This bug got introduced in JDK1.6 and does not exist on 1.5. So make sure that you are not running the server in debug mode.

To summarise, you can do following to increase the performance of image loading in cfdocument.

If you are on JDK1.5, there is not much you can do. The only option is to change the colorspace of images. I will try to see if we can address this in ColdFusion code.
If you are on JDK1.6,
- Disable codecLib as mentioned above.
- Disable debug mode by removing the complete debug string mentioned above
In addition to this, you might want to use 'localurl' attribute for cfdocument tag. See this for more details.

Missing text in pdf created by CFDocument

2007-12-12T10:27:00.001-08:00

I had heard about people not getting images in the generated pdf but this one was something new and spooky. Yesterday Andy Matthews posted it on cf-talk where he says that for his content, though a pdf containing 15 pages gets created but only the first page has some content and rest of the pages are blank. I ran his code on my machine and sure enough it was happening. All the content including text and image on all the pages except the first page was gone. So what was wrong? I simplified his code and here is a simple example which you can try.

<cfdocument format="pdf">
Text Outside div
<div style="width:300px;overflow:auto;">
 This is text inside the div. Will it show up?
</div><br>
</cfdocument>

In the generated pdf you will see only "Text Outside div". Spooky.. isn't it? The reason it happened was because of css style 'overflow:auto'. The rendering engine used by CFDocument underneath does not handle overflow:auto and it simply ignores the content in the div. The weird part is - it considers that content for all rendering calculations including page number and page break calculation.

Something you should watch out for if you use css styles in cfdocument. A workaround for the time being is not to use overflow:auto style. So modifying the above code like this will make it work.

<cfdocument format="pdf">
Text Outside div
<div style="width:300px;">
 This is text inside the div. Will it show up?
</div><br>
</cfdocument>

64 bit Support on ColdFusion 8 (Windows, Linux and Leopard)

2007-11-21T08:04:00.001-08:00

ColdFusion team is working to get ColdFusion 8 working on 64 Bit platforms(Windows, Linux and Leopard). If you are interested and want to participate in the testing of early releases of 64 Bit support on ColdFusion 8 on these platforms, please enroll yourself for prerelease program here.

JRun updater 7 released

2007-11-21T08:02:00.001-08:00

JRun 4 updater 7 got released last week and is publicly available for download. It now supports a host of new platforms, includes upgrades to existing software components and fixes for many critical bugs.

Some of the important platform support include

JDK 1.6 support
Support for Windows Vista.
Mac OS X 10.4 on Intel
64-bit Support on Solaris-SPARC.
IIS7 support.
Apache 2.2.x support
SunOne Webserver 7
IBM JDK 1.5

Another important reason why you might want this is - a very significant performance improvement in web clustering. Check out the detailed release notes here.

The Week at Max

2007-10-10T02:38:00.001-07:00

Wow ! What an incredible week it was. A week that saw the largest ever MAX, a week that saw the largest get together of ColdFusion community, a week that saw the best ever visibility of CF at MAX general sessions, a week that saw U2's Bono (err.. his fake??) having fun at a tech conference like MAX and a week that saw lot of cool and amazing Adobe technologies shaping up.

It also had moments of a Hollywood thriller for me. I was presenting a session on "Leveraging ColdFusion for AIR application". My first session was on Tuesday, 2nd October at 4 o'clock and at around 2 o'clock my laptop crashed and after that it just refused to boot up. Luckily I had the backup of my ppt and some demo files and hence I decided to use Sanjeev's (Pdf and compiler guy in the CF team) laptop to make the presentation. I just had one and half hour and I needed to setup everything back on his machine - install the air runtime, air sdk, cfeclipse, copy all my files etc and keep the setup ready to run whatever demos I was left with. To add to this, somehow there was no Internet available in the conference area - So where to get AIR runtime and SDK from? Sanjeev suggested that may be AIR guys might have it and so we ran across to the AIR park to get those on my flash drive. I can't tell you what those one and half hour were like :-). I barely managed to get the presentation ready in time and thankfully the presentation went well.

As if this was not enough, the adventure continued for the repeat session as well. No this time the laptop didn't crash.. but this time I could not find Sanjeev so that I could take the laptop from him. I just had half an hour left for the session and I did not have any laptop. And I didn't even have the patience for setting it all up once again on another laptop. Finally found him just 15 minutes before my session while he was chatting with BEA guys. phew..

Anyway, here is my presentation and some of the demo files if you are interested to take a look at that. I could not upload it earlier as I got a temporary laptop just yesterday. I will upload the flex part of demo in a day or two.

If you were there at MAX, you must have seen Hemant talking about ColdFusion and AIR at sneak peek. Though this was just a prototype, it surely gives an idea what can be done in CF for air apps. It would be great if you can share what exactly or what else would you like to see in ColdFusion to help you build AIR application easily. Thoughts??

CFDocument hotfix for ColdFusion 8 released

2007-09-20T09:47:00.001-07:00

We have released a cumulative hotfix for CFDocument related issues in ColdFusion 8. Bugs being addressed in this hotfix are

PDF/Flashpaper document generated in CF8 are nearly 70% scaled down than CFMX 7. This was making document appear smaller in CF8 as compared to CF7.
Using cfdocumentitem in a loop inside cfdocument causes the error, "routines cannot be declared more than once".
StackOverFlowError when a large document with table is created. A simple report with over 1500 table row causes this problem. More complex, but shorter documents have also resulted in this error. This was a pretty nasty one and it deserves a separate blog entry :-).

You can download the hotfix directly from here. You would need to extract the hotfix jar from the downloaded zip file and apply that jar.

ColdFusion 8 : Changes with File Upload

2007-08-16T10:31:00.001-07:00

Till ColdFusion 7, it was not possible to upload a large file. The server could not handle uploading a file of probably anything more than 300-400 MB. And on top of that, if there are multiple people trying to upload large files simultaneously to the server, only thing that you can do is to pray :-)

You might wonder and I agree with you, that its a huge limitation. So why was it this way ? Well.. that was because the server had to keep the entire uploaded file in the memory. Whoa.. Why the heck does the server need to keep the entire uploaded file in memory? - For a single reason of serving the function GetHttpRequestData(). As you know, this function returns a 'struct' containing the metadata for the Http request that comes to the server. Since this struct also includes request body, and the uploaded file is part of the request, the file has to be retained in the memory in order to honor this function.

With ColdFusion 8, we have removed the limitation on file upload size. So now you can upload files of any size without causing any issue with the server. I have successfully tried with 5 GB file because that was the biggest I could get hold of but you can absolutely upload files of any size. ColdFusion nicely handles it, reads it the way it should without requiring much memory.

But hey, hold on a sec.. What about the function GetHttpRequestData()? Wasn't that the reason for this limitation in the first place? Well, the function GetHttpRequestData() works perfectly fine except in this one case - In the multipart request, i.e; request in which a file is uploaded - "content" in the returned struct will be empty. If you need to get the uploaded file content in memory, you can always read the uploaded file and keep it in memory. So that should not be much we are asking for. Are we?

If in case, you absolutely want the content from GetHttpRequestData() and file upload size is of no concern to you, we allow you to revert back to ColdFusion 7 behaviour. All you need to do for this is to set the Java system property "coldfusion.markResetForMultiPart" to "true" and you are back to the old behavior.

Unlimited aliases with Gmail and SPAM protection

2007-08-09T11:54:00.001-07:00

Though this hack in Gmail has been known for a while, I read about it today and found it interesting enough to share it. As Amit decribes in his post, GMAIL lets you create unlimited number of aliases for your gmail id and you can then give these aliases on other websites without worrying about spams. The trick is to use '+' addressing!

As he says, gmail ignores anything between '+' and '@' in the email address. So a mail sent to paris.hilton+model@gmail.com or paris.hilton+celebrity@gmail.com will all go to paris.hilton@gmail.com. In the received mail, "To" will still point to the original id to which it was sent which makes it easy to create spam filters based on the 'To' address. If used intelligently, you also get to know which site leaks your address and creates SPAM. So, if I have to give my mail id to any website xyz.com, I can give 'mymailid+xyz@gmail.com'. If I get the spam with 'to' address as mymailid+xyz@gmail.com, I get to know that its xyz.com that had leaked the id and creates spam. Pretty cool..

ColdFusion and WebService : "class file has wrong version"

2007-08-08T12:39:00.001-07:00

I can't even count the number of times I have seen this error while QA folks were running webservices regressions for ColdFusion 8. Everytime, someone was running webservices tests on a J2EE configuration they used to get an error :

coldfusion.jsp.CompilationFailedException: Errors reported by Java compiler:
....
class file has wrong version 49.0, should be 48.0

I saw the same error posted in the public forum where the user had deployed ColdFusion 8 on JRun with JDK1.5 and was trying to invoke webservice. And hence this blog entry :-)

This error comes because of incompatibility between the tools.jar and the JVM version. ColdFusion 8 ships Java 1.6's tools.jar (you will find it in cfusion/lib) which is of course incompatible with any other JDK/JRE version. You will see the same behavior with ColdFusion 7 as well, if you run it on a JVM other than 1.4. So the rule of thumb is - if ColdFusion is running on a JVM version other than the shipped one, you must ensure that the tools.jar is in synch. In short, you will need to copy JDK's tools.jar to cfusion/lib.

Using CFFeed with URL sending compressed content

2007-08-03T07:49:00.001-07:00

Today someone posted on ColdFusion forum regarding this problem where cffeed was not able to handle a particular URL and it was throwing an error. The URL which he tried was http://movies.msn.com/rss/topcelebs and it failed with an error

"Unable to read the source URL.
unknown compression method"

The reason it happens is - The URL returns the response in gzip compressed format only. So when ColdFusion sent a request to this URL and asked for uncompressed data, it could not get anything and hence it was unable to read it. A simpe workaround for this is to use cfhttp to fetch the content, write to a temporary file and then use the cffeed tag to read this file. Important thing to keep in mind here is to set an additional header in the cfhttp tag using cfhttpparam to indicate that it can accept compressed data as well.

Here is the modified code where it first tries cffeed with the URL. If that fails, then it tries to use cfhttp to fetch the content and writes to a temporary file and then uses it in cffeed.

<cfset tempDir=GetTempDirectory()>
<cfset tempFile = GetTempFile(tempDir, "myfeed")>
<cfset tempFileName = GetFileInfo(tempFile).name>
<cftry>
    <cffeed action="read" source="http://movies.msn.com/rss/topcelebs" name="feedInStruct" >
<cfcatch any>
    <cfhttp url="http://movies.msn.com/rss/topcelebs" path="#tempDir#" file="#tempFileName#">
        <cfhttpparam type="header" name="Accept-Encoding" value="compress,gzip,deflate">
    </cfhttp>
    <cffeed action="read" source="#tempFile#" name="feedInStruct" >
</cfcatch>
</cftry>
<cfif FileExists(tempFile)>
    <cfset FileDelete(tempFile)>
</cfif>

<cfdump var="#feedInStruct#">

ColdFusion 8 : IsInstanceOf

2007-08-02T08:52:00.001-07:00

If you use lot of CFC inside your ColdFusion application, I am sure you would have come across a situation where you would need to know whether the object is an instance of a particular CFC. This is specially needed when you have components extending other component or you are passing the objects around. ColdFusion 8 introduces a new function IsInstanceOf to do exactly the same. It becomes even more useful after we have interfaces in ColdFusion. And the icing on the cake is that it works even with java objects which means that you can use this function to find out if a particular object is of a particular java class type.

Here is how the function looks.

IsInstanceOf(object, typeName)

where typeName is name of the component/Interface or fully qualified java class name.

It returns 'true' if

The object passed is an instance of a component which is same as specified type or inherits it or implements the specified interface. Just to be clear, a component 'A' inherits a component 'B' if A or any of its super component extends 'B'. Similarly a Component 'A' implements an interface 'B' if A or any of its super component, implements interface 'B' or any of the interface that 'A' or its parents implement, extends from the specified interface.
The object passed is an instance of a java class (created using cfobject or createObject for java class) which is same as specified class name or inherits the specified class name or implements the specified interface.

Here is an example

Intf.cfc

<cfinterface>
<cffunction name = "foo">
</cffunction>
</cfinterface>

Comp.cfc

<cfcomponent implements="Intf">
<cffunction name = "foo">
   <cfoutput>In method foo</cfoutput>
</cffunction>
</cfcomponent>

test.cfm

<cfset obj = CreateObject("Component", "Comp")>
<!--- Create a Java object --->
<cfset javaObj = CreateObject("java", "java.lang.StringBuffer")>

<cfoutput>object is of type Comp : #IsInstanceOf(obj, "Comp")#</cfoutput><br>
<cfoutput>object is of type Intf : #IsInstanceOf(obj, "Intf")#</cfoutput><br>
<cfoutput>java object is of type String : #IsInstanceOf(javaobj, "java.lang.String")#</cfoutput><br>
<cfoutput>java object is of type StringBuffer : #IsInstanceOf(javaobj, "java.lang.StringBuffer")#</cfoutput><br>

Printers harmful to health..

2007-08-01T04:46:00.001-07:00

I am not kidding. There was a recent study done by Queensland University of Technology, Australia which said that printers are hazardous and can cause sever illness including cancer ! Check out the report here.

ColdFusion 8 performance whitepaper

2007-08-01T03:58:00.001-07:00

I am glad to see the ColdFusion community so excited with CF 8 performance. We got very encouraging response regarding the performance in our pre-release forums and we are thankful to everyone who gave such invaluable feedbacks. Though people started seeing a very good performance gain in their application on ColdFusion 8, we never talked about the performance benchmark numbers until CFUnited 2007, where Ben showed the performance numbers for ColdFusion 8. No wonder this is the #1reason in Top 8 reasons why you want ColdFusion 8 :-).

Extracting performance up to this extent was not an easy task. We analyzed nearly 2.4 million lines of real life CF application code, to zero-in on the most commonly used tags and functions. Main challenge was after that - analyze the generated java code for all the tags, change the compiler to generate more optimized code, run it through profiler and optimize CF engine for each of the tags and functions and their various combinations. And this went on and on in many many iterations. Overall it was real fun :-)

I still remember the most exciting moment when I ran the load test for CFC, after I had made some code changes, and the result was freaking unbelievable. It looked too good to be true and I was literally running around with the code changes to run it on other machines and verify it . That one small change gave nearly 6x gain :-). It is not that the CF7 code was inefficient or poorly written. It was only matter of extracting juices as much as possible and putting some smart intelligence.

Check out the ColdFusion 8 performance whitepaper which talks in much detail about the performance numbers for different areas, the methodology used for benchmarking etc. Manju Kiran, who was my QA-buddy for most of the features I worked on, did a tremendous job in setting up and running the benchmark and creating the meat of this wonderful document (and of course keeping me on toes).

ListToArray in ColdFusion 8

2007-07-19T13:00:00.001-07:00

There has been lots and lots of discussion in CF blogs and forums about ListToArray not supporting empty elements. Seeing which I had blogged about a simple way which could do the same. Thanks to Charlie Griefer for pointing out the problem in it and then thanks to Ben Nadel and Andrew Clark for pointing me in the right direction.

Though it got pretty late, I was able to sneak-in this change in ColdFusion 8. ListToArray() now takes an additional optional argument "includeEmptyElements", which if 'true' will include the empty elements of list into the array. Default is of course 'false'. It also takes care of empty elements at the end of list and multiple delimiters. Here is how the function looks

ListToArray(list, delimiter, includeEmptyElements) returns Array

Lets take a look at couple of examples to see it working

<cfset list = "a,b,,c, ,d,,">
<cfset arr = ListToArray(list, ',', true)>
<cfdump var="#arr#">

Here is how the output looks.

Here is another example.

<cfset list = "one,/$/,six">
<cfset arr = listToArray(list, ",$/",true)>
<cfdump var="#arr#">

The output for which looks like this

Though we wanted to, there was just not enough time to make similar change in all the list functions for CF 8. Something for CF 9 :-)

Enhancements to CFDocument in ColdFusion 8

2007-07-19T04:51:00.001-07:00

As you all know, CFDocument tag is used to easily create pdf or flashpaper documents from HTML/CFML content. ColdFusion 8 has added lot of enhancements to it and in this post we will talk about those enhancements.

1. Bookmark : You can create bookmarks for each section of the pdf using "bookmark" attribute. The bookmark created is of only one level and its name is set to the documentsection's name. Here is a sample code for creating pdf with bookmarks.

<cfdocument format="PDF" bookmark="yes">
   <cfdocumentsection name="Introduction">
       <h3>Introduction</h3>
       <p>The introduction goes here.</p>
   </cfdocumentsection>
   <cfdocumentsection name="Chapter 1">
       <h3>Chapter 1: Getting Started</h3>
       <p>Chapter 1 goes here.</p>
   </cfdocumentsection>
   <cfdocumentsection name="Chapter 2">
       <h3>Chapter 2: Building Applications</h3>
       <p>Chapter 2 goes here.</p>
   </cfdocumentsection>
   <cfdocumentsection name="Conclusion">
       <h3>Conclusion</h3>
       <p>The conclusion goes here.</p>
   </cfdocumentsection>
</cfdocument>

2. Proxy Support : With ColdFusion 8, you can now provide proxy configuration to cfdocument for retrieving external content like images. This will be useful in situation where the machine hosting ColdFusion is connected to the external world via a proxy. The new attributes added for this (which are self explanatory) are listed below

proxyHost
proxyPort
proxyUser
proxyPort

3. Content from URL : Though this was added in 7.0.2, I think it makes sense to add here because it was not there in 7. :-) Have you ever created or wanted to create a pdf from a web page? If yes, then the new attribute "src" in cfdocument and cfdocumentsection tag makes it very easy to do this. Here is an example to do this.

4. Basic Authentication : If the CFDocument body contains a resource (e.g; image or URL) that is protected with basic authentication, ColdFusion 7 can not retrieve it and it was one of the reason for getting "red-x" for images. (See my old post on this). ColdFusion 8 addresses this by adding these two attributes to cfdocument and cfdocumentsection tag.

authuser - user name to be used for basic authentication.
authpassword - password to be used for basic authentication.

5. User Agent : There are some cases where the web server is configured to allow requests only from a certain set of browsers (User agents to be precise) to prevent spiders and bots from overloading the server. In ColdFusion 7, when CFDocument creates a URLConnection for an image, it sends a "User-Agent" header, that looks like "User-Agent:Java/1.4.2_07", in the HTTP request. If the web server does not recognize "Java" user-agent, it returns a status code of 404 (resource not found) and hence the images can not be displayed. ColdFusion 8 adds a new attribute "user-agent" to address this.

user-agent : User agent to be used for making http connection. The default value will now be "ColdFusion". If this also does not work, you can use the same string that browsers like IE or firefox use. This attribute has been added to <cfdocument> and <cfdocumentsection> tag.

6. PDF name : When a pdf generated by cfdocument is sent to the browser and you try to save it, the browser will prompt with the cfm name for the pdf which is generally not desirable. With ColdFusion 8, you can provide the appropriate name in "saveAsName" attribute of cfdocument.

saveAsName - Name that appears in the 'saveAs' dialog when you try to save the pdf from the browser (Generally File -> saveAs). This name will not work if you use "save" dialog of the pdf plugin.

Here is a sample code for the same.

<cfdocument format="pdf" saveAsName="mypdf">
 <p>This is a PDF document.</p>
</cfdocument>

7. Local URLs : When CFDocument body contains a relative URL, ColdFusion will resolve the relative URL to an absolute URL and will send an HTTP request for this url. A side effect of this is - Server ends up sending HTTP request even for local URL or images that are lying on the local file system which obviously hurts the performance. In ColdFusion 8, we have added a new attribute "localURL" to cfdocument tag which if enabled, will try to resolve the relative URLs as file on the local machine.

localURL : "true" | "false" - It should be enabled if the images used in cfdocument body are on the local machine. This would make the cfdocument engine retrieve the images directly from the file system rather then asking the server for it over http.

This attribute helps reducing the load from the server so that the same web server thread can now serve user request instead of serving local images to CFDocument. This also addresses some of the "missing image" problems which I mentioned here. Here is a sample code using this attribute.

<cfdocument format="PDF" localUrl="true">
 <table>
  <tr>
     <td>bird</td>
     <td><image src="images/bird.jpg"></td>
  </tr>
  <tr>
     <td>Rose</td>
     <td><image src="images/rose.jpg"></td>
  </tr>
 </table>
</cfdocument>

8. Section Page Counts : CFDOCUMENT scope contains two new variables which give you the page counts for document section.

TOTALSECTIONPAGECOUNT - total no of pages in the current section
CURRENTSECTIONPAGENUMBER - Current page number in the current section.

9. Dynamic header and footer : CFDOCUMENT scope variables can now be used in expressions inside <cfdocumentitem> which makes it possible to have dynamic header and footers. You can now build logic for header/footer content based on the page number. Here is a sample code which prints section title if the page is even and prints the page no otherwise. Below is a code snippet which creates a dynamic header.

    <cfdocumentitem type="header">
       <cfif (cfdocument.currentpagenumber mod 2) is 0>
           <cfoutput>#sectionTitle#</cfoutput>
       <cfelse>
           <cfoutput>#cfdocument.currentpagenumber# of #cfdocument.TOTALPAGECOUNT#</cfoutput>
        </cfif>
   </cfdocumentitem>

10. We have also fixed most of the CFDocument related bugs e.g text chopping, image cropping or red-x, image scaling, "Document has no pages" etc.

New File I/O in ColdFusion 8 - Part II

2007-07-17T09:22:00.000-07:00

In my previous post I talked about how new file I/O functions address working with large files. In this post I will talk about other file functions that have been added in ColdFusion 8. Most of these functions are equivalent to the cffile operations and we have retained the same behaviour as cffile. A side effect of that is - relative paths used in these functions are relative to the temp directory. I don't really like that and I believe that it should have been relative to the template. But since these functions were supposed to replicate cffile behaviour, we had to live with it. :-)

Here we go with the list of those new functions

FileRead(filepath, [charset]) - Similar to cffile, this function reads the entire content of a text file and returns the read content. you can also opitonally pass the charset to be used to read the text file.

FileReadBinary(filepath) - This reads the entire content of a binary file and returns the byte array.

FileWrite(filepath, textdata | binarydata, [charset]) - Writes the specified content to the file. The content can be binary as well as text. If the specified content is a text data, you can optionally specify the charset so that the data can be written properly to the file.

FileCopy(source, destination) - As the name suggests, it copies the source file to destination file. Similar to cffile, if the destination is a directory, then source will be copied to that directory otherwise source file will be copied to the destination file.

FileMove(source, destination) - Moves the file from source to destination. Here again, if the destination is a directory, then source is moved under destination directory. Otherwise source is renamed to the destination.

FileDelete(filepath) - Deletes the specified file. The only important thing to note here is that if the file is readOnly, it will not be deleted.

FileSetAttribute(filepath, attribute) - Sets the attributes on file. Applies to Windows. 'attribute' here is a comma-delimited list of attributes to set on the file. Possible attribute values are "readOnly" | "hidden" | "normal".

FileSetAccessMode(filepath, mode) - Sets the file access mode for Unix or Linux systems where the mode is octal values of UNIX chmod command assigned to owner, group, and other, respectively. To give full permission to everyone for a file, the mode should be 777.

GetFileInfo(filepath) - So far till ColdFusion 7, there was no good way to find information like size, last modified date etc about a file. Only way you could do that was to use cfdirectory tag to list the directory, get the query from it, loop over the query until you hit the desired file and then fetch the required metadata. The new function GetFileInfo in ColdFusion 8 provides an easy way to get all the meta-data about a file or directory. This returns a struct which is described below.

name - Name of the file/directory specified. This is just the file name and not the absolute path.
path - Full path of the file/directory.
parent - Full path of the parent directory.
type - "directory" if the filepath is a directory else "file".
size - size of the file in bytes.
lastmodified - DateTime at which this file/directory was last modified.
canRead - "true" if this file/directory has 'read' permission. "false" Otherwise.
canWrite - "true" if this file/directory has 'write' permission. "false" Otherwise.
isHidden - "true" if this file/directory is hidden. "false" Otherwise.

Few more details on File handle

2007-07-11T13:49:00.000-07:00

In my previous post I talked about file object that you get on FileOpen() which is nothing but handle to the native file. Did you ever try to dump this object? This file object provides lot of valuable information. If I run this code below


<cfset myfile = FileOpen("C:\cfunited_notes.txt")>
<cfdump var="#myfile#">
<cfloop condition="Not FileIsEOF(myfile)">
    <cfset line = FileReadLine(myfile)>
    <cfoutput>#line#</cfoutput><br>
</cfloop>
<cfset FileClose(myfile)>

this is what gets dumped.

As you can see, it gives you information like lastmodified time, mode in which the file was opened, name, path, size in bytes and status of this file object whether this is still open or closed.

This object acts very much like a struct. So you can access these data from the file object using the simple dot notation. For example, to find out the last modified time, you can use fileObj.lastmodified

That gives another useful tool in your hand. While you are writing a file incrementally, you can easily find out the size of the file written so far using fileObj.size. This will be very helpful if you want to build a logging application where log files are rotated. While you are logging the data, as soon as the file size becomes more than your certain limit, you can close the file object and start writing to a new log file.

New File I/O in ColdFusion 8

2007-07-10T12:35:00.000-07:00

Till now we have been using <cffile> for all kind of file operations and it does a very good job. If you want to read a file, give the file to this tag and this tag gives you the read content. If you want to write content to a file, you give the content and file name to this tag and it will do that. You want to copy/delete/move your files, this tag will do all of that. All very simple and short. But there are two particular issues which <cffile> does not address.

1. Reading/writing big files - Since <cffile> is a tag, it can only perform one-shot operations. So, to read, it has to read everything in one shot and to write, you have to provide the entire content and that means that <cffile> will have to keep the entire content in memory. It is not of much concern if the file size is just few KBs but as the size increases beyond 100 kb or when it reaches few megs, it can really hurt. It would create a memory crunch on the server and if the load is high and there are many read/write happening simultaneously with large files, it can even lead to OutOfMemory error in server. Apart from creating memory crunch, it will also slow down the server because VM would need to allocate/deallocate larger chunk of memory which would lead to larger and frequent Garbage Collection cycle. At this point, you might ask, why would I ever read or write such a big file? Well I can think of few

You need to process the data that comes in a flat file
csv parsing
Finding the mime type of a file like mp3, image, video etc
you want to create a log viewer
... many more

You get the idea.. right?

2. Again since <cffile> is a tag, it is not very easy to use inside cfscript. Either you have to move out of cfscript to use this tag or you wrap this tag in a function and call that function. Though thats true with all the tags but cffile is so commonly used that this looks like a limitation.

New File I/O introduced in ColdFusion 8 addresses both these problems. New File I/O is all based on functions and hence that automatically takes care of problem 2. That means you no longer need to use cffile if you are inside cfscript. I will give more details on handling problem 2 in my next post. In this post I will mainly focus on reading/writing files in chunk using new IO .

The new I/O is based on the same philosophy that is used in other languages i.e;

You first open a file
Perform read/write operations on it
and close the file.

Lets see each of the steps in little detail.

Step 1 : Open a file : Here is the function to open a file

FileOpen(filepath [,mode] [,charset]) -> fileobject

Both mode and charset here and optional. Mode can be "read", "readBinary", "write" or "append"

"read" mode, which is default, is used to read a text file and hence any read operation will give you text data from it. When the file is a text file, you can also optionally specify the charset of the file. So if the file contains UTF-8 or UTF-16 characters (or characters from any other charset), you need to specify it while opening the file.

"readBinary" is used to read a binary file and hence any read operation will give you the binary data i.e byte array.

"write" mode will open the file in write mode which means that if the file already exists, it will be overwritten.

"append" mode, as the name suggests, will open the file in append mode which means that any write operation on that file object will write it at the end of file.

FileOpen function returns you a handle to the native file and you need to use this handle for all further read/write operation. Of course you should keep in mind that you can not perform "read" operation on a file handle that was opened in "write" mode and vice versa.

Step 2 : Do Read/Write operations : Once you get the handle to file object, you can perform multiple read/write operations using this handle. There are several functions to do that.

2A. Read Operation :

i) FileRead(fileobj, no of character/bytes to read) : This provides you a way to read a chunk of data (say 1 kb at a time) from the file at a time. Since you only read a chink of data at a time, it does not create memory crunch on the server. Since this is read operation, file must have been opened in "read" or "readBinary" mode. Depending on which mode the file was opened, this function will return the text or binary data read. One thing to note here - If the data remaining is less than the requested size, this method will return you only the remainign data. i.e if 100 character are remaining in the file being read, and you request for 1000 characters, it will return you 100 characters only.

ii) FileReadLine(fileobject) - This reads one line from the text file. To call this method, the file must have been opened in "read" mode.

Both these read operations can be called multiple times until you reach end of the file. One the end of file has reached, any further read call will result into an "EndOfFile" error. So in order to avoid this error, you should always check whether you have reached the end of file. And the function to do that is

FileIsEOF(fileobj) : Just to be more clear, EOF here stands for "End of File". This function will return true if the end of file has been reached otherwise will return false.

Here are few examples of reading content from file
Read 1 kb binary data at a time.

<cfscript>
    myfile = FileOpen("c:\temp\song.mp3", "readbinary");
    while (! FileIsEOF(myfile)) { // continue the loop if the end of file has not reached
       x = FileRead(myfile, 1024); // read 1 kb binary data
       ...// process this binary data..
    }
</cfscript>

Process a text file line by line

<cfscript>
    myfile = FileOpen("c:\temp\myfile.txt", "read");
    while (! FileIsEOF(myfile)) { // continue the loop if the end of file has not reached
        x = FileReadLine(myfile); // read a line
        ...// process this line..
    }
</cfscript>

2B. Write operation

i)FileWrite(fileobject, content) - This will add the text or binary content to the file. The file must have been opened in "write" or "append" mode.

ii) FileWriteLine(fileobject, text) - This will add the text followed by a new line character to the file. Here again, the file must have been opened in "write" or "append" mode.

You might wonder that if both the write operations add the content to the file, whats the difference between "write" and "append" mode? The difference is only at the time of opening the file. As I said earlier, opening the file in "write" mode will overwrite the file if already existsed and put the file pointer at the the beginning of file. Whereas opening file in "append" mode will simply put the file pointer at the end of file.

Any subsequent "write" calls, irrespective of "write" or "append" mode, will append the content to the file.

Here is an example of writing content to a file. This reads one line from an input file, does some processing on it, and writes the resultant data to another file.


<cfscript>
    infile = FileOpen("c:\temp\input.txt", "read");
    outfile = FileOpen("C:\temp\result.txt", "write");
    while (! FileIsEOF(infile)) { // continue the loop if the end of file has not reached
        x = FileReadLine(infile); // read a line
        data = processLine(x);
        FileWriteLine(outfile, data);
    }
</cfscript>

Step 3 : Close the file : Once you are done with read/write operations, you *must* close the handle to file. And the way to do that is using function

FileClose(fileobj)

What if you don't close the file object? Well, that file will remain locked by the server as long as the file is open, and no other process can modify/rename or delete that file.
You might also ask, why does not ColdFusion automatically take care of closing the file? Why should the developer be bothered about it? Well.. ColdFusion does take care of it when the file object goes out of scope and if it is not kept in any accessible scopes but you can never be certain when exactly this will happen. This might happen immediately or this might happen hours later :-).
Bottomline, you should make it a practice to call FileClose() once you are done with the file object.
Just to show its usage, I will complete the example I used in write.

<cfscript>
    infile = FileOpen("c:\temp\input.txt", "read");
    outfile = FileOpen("C:\temp\result.txt", "write");
    while (! FileIsEOF(infile)) { // continue the loop if the end of file has not reached
        x = FileReadLine(infile); // read a line
        data = processLine(x);
        FileWriteLine(outfile, data);
    }
    FileClose(infile);
    FileClose(outfile);
</cfscript>

These set of functions would greatly help if you need to work with a file of more than 10 kb size.

Apart from these set of functions, ColdFusion 8 also adds a new language struct to read text files. With ColdFusion 8, you can use <cfloop> to iterate over "lines" or "characters" in a text file. This makes it very easy and convenient to do any kind of text file parsing or processing in your application. Lets take a look at the new syntax of cfloop for reading file (and I really love this syntax :-)).

New attributes in cfloop for reading file :

"file" - path of the file to read
"characters" - no of characters to read in one iteration.

Reading Lines : Below is the simplest syntax to read one line at a time from the file in a loop. This would read the entire file and the loop would end when the file has been completely read. The read content will be available in the index variable specified.
```
<cfloop file="c:\temp\myfile.txt" index="line">
    <cfoutput>#line#</cfoutput> 
</cfloop>
```
With cfloop, you can also iterate over a part of the file by specifying "from" and "to" values.
Here is an example to loop over lines between 10 and 20.
```
<cfloop file="c:\temp\myfile.txt" index="line" from=10 to=20>
    <cfoutput>#line#</cfoutput> 
</cfloop>
```
"from" and "to" both are optional attributes where "from" defaults to '1' i.e start of file and "to" defaults to the last line of the file.

So to read first 10 lines from the file, you can use
```
<cfloop file="c:\temp\myfile.txt" index="line" to="10">
    <cfoutput>#line#</cfoutput> 
</cfloop>
```
One word of caution here - If you use "to" attribute here, its value must be less than the number of lines in the file otherwise you would get an "EndOfFile" error. For example, if I had only 5 lines in my file and value of to is 7, this would throw an error because line 6 and 7 do not exist.

Reading characters : For reading characters instead of line, you need to provide the value for "characters" attribute and as many characters will be read in one iteration. The loop will automatically end when the end of file has reached. The read content will be available in the index variable specified.

An example for that is
```
<cfloop file="c:\temp\myfile.txt" index="chars" characters="1000">
    <cfset x=chars>
    
</cfloop>
```
One important thing to note here. In the last iteration, when the end of file has reached, index variable will only have the remaining characters. For example if I have 130 characters in the file and I run the loop to read a chunk of 20 characters, in the last iteration, index variable's value will only have last 10 characters.

This completes the first part of new File IO which mainly addresses the problem of working with larger files. However this does not mean that you can not or should not use these for smaller files. You can very much use these for all kind of files. These are very simple to use and perform really well. Go ahead and play around with it !

CFUnited '07 experience

2007-07-05T13:37:00.000-07:00

This was my first CFUnited conference and I was amazed with the energy and passion of ColdFusion community. I have been to other non-CF conferences earlier and I can say that I had never seen such passionate and loyal developer community. The interaction that I had with folks there was really awesome and much valuable. The excitement and buzz around ColdFusion 8 was tremendous and all of us in the team are really excited with the response. I am sure ColdFusion 8 will be the best ColdFusion release so far.

All the sessions around CF8 including my session on "CFML Enhancements in ColdFusion 8" were full packed sessions and were very well received. Thanks a lot guys if you attended my session :-). Here are the slides for my session in case you missed it.

At CFunited 07

2007-06-26T03:41:00.000-07:00

I am posting this from Bethesda Marriott where I am attending CFUnited. I would love to meet you guys if you are here. You can catch me near our booth or at the end of my session where I will be speaking on "Language Enhancements in ColdFusion 8" on Thursday morning 8.30 - 9.30. Apart from tons of feature, we have done lots and lots of enhancements in Scorpio which should help you write better and faster application. I have selected some of the exciting enhancements for this session which I am sure you would love.
See you there !

New CFDocument hotfix released !

2007-06-18T09:44:00.000-07:00

We have released ColdFusion MX 7.02 - CFDocument Cumulative Hot Fix that fixes lot of CFDocument related issues.
Some of the important issues that we fixed in this hotfix are

1. Text getting chopped / clipped
2. Image zoom or image cropped
3. Header/Footer overwriting the text in the page body.

If you use cfdocument, it is a must have hotfix for you.

You can download the hotfix here but do not forget to read the instructions to apply this hotfix. Enjoy !

Finding Image Type for a file

2007-06-08T15:22:00.000-07:00

This is a follow up on my previous post on URLConnection APIs
We will use this API to find out the image type of a given file. Here is the udf.


<cffunction name="getMimeType">
    <cfargument name="filepath">

    <cfset var urlConn = createObject("java", "java.net.URLConnection")>
    <cfset var fileobj = fileopen(filepath, "readbinary")>
    <!--- just read the first 20 characters of the file as thats sufficient --->

    <cfset var bytes = fileread(fileobj, 20)>
    <cfset var istream = createObject("java", "java.io.ByteArrayInputStream").init(bytes)>
    <cfset fileobj.close()>
    <cfreturn urlConn.guessContentTypeFromStream(istream)>
</cffunction>

<cffunction name="GetImageType">
    <cfargument name="filepath">
    <cfset var mimetype = getMimeType(filepath)>
    <cfset var imagetype="">
    <cfif not isDefined("mimetype")>
        <cfthrow message="Not an Image file">
    </cfif>
    <cfswitch expression="#mimetype#">
        <cfcase Value="image/gif">
            <cfset imagetype="gif">
        </cfcase>
        <cfcase Value="image/x-bitmap">
            <cfset imagetype="bmp">
        </cfcase>
        <cfcase Value="image/png">
            <cfset imagetype="png">
        </cfcase>
        <cfcase Value="image/jpeg">
            <cfset imagetype="jpeg">
        </cfcase>
        <cfcase Value="image/jpg">
            <cfset imagetype="jpeg">
        </cfcase>
        <cfdefaultcase>
            <cfthrow message="Not an Image file">
        </cfdefaultcase>
    </cfswitch>
    <cfreturn imagetype>
</cffunction>

If I run the code below


<cfset filepath = "C:\temp\test.jpg">
<cfoutput>#GetImageType(filepath)#</cfoutput>

It nicely prints out "jpeg".

You should note that I used the new File IO function added in ColdFusion 8 using which I can read as many no of bytes from the file as I want. No more reading the entire file into memory.

More extensive post on File IO functions coming next !

Finding mime type of any file

2007-06-08T14:51:00.000-07:00

Hemant recently told me about this cool API in java.net.URLConnection which can tell you the mime type for common file types.

guessContentTypeFromName(String name) - that simply checks the extension of the file name specified and gets the mime type from a static map it maintains. This can be very useful in many scenarios. The map it uses is quite extensive and contains almost all the mime types.

But this has a problem. What if some one has renamed a gif to jpg? This method will say that the file is a jpg image whereas it was a gif.

Thankfully there is another method in the same class which can address this problem.

guessContentTypeFromStream(InputStream stream) - which reads the stream, takes a look at the initial bits of the file and uses that to determine the file type.
This method does a check for the following mime types :-


- application/java-vm
- application/x-java-serialized-object
- text/html
- application/xml
- image/gif
- image/x-bitmap
- image/x-pixmap
- image/png
- image/jpeg
- image/jpg
- image/vnd.fpx
- audio/basic
- audio/x-wav

Though this method leaves out many of other common types such as mp3, it is still very useful. This makes it so easy to find out the type of an image.

ListToArray with empty elements - Part II

2007-06-02T08:01:00.000-07:00

Yesterday I had posted about how to convert List to Array that would include empty elements. Charlie rightly pointed out a problem with that which is - it does not consider the empty elements at the end of the list.
So, a list "a,b,,c,d,," when converted to array will have only 5 elements instead of 7.

Incidently Charlie had also talked about the same some time back here and he mentions another problem with this approach. The array that is returned after split method is read only and you can not modify this array.
In order to address both these, I had to write this UDF which I guess many people would have written anyways..



<cfscript>
   function list2Array(list)
   {
       var endCommaCount = 0;
       var i = 0; var c = "";
       var stringForSplit = ""; var retList=""; var arr = "";

       for(i=len(list); i > 0; i--)
       {
           c = list.charAt(i-1);
           if(c == ',')
               endCommaCount++;
           if( c != ',' && c != ' ')
               break;
       }
       retlist = ArrayNew(1);

       if(i != 0)
       {
           stringForSplit = left(list, i);
           arr = stringForSplit.split("\s*,\s*");

           for(i = 1; i <= ArrayLen(arr); i++)
               ArrayAppend(retList, arr[i]);
       }

       for(i = 0; i < endCommaCount; i++)
           ArrayAppend(retList, "");

       return retList;
   }
</cfscript>

UPDATE : Thanks to Andrew Clark for the split tip which makes it more slim and elegant. The code above changes like


<cfscript>
   function list2Array(list)
   {
       var i = 0; 
       var retlist = ArrayNew(1);
       var arr = list.split(",", -1);
       for(i = 1; i <= ArrayLen(arr); i++)
           ArrayAppend(retList, arr[i]);
       return retList;
   }
</cfscript>

Lets test this udf and see if it works..


<cfset cflist = "a,b,,c,,d, ,">
<cfset cfarr1 = ListToArray(cflist)>
<cfset ArrayAppend(cfarr1, "END")>
<cfset cfarr2 = List2Array(cflist)>
<cfset ArrayAppend(cfarr2, "END")>

With CF Function : <cfdump var="#cfarr1#" format="text">
With UDF : <cfdump var="#cfarr2#" format="text">

And this produces this result.
With CF Function :

array - Top 6 of 6 rows

1) a
2) b
3) c
4) d
5) 
6) END

With UDF :

array - Top 9 of 9 rows

1) a
2) b
3) [empty string]
4) c
5) [empty string]
6) d
7) [empty string]
8) [empty string]
9) END

One thing to note here is that I am using new cfscript operators added in ColdFusion 8 which is so cool!!!

ListToArray with empty elements

2007-06-01T12:13:00.000-07:00

Recently there was a request on our forum to support empty elements in ListToArray function for Scorpio i.e ColdFusion 8. it might not be done in Scorpio since it is too late to add a new function. However it can be very easily achieved in cfm itself. Here is the code to do it.


<cfset cflist="a,b,,c,,,d">
<cfset cfarr1 = ListToArray(cflist)>

<cfset cfarr2 = cflist.split(",")>
<cfset cfarr2 = cflist.split(",", -1)>

<cfdump var="#cfarr1#">
<cfdump var="#cfarr2#">

<cfscript>
for(i=1; i <= ArrayLen(cfarr2); i++)
{
 writeoutput("Element #i# : #cfarr2[i]# <br>");
}
</cfscript>

As you can see, 3rd, 5th and 6th element in this cfarr2 are empty.

ColdFusion 8 beta has arrived

2007-06-01T11:26:00.000-07:00

I think I am the last person to say that ColdFusion 8 i.e Scorpio beta is out. Yayyy !!! :)

And that means I am back on blogsphere after a long hiatus. Seriosuly, what a fabulous journey it has been! It is a result of nearly one and half year of hard work, huge amount of research, so many design discussions, fights over nitties-gritties of features, many spec review meetings (which many a times reminded me of the college time ragging or soap box for elections at IIT kgp), many beer parties (because some one was beer boy coz he broke the build) and eating rubber chicken few times (becoz the build was broken again !!) etc etc etc.. -- all to ensure that there is enough sting in the Scorpio. :)

Everyone of us in the team is really proud of the stuff that we have built and I am sure people gonna love it.

And we are not done yet! Ashwin's cartoon says it all !!!

Use Java 5 feature in Jdk1.4

2007-03-02T11:50:00.000-08:00

I have been getting the itch for a long tome to use Java 5 features like generics, new concurrency (java.util.concurrent), autoboxing, "for each" loop and many more. But sadly I can not do it at work because ColdFusion has to be supported on all application servers. Since many of the application servers are still on J2EE 1.4 and not all of them are compatible with Java 5, we have to write JDK1.4 compliant code. Which means that all those wonderful features of Java5 can not be used for some more time to come. Or so I thought until yesterday when I read Brian Goetz's (of "Java concurrency in practice" fame) article which says that we can use some of the Java features in JDK1.4 also.
At first I was kind of puzzled as in how could that be possible? Did they had these features and they just did not mention about it in JDK1.4 or probably they just sneaked it in 1.4 in last moment but not talk about it because of some reasons.

It dawned to me when I went halfway through the article. It looks like most of these new syntaxes did not need any change in the bytecode instruction set which means that the compiler would generate the same bytecodes or instruction set for it as it would have for equivalent code without using these features. So it should be possible to use a class created by java5 compiler in 1.4. But don't we get UnsupportedClassVersionError when we try to do that?

Here is the catch. Due to some reasons they had to change the class version and that is why this UnsupportedClassVersionError. However these guys added a flag to java compiler (which ofcourse is undocumented) that can allow this. You need to invoke javac with "-source 1.5 -target jsr14" option and this will generate 1.4 compatible bytecodes for java5 syntax. This means I can happily write my code using generics, autoboxing, for..each loop, compile it with java5 compiler with this flag and be assured that this can run on 1.4 VM also.

That leaves one more set of features i.e; new concurreny, which I really really like in java5 and want to use. Thankfully some guys at Distibuted Computing Lab at Emory University have backported java5 concurrency (java.uti.concurrent) features that can be used in java 1.3+. I have infact used it and it is awesome !

So most of the juicy stuffs that were added in java5 can run on 1.4. Off to checking it out little more. :)

java.net article on ColdFusion and JavaEE

2007-03-01T07:12:00.000-08:00

I found this interesting article by Kola Oyedeji on java.net. He talks about integration between ColdFusion and JavaEE and how this could be of interest to JSP developers.
We in our team here have discussed it so many times that why can not ColdFusion be used by J2EE/JSP developers or enterprises that standardize on J2EE. ColdFusion is a pure J2EE solution, runs on all the major applications servers, is so feature rich, so productive that it should be a no brainer for everyone to adapt this. And to this when I add the features that we are adding in Scorpio, it makes even more sense. I don't think there is any other platform which can come even close to CF in terms of features and productivity.
At MAX, I had a conversation with one of our customers and he said something very interesting. His client wanted a pure J2EE solution and his competitors were asking for nearly 6 months to implement this. So this guy did everything in CF, bundled it as an ear (that can be deployed on application server) and shipped in flat one month !! And his client never ever thought that this was a CF application. :D

Good to see this kind of article on a java website.

Accessing password protected URL

2007-02-25T12:12:00.003-08:00

How do you access a password protected URL (requiring basic authentication) programmatically? well, there are two ways

If you know that the URL is protected : This is the simple one. Since you know the URL is protected, add the required authorization header in the request and you are done. What if you are not making the socket connection but instead using java.net.URL or java.net.URLConnection to retrieve the content of url? You can use setRequestProperty() method of URLConnection to add any request header you want. So the code would look like
```
URL url = new URL(someurlstring)
URLConnection conn = url.openConnection();
String encoding = new sun.misc.BASE64Encoder().encode("username:password".getBytes());
conn.setRequestProperty ("Authorization", "Basic " + encoding);
InputStream in = conn.getInputStream()
...
```

What if you dont know in advance whether the URL is protected or not? You can not arbitrarily add the authorization heder for all the URLs. The answer to this is "java.net.Authenticator". You need to install an instance of java.net.Authenticator using setDefault() method of Authenticator. Whenever URLConnection sees that the url is protected, it will use this installed authenticator to get the username and password and set the required authorization header automatically.
Here is how the code snippet for doing this


Authenticator.setDefault(new MyAuthenticator());
....

public class MyAuthenticator extends Authenticator
{
 protected PasswordAuthentication getPasswordAuthentication()
 {
     return new PasswordAuthentication ("username", "password");
 }
}

This particular technique is particularly useful for people who need a workaround for retrieving password protected images in cfdocument. Here is what you need to do.

Use the source below to create Authenicator class


import java.net.*;
public class MyAuthenticator extends Authenticator
{
    private String user;
    private String passwd;

    public MyAuthenticator(String user, String passwd)
    {
        this.user = user;
        this.passwd = passwd;
    }

    protected PasswordAuthentication getPasswordAuthentication()
    {
        return new PasswordAuthentication(user, passwd.toCharArray());
    }
}

Compile this and put MyAuthenticator.class in wwwroot/WEB-INF/classes.

Put the following code in your application.cfm or application.cfc


<cfif not IsDefined("Application.authenticator")>
      <!--- Do initializations --->
      <cfset authenticator= createObject("java", "java.net.Authenticator")>
      <cfset myauthenticator = createObject("java", "MyAuthenticator").init("username", "password")>
      <cfset authenticator.setDefault(myauthenticator)>
      <cfset Application.authenticator=myauthenticator>
</cfif>

replace "username" and "password" with actual username and password.

give the request for the cfdocument page for which you were getting the red-x for image. The images should appear this time.

A page in CFSwitch-CFCase' diary

2007-02-15T13:07:00.000-08:00

Undoubtedly I am one of the most powerful programming construct of ColdFusion language. And definitely more powerful than all my cousins in other programming languages. All my cousins and even my brother (read java switch-case) work only on integers but I work on almost every kind of objects be it string or any numeric data or even date. Since I can work on almost all datatypes, I make cf developers life so much simpler. I was created this way because the world I was going to be in was UnTyped - where every one was equal and where there was no discrimination between datatypes and I really thank God (ColdFusion Architects) for creating such a wonderful world.
Recently I heard someone talking about me that I am not as fast as my cousins. Some one even talked about ignoring me and taking help of kiddo cfif-cfelse. I dont want to say anything against anyone coz I know "with great power comes great responsibility err..cost".

Its sheer hardwork that makes me so much more powerful than all my cousins. This is what I do when any object comes to me. First I try to see if it is numeric. I do that because that is what all my cousins are used to and I have to remain as fast as them in that case. If it is not numeric, then I check if it can be date. If it is not even date, I try converting it in String. Once I arrive at the data, I use my own data structure to match it with appropriate CASE. So when the data is string, i will take some more time as compared to what i will take when data is numeric. Is that so bad? My cousins dont even do that!

Sometimes I compare myself to a busy lawyer who likes working on many CASEs. I dont like to work just for 2-3 cases. I prefer my grandson cfif-cfelse take care of those small no of cases.
I hope someday people will read these pages and if even after reading this they think that I am too slow and I should be ignored I only want to say "God, forgive them, for they dont know what they are doing!".

Performance Tips : ColdFusion List

2007-02-15T11:39:00.000-08:00

how many of you have written/seen code like this?


<cfset mylist="jan,feb,mar,apr,may,jun,jul,sep,oct,nov,dec">
<cfloop from="1" to=#ListLen(mylist)# index="i">
 <cfset month = ListGetAt(mylist, i)>
 <!--- do something with this month --->
 <cfoutput>#month#</cfoutput>
</cfloop>

While there is nothing wrong with it syntactically or functionally, performance wise it is very poor. Why? ColdFusion list is nothing but String (delimited by delimiter). ColdFusion does not have any way to build any intelligence to keep it in any other datastructure because you can use it like a normal string also. So what happens when you call any List function on this string? We parse the string using the delimiter and get the delimited tokens and process that.
Now lets take ListGetAt(list, index) function. It will keep parsing and getting the token unless it reaches the required index. Imagine doing in a loop. We will be parsing the same string again and again and traversing from the beginning everytime till we reach the next loop index. So, in the Nth iteration, it will start from beginning and tokenize N times. Thus by the time you have completed the loop, you have parsed/tokenized the string N*(N+1)/2 times. Isn't that too costly? Lesson - Never ever use ListGetAt() in a loop. Either iterate using OR convert the list into array using ListToArray() and iterate over it. Using cfloop is the most optimized way to do this.

Even if you are not iterating over list but you need to call ListGetAt() many times, it is better to convert it to array and then search the index in that.

Same thing applies to search functions like ListFind, ListContains etc. If you need to call these multiple times on the same list, you will be better off converting the list to array and searching in that.

If you need to append many items to the list, then also you will get a better performance by converting the list to array and doing all appends on that.

This does not mean you should not use list at all or you should always convert the list to array and work on that. If the number of operations that you are doing on the list is less, you should stick to list because converting the list to array is also costly. If you are inserting an element in the middle of list, list will be better than array in most cases.

ColdFusion Array : pass by reference or Value.

2007-02-06T07:04:00.000-08:00

Ben rightly pointed out in my last post that since ColdFusion array are always passed by value, the second technique can not be used if you want to build the array over multiple method calls. In each of the function call, ColdFusion will create a copy of the array passed and that cost (cost of creating a new instance and copying old array to new one) might even exceed the cost of appending string.

ColdFusion array is one unique data structure in ColdFusion. Unique in the sense that this is the only data structure that is passed by value and not by reference. I do feel it is a limitation sometimes but thats legacy and can not be changed. (You would not want us to break your code. Would you? ;) )

There does exist one hack if you absolutely need to pass the array by reference. Here is a code snippet that uses pass by reference.


<cfset x = ArrayNew(1)>
<cfloop from=1 to=5 index=i>
 <cfset Arrayappend(x,"something")>
</cfloop>

<cfset x = CreateObject("java", "java.util.ArrayList").init(x)>
<cfset foo(x)>
<cfset foo(x)>
<cfset x[8] = "after the method call">
<cfset x[9] = "end of method call">
<cfset foo(x)>
<cfdump var="#x#">

<cffunction name="foo">
 <cfargument name="arr">
 <cfset ArrayAppend(arr, "from function")>
</cffunction>

So what did we do here? We created an ArrayList from the ColdFusion array. Since ColdFusion Array is an implementation of java.util.List, almost all Array functions work on all implementations of java.util.List. And this list implementation will not be passed by value but will be passed by reference. Thats the power of using java in ColdFusion !

The Mystery of "Too many open files"

2007-02-05T10:26:00.000-08:00

Last week we saw an interesting problem while running the regression tests on Linux. Immediately after the tests were started, the VM started throwing error "FileNotFoundException : Too many open files". Ofcourse the files were there but the VM was trying to say that there are too many file descriptors open which were not closed. It was hitting the open file limit set by the OS. This was kind of absurd because we always close all the files that we open. And moreover we had never seen this problem before. So we started suspecting 1.6 VM on which we were running it.

Immediately after this, it got worse. The exception changed into "SocketException: Too many open files" and all the socket connection started getting rejected. So merely after serving 100 requests, the server was down to its knees.

A quick google search suggested to increase the open file descriptor limit on the machine.

"ulimit -aH" that gives the max limit for number of open files returned 1024.

I added every possible way to increase it. Here are few

1. In /etc/security/limits.conf
* soft nofile 1024
* hard nofile 65535
2. Increase ulimit by "ulimit -n 65535"
3. echo 65535 > /proc/sys/fs/file-max
4. In /etc/sysctl.conf
fs.file-max=65535

Increasing file descriptor limit did not help much either. Even after increasing this limit, we were still getting this error.And then Sanjeev (another brilliant CF engineer with an amazing sense of humour) cracked it !!
Just before we started getting these errors, there was another error which I had overlooked assuming it was test problem which infact was the clue.
The error was something like
coldfusion.jsp.CompilationFailedException: Errors reported by Java compiler: error: error reading /opt/coldfusionscorpio/lib/./././././././././././././././././././././././././././././
././././././././././././././././././././././././././././././././././././././././././.
/././././././././././././././././././././././././././././././././././././././././././
././././././././././././././././././././././././././././././././././././././././././.
/././././././././././././././././././././././././././././././././././././././././././
././././././././././././././././././././././././././././././././././././././././././.
/././././././././././././././././././././././././././././././././././././././././././
././././././././././././././././././././././././././././././././././././././././././.
/././././././././././././././././././././././././././././././././././././././././././
././././././././././././././././././././././././././././././././././././././././././.
/././././././././././././././././././././././././././././././././././././././././././
././././././././././././././././././././././././././././././././././....
at coldfusion.jsp.JavaCompiler.compileClass(JavaCompiler.java:138)

Sanjeev ran it through the debugger and he nailed the culprit. It was something which no one had even suspected. It was javac. :)
We had hit upon a Sun's bug in javac where if the classpath contains a jar which has a manifest and manifest contains classpath entry with relative paths of other jars as well as path to itself, javac goes in an infinite loop. (Sun's bug no 6400872, 6446657, 6456960, 6485027, 6206485)

I dont have the source for javac to pin point what exactly it did, but probably it kept opening all the entries in the classpath and because of the stack overflow it could not close those files - hence reaching the max limit of open file descriptors. This was a third party jar that had this manifest entry. Once we found the problem, the solution was simple - just remove the classpath entry from the manifest! Whew !!!

String Concatenation optimization

2007-01-30T11:01:00.000-08:00

String concatenation is one of the most common, but, a pretty expensive operation. It can hit the performance severly if not used correctly. The performance goes down drastically if you append strings using '&' OR ListAppend() in a loop. I have seen application performance improving by 50-100% just by optimizing String concatenation (though that depends on how much concatenation is used in the app). So what do you about it?
The simplest and the most optimized way to do these append operations is using java's StringBuffer. (I am sure you must be aware of it but still.. :)) .
The code would look like


<cfset sb = createObject("java", "java.lang.StringBuffer")>
<cfloop from=1 to=100 index=i>
  <cfset sb.append("something")>
  <cfset sb.append(i)>
</cfloop>
<cfset result=sb.toString()>

Sometimes I feel that we should have a datastructure like this in ColdFusion directly but again I think whats wrong with using StringBuffer? Its like any other function which we would create. Isn't it so?

If you are a puristic and don't want to use any java API inside your CF app, there is another simple way to do the same thing. It uses ColdFusion Array to do the same thing what StringBuffer does. Instead of appending the string in the buffer, you can append to the array using ArrayAppend() and then once you are done and want to get the string back, use ArrayToList() with empty string ("") as delimiter. The code would look like


<cfset arr = ArrayNew(1)>
<cfloop from=1 to=100 index=i>
  <cfset ArrayAppend(arr, "something")>
  <cfset ArrayAppend(arr, i)>
</cfloop>
<cfset result=ArrayToList(arr,"")>

This would give a much better performance as compared to concatenation using '&' or using ListAppend() but will have lower performance as compared to StringBuffer. That is because of the overhead of Array object creation and array append operation. ArrayToList() will anyway create the string buffer and append the strings

You should use '&' or ListAppend() only when there are only 2-3 strings to be concatenated. Otherwise always use either of the two techniques above.

Optimizations with literals

2007-01-17T11:50:00.000-08:00

Look at these two pieces of code carefully. Is there any difference between these two apart from the fact that the second one is shorter?


<cfset x = "sun,mercury,venus,earth,mars,jupiter,saturn,uranus,pluto,neptune">
<cfset y = ListSort(x,"text")>

and


<cfset y = ListSort("sun,mercury,venus,earth,mars,jupiter,saturn,uranus,pluto,neptune","text")>

If you think there is not much, read on.

There is a huge difference between these two piece of code - in terms of performance. The second one will have much better performance as compared to the first one. How?? Because the ListSort() method in the second case will not even be executed in the page request. Still scratching your head?

It is because of the intelligence that is built into CFML compiler (really superb code written by Edwin Smith). During compilation, it analyzes all the code and wherever there is a literal or functions executing literals, it tries to optimize it. In the second piece here, it sees that ListSort method is being called on a literal and it can be done statically. So compiler will execute this call during compilation itself and set the sorted value on 'y'. During the page execution, only thing that will get executed will be an assignment. Smart.. isn't it? Even java compiler, which does quite a many compile time optimization for java source files, does not have this intelligence of executing calls at compile time :)

It is not only about 'ListSort'. This is true for most of those CF functions which can work on a literal and can return a literal.

Extend CF native Objects - Harnessing Java

2007-01-12T07:02:00.000-08:00

Since Coldfusion native objects are java objects, you can harness the java APIs to extend the functionality of these objects. In this post we will take CF Array and see how we can use these APIs to get some cool functionalities from them.

ColdFusion array is actually an implementation of java list (java.util.List). So all the list methods are actually available for Array.
CF provides most of the list functionality using Array functions but there are few things possible with java list which you can not do directly with CF functions.

1. Merge : Lets say you create two arrays and you want to merge these two arrays to create one bigger array. There is no CF function to do this.
However You can call List.addAll() methods to do it.

Here is how it would look.

I am creating array this way just because it is easier and I don't have to write whole lot of code :)


<cfset y = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
<cfset z = ListToArray("dean,manju,jason,tim")>
<cfset y.addAll(z)>
<cfdump var="#y#">

2. Merge in middle : Lets say you want to add the second array somewhere in the middle of first array say after 4 elements. The code would look like


<cfset y = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
<cfset z = ListToArray("dean,manju,jason,tim")>
<cfset y.addAll(4, z)>
<cfdump var="#y#">

3. Search : I have heard people complaining that there is no find method in Array. Actually you had it all the time. Just that it was hidden :)
You can use List.Contains() or List.indexOf() methods to achieve that. Here is the code.


<cfset y = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
<cfoutput>Contains Hemant: #y.contains("hemant")#</cfoutput>
<cfoutput>Index of damon : #y.indexof("damon")#</cfoutput>

Please note that the java index starts at 0 where as CF index starts at 1. So the index here will be 2. You must also note that since java is case sensitive, this search will also be case sensitive. To build case insensitiveness, you will have to make the list as well as the search in same case - either uppercase it or lowercase it.

4. Search whole array : You can also search if all the elements of one array are present in another array using containsAll() method of java list.


<cfset y = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
<cfset z = ListToArray("ram,prank,rupesh")>

<cfoutput> y Contains z: #y.containsAll(z)#</cfoutput>

5. Equality check : You can find out if two arrays are same or not using list's equals method.


<cfset y = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
<cfset z = ListToArray("dean, manju,jason,tim")>
<cfset x = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
<cfoutput>x equals y : #x.equals(y)#</cfoutput>
<cfoutput>x equals z : #x.equals(z)#</cfoutput>

6. RemoveAll : You can remove all the elements of one array from the second array using removeAll()


<cfset y = ListToArray("rupesh,tom,damon,hemant,ashwin,ram,prank,sanjeev")>
before removal <cfdump var="#y#">

<cfset z = ListToArray("ram,prank,tom")>
<cfset y.removeAll(z)>
After removal <cfdump var="#y#">

Go ahead and play around with it!

Scorpion Queen !

2006-09-21T10:35:00.000-07:00

When Tim asked for spiffy Scorpio logo, a few CF engineers (Sanjeev, Chandan, Jayesh, Sandeep and Vamsee. special mention - Hemant & Praveen.) got together to create her. Of course they didnt use Photoshop. We are very much used to Whiteboard :)

Did you notice the cool scorpio tatoo she is wearing. And No, she would not execute <CFLAPDANCE /> !!!

Connecting to URL from behind a proxy server

2006-09-20T12:11:00.000-07:00

I needed a way to set the proxy information on URL/URLConnection and I could not find any good way. One simple way that java recommends is to set the information as system property. These properties are "http.proxyHost" and "http.proxyPort".

So it can either be set as jvm arguments like

-DproxySet=true -Dhttp.proxyHost=proxyIP -DproxyPort=port

or set them in the code using System.setProperty()

However since this is a system property, it gets set on the VM itself and hence it is not dynamic. In ColdFusion, since tags like 'cfhttp' keep them dynamic, I wanted a similar behaviour. After looking around for a while, I noticed that this capability was added in Java 1.5 aka Tiger release. (Is it only me? I keep hitting things which I feel is lacking in java API and then I find them added in 1.5 :) )

In Java 1.5, you can use

URLConnection conn = url.openConnection(proxy); // added in 1.5

where proxy is an object of java.net.Proxy. Pretty neat.

However this was not a solution for me as we are still developing on JDK1.4 (need to consider all the application servers that we have to support). I stumbled upon an interesting article by Daniel Horn who faced the exact same problem. And guess what he did? Since you send an HTTP request to the proxy and then proxy sends out the actual request, he created the URL object by passing proxyHost and proxyPort as IP and port and then he gave the target url string as 'file' argument. This is what he did.

String actualUrl = "http://www.adobe.com";
URL url = new URL("http", proxyHost, proxyPort, actualUrl);
URLConnection conn = url.openConnection();
..
..

And it works ! Brilliant !! I wonder why this is not documented in the java API.

A workaround for cfdocument missing images

2006-09-19T11:57:00.000-07:00

This is with reference to the post Missing images in CFDocument. There are some cases when the images are locally on the machine running ColdFusion but even then cfdocument is not able to show the images. The reasons could be
1) ColdFusion is behind a firewall because of which it is not able to send any HTTP request (even though to itself).
2) The images are under a protected directory which needs authentication. Since cfdocument can not send authentication information currently, it is not able to fetch the image.
3) ColdFusion is using HTTPS and it is not configured properly to trust itself. So cfdocument can not send a https request to itself.
4) Any other reason which is preventing CFDocument from sending request to the local server.

If the images are on local machine, it is possible to use the file url for images (or CSS,javascripts, etc). CFDocument in that case will not send requests for the images over HTTP and fetch the image directly from the file system. Here is a simple way to use the file url.


<cfdocument format="pdf">
 <cfoutput>
   Some html content
   <br>
   <img src=#localUrl("img1.gif")#><br>
   <img src=#localUrl("images/img.jpg")#>
 </cfoutput>
</cfdocument>

<cffunction name="localUrl" >
  <cfargument name="file" />
  <cfset var fpath = ExpandPath(file)>
  <cfset var f="">
  <cfset f = createObject("java", "java.io.File")>
  <cfset f.init(fpath)>
  <cfreturn f.toUrl().toString()>
</cffunction>

basically here I have an UDF which converts any path to local URL and then I am using that UDF in 'src' attribute of image. This can be used to fetch images, css or any other similar contents from the local machine. You should note the <cfoutput> right under cfdocumet tag that allows the evaluation of UDF before it goes to cfdocument body.. This workaround is applicable only when the these contents are present on the same machine as ColdFusion.

This workaround has another advantage too. Normally when CFDocument body has any images, it fetches those images by sending HTTP request to the local server which is served by web threads. This has its own overhead. In a way, CFDocument uses server resource for getting something which is available locally on the server. This resource can instead be used to serve actual client http requests. Converting the image path to local urls will not go through HTTP and thus should have a better performance.

Update to CFThread POC tags

2006-09-15T09:12:00.000-07:00

Damon posted an update to the CFThread proof of concept tag that was published some time back. I wanted to do it for a long time but was busy in other Scorpio features and had to keep delaying this. Neverthless better late than never :) This update includes "thread safety" while retaining the old syntax in the original post. Thanks to Dan Switzer, Derek, Mike and all others who provided the valuable feedback on it !

Whats there in the update

Thread very much acts like a function and some time little more than that.
Any attribute can be passed to cfthread. These attributes can be accessed using 'attributes.<varname>'. These attributes are passed by value and hence they are completely thread safe.
Any variable unless defined with a scope prefix goes in thread local scope. So this is slightly different from function. Like function, variable defined in var scope will also go in thread local scope. So in following snippet, x, y, z and a all will be in thread local scope. However since b is directly used inside thread without defining it, it will use it from the page scope.

<cfset x = 10>
<cfset b = 20>
<cfthread name="t1">
 <cfset var y = 10>
 <cfset x = 20>
 <cfset z = y*x>
 ...
 <cfset a = z*Variables.x>
 <cfset a = a/b>
</cfthread>

All other scope variables will be accessible using the appropriate prefix like "Variables", "request", "Server" etc.

Threads have an another scope called thread scope in which only the owner thread can write but all other can read. This scope can be accessed using 'thread' prefix by owner thread or using the thread name by other thread or main page thread. This scope will be useful when the owner thread wants to put some data in it which needs to accessed by other thread. The same thing could have been achieved by all the threads writing to the page scope in its own variable but that needs some discipline from developers and is error prone. Having a separate thread scope which other can read makes it threadsafe and easier for developers.

If thread name is dynamic, it will be everyone's question how to access that thread data from another thread or main page. It can be easily done using Variables[threadname].xxx. See the example below.

Threads can continue even after the main page is done.

Below is a sample cf code that uses cfthread. (modified version of Dan's example. Thanks Dan !!).

<cfset CRLF = CHR(13) & CHR(10)>
<cfset sDirectory = expandPath(".") & "\tmp" />

<cfsavecontent variable="sContent">XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX </cfsavecontent>
<cfloop index="loopcounter" from="1" to="50">
<cfset threadname = "thread_" & #loopcounter#>
<cfthread name="#threadname#" filename="file#loopcounter#.txt" counter="#loopcounter#">
<cfset sOutput = "Content written by thread " & #attributes.counter#>
<cfloop index="i" from="1" to="100">
   <cfset sOutput = sOutput & sContent>
</cfloop>
<cfset sOutput>
<cfset dest="#sDirectory#\tmp_#attributes.filename#">
<cffile action="write" file="#dest#" output="#sOutput#" />
<cfset thread.msg="Message from thread "&#attributes.counter#>
</cfthread>

</cfloop>
<cfloop index="loopcounter" from="1" to="50">
<cfset threadname = "thread_" & #loopcounter#>
<cfjoin thread="#threadname#"/>
<cfoutput><br>#threadname# output : #Variables[threadname].msg#</cfoutput>

</cfloop>
<br> Work Complete!<br>

Feel free to play around with this tag and send feedbacks if you have any. Once again, as Damon said, this is not a CF feature and hence it is unsupported. On whether it will make it to the Scorpio or not, I can not guarantee anything but stay tuned ;)
Enjoy !

Handling J2EE session with cookies disabled

2006-09-12T11:59:00.001-07:00

Someone recently reported that when cookies are disabled and J2EE session is enabled, his sessions are not maintained in case of POST request. As per that, CF or rather the app server always creates a new session everytime. His code looked like


 <form method="post" action="test.cfm?#session.urltoken#">
    ...
   <input type="submit" value="Submit" >
 </form>

can you see whats wrong with above code?

As per the Servlet spec of J2EE, when cookies are disabled, session is maintained by url rewriting and that is done by appending ';jsessionid=' to the URI. Note the semicolon ';' before 'jsessionid'.

In the above code, it is appending session.urltoken which looks like 'CFID=1600&CFTOKEN=59663989&jsessionid=2830a9edcf6f794ff481'. Therefore the url becomes "test.cfm?CFID=1600&CFTOKEN=59663989&jsessionid=2830a9edcf6f794ff481" whereas it should been like "test.cfm;jsessionid=2830a9edcf6f794ff481?CFID=1600&CFTOKEN=59663989". Since jsessionId is not correctly specified, server does not get this and hence creates a new session.

So how do you handle it? One way is to get the sessionId and urltoken from the session and create the url as expected (which is some effort on developer part). Alternatively, you can use a rather simple approach of using URLSessionFormat(url) which will do the exact thing which is required here. URLSessionFormat() appends the necessary information if cookies are disabled. If they are enabled, it does not do anything. Therefore it might be a better idea to always use this function for any GET or POST url.

The above code should actually have been


 <cfset myurl=URLSessionFormat("test.cfm")>
   <form method="post" action="#myurl#">
    ...
   <input type="submit" value="Submit" >
 </form>

JRun Threadpool settings

2006-09-12T07:07:00.000-07:00

The other day I was looking at JRun's Threadpool implementation and it really took me SOME time to understand that piece of code. It is one of those codes which are not meant to be understood by others :D. It got me really confused about 'Active Handler Threads', 'min Handler Threads' and 'max Handler threads'. It was very much different from what I had assumed. Too much of 'creating runnables', 'swapping runnables', 'destroying runnables'.. phew.. I think this is the exact reason why Doug Lea and team had to introduce a standard implementation of ThreadPool in 'Tiger' release. Its so simple, neat and elegant I wonder why wasn't it introduced earlier in JDK.

Anyways, enough of cribbing. After my enlightenment of JRun's or CFMX's threadpool, I thought it would be nice to share it with you all. So what are these thread counts? (I am sure most of you would have it figured out. Neverthless.. :) )

Min handler threads - It is the number of web threads that will be spawned initially and will be waiting for HTTP requests. Which effectively means that it is the no of threads which will be waiting on serversocket.accept(). Thus it controls the no of requests that will be accepted concurrently. It is ideally the minimum concurrent users that you expect on the server. As soon as a thread gets a client request (i.e comes out of serversocket.accept()), it enters into a throttle before processing the request. This is where active handler threads come into the picture. Before the thread starts processing the request, it spawns another thread, if required, which can listen to the incoming requests.

Active handler threads - This decides how many requests would be concurrently processed. The throttle we talked about above, allows a maximum of "active handler" threads to continue and rest of threads wait until another thread exits the throttle. This along with "min handler thread" controls the throughput of the server. The value of active handler threads must be between min Handler count and maxHandler Count.

Max handler threads - This is maximum number of threads that can be created in the pool. This includes the threads queued in the throttle + threads processing the requests + threads waiting on the server socket. Once the server reaches the max Handler thread count, server will start denying the request throwing "Server Busy Error". You can see the "Server Busy" error even without server reaching the "max handler" limit if the thread in the throttle queue timeout. So if you see this error, dont start increasing the max handler thread count. You might need to tune all the three counts.

Having both MinHandler and activeHandler counts help JRun in addressing any sudden spike in the load. Lets say your minHandler count is 20 and activeHandler count is 40 and suddenly you have 40 concurrent requests, all of them will be served without any queuing and delay. When the load eases down on the server, it will let the extra threads die and bring the thread count down to minhandler count i.e 20.

The next question that naturally comes to the mind is what should be the appropriate values of these for my server? Having too less value for it would mean that you are not utilizing the potential of the server well and requests start queuing up even though server can handle it. Having a too high value for these would mean too many context switches and the server performance will deteriorate. (Too many context switches means CPU is busy scheduling the threads rather than executing them and thus hurts the performance) So what should be the appropriate value? well.. there can not be one answer or a formula to compute these values. It depends on your application, the traffic that your application expects, memory and processors of the machine on which you are going to run it etc etc.

By default the values in "ColdFusion standalone" are

- Min Handler Thread - 1
- Active Handler Thread - 8
- max Handler thread - 1000

min and active counts here are fine for a development machine but definitely not for a production machine. And in my opinion the value of 'max handler' is bit high even for a production machine. Creating a large no of threads does not necessarily increases the throughput of your server. It can actually lower it down because of the high no of context switches VM will have to make. Moreover, it might not be possible to create 1000's of threads because of OS limitations. On many of the OS, you will get an OutOfMemory Error because the VM will not be able to spawn so many native threads for you. I think max handler count in the range of 300-400 should be good enough.

Regarding tuning these counts, there are huge no of articles around which will tell you how to go about it. Since notion of these counts exist on all the application/web servers, articles need not be CF or JRun specific. In brief, you would need to run some kind of load tests with different values of minHandler and active handler counts, note the throughput and plot a graph. This graph should help you arrive at the appropriate value for these settings.

Weird Error while JRun/CF Startup

2006-07-20T12:36:00.000-07:00

Today suddenly one of the test machine here started to give one weird error. It had Coldfusion standalone and the CF server would not start up. I checked up all the logs and there was nothing in it. In fact JRun process itself didnt start up.
When I tried to run it from the console using "jrun -start coldfusion", I saw the following error laughing at me.


Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: java.security.AccessControlException: access denied (java.util.PropertyPermission jrun.home read)
 at java.security.AccessControlContext.checkPermission(Unknown Source)
 at java.security.AccessController.checkPermission(Unknown Source)
 at java.lang.SecurityManager.checkPermission(Unknown Source)
 at java.lang.SecurityManager.checkPropertyAccess(Unknown Source)
 at java.lang.System.getProperty(Unknown Source)
 at jrunx.kernel.JRun.(JRun.java:52)

It was spooky as I could see that the policy file has all permission and the JVM was complaining that the JRun process does not have the permission to read jrun.home property.
I was going bonkers before I realized that policy file was not getting picked up in the first place.
Finally it turned out that some test setup had messed around with the jvm.config file and misplaced some quotes. java.args in this file looked like


java.args=-server -XX:MaxPermSize=128m -Xmx512m -Dsun.io.useCanonCaches=false
-Dcoldfusion.classPath={application.home}/../lib/updates,{application.home}/../lib/,{application.home}/../gateway/lib/,{application.home}/../wwwroot/WEB-INF/cfform/jars
-Dcoldfusion.libPath={application.home}/../lib
-Dcoldfusion.rootDir={application.home}/../ -Djava.security.manager
-Djava.security.policy="C:/CF/lib/coldfusion.policy"
-Djava.security.auth.policy="C:/CF/lib/neo_jaas.policy"

where it should have been like


java.args=-server -XX:MaxPermSize=128m -Xmx512m -Dsun.io.useCanonCaches=false
-Dcoldfusion.classPath={application.home}/../lib/updates,{application.home}/../lib/,{application.home}/../gateway/lib/,{application.home}/../wwwroot/WEB-INF/cfform/jars
-Dcoldfusion.libPath={application.home}/../lib
-Dcoldfusion.rootDir={application.home}/../ -Djava.security.manager
"-Djava.security.policy=C:/CF/lib/coldfusion.policy"
"-Djava.security.auth.policy=C:/CF/lib/neo_jaas.policy"

So in case you are touching jvm.config file (Ideally you should not), take extra care about the quotes.

Hotfix for CFDocument thread hang in CF7.0.1 standard.

2006-05-04T08:40:00.000-07:00

There was a problem in CF7.0.1 standard edition where if you hit any error in a page using CFDocument, any other page using cfdocument will hang. This was a threading issue we had for standard edition and was logged as bug 61378.

This was fixed in the hotfix 1 and is available at
http://www.adobe.com/support/coldfusion/ts/documents/aae43964/chf7010001.jar

However I would recommend you to use hotfix 2 at
http://www.adobe.com/support/coldfusion/ts/documents/aae43964/chf7010002.jar
See the details of this hotfix at http://www.adobe.com/cfusion/knowledgebase/index.cfm?id=aae43964

Scale to fit for CFDocument

2006-05-04T07:04:00.000-07:00

I have seen couple of posts on the forum in which people wanted to know a way to fit the document in a page. I am wondering about the use case if this is really required. Could you please let me know the use cases and how are you working around it currently. You need to build a case to get this attribute in the tag :)

Host name verification in HTTPS

2005-11-24T04:18:00.000-08:00

As I mentioned in my blog on Missing images in CFDocument, in case you are using HTTPS, you must ensure that the certicate host name matches the host name in the URL. Lets say your certificate is issued to "www.mysite.com" then the request URL must have the host as "www.mysite.com". It can not be accessed using 'localhost', '127.0.0.1', that machine's IP address or machine's name.
Till JDK1.3, Sun's SSL implementation never used to verify the host name of the certificate. Since JDK1.4, it now verifies the hostname to prevent URL spoofing (When I request for some URL, some other guy in between intercepts and sends his own certificate and I will remain under the impression that I was getting the certificate of the requested server and hence a threat).

In case you want to access the URL using localhost or IP address or using machine's name, there is a workaround possible but that would invlove wrting some java code.
HttpsURLConnection that is used to make the connection, uses an interface HostnameVerifier to verify the host name of the certificate. A default implementation is used by default. You can provide your own implementation of this interface and set it on HttpsURLConnection. That will give the control of host name verification in your hand and you can verifiy it the way you want.

Unfortunately, this interface and HttpsURLConnection are present in both javax.net.ssl and com.sun.net.ssl package. So depending on which SSL packages are being used, you will have to implement appropriate interface and you will have to set this on appropriate HttpsURLConnection. To be sure, let your implementation class implement both the interface and set it on both the HttpsURLConnection by calling the static method

setDefaultHostnameVerifier(HostnameVerifier)

A simplistic implementation which disables any host name verification could be like


    class MyHostnameVerifier implements com.sun.net.ssl.HostnameVerifier, javax.net.ssl.HostnameVerifier{
        public boolean verify(String urlHostName, String certHostName){
            return true;
        }

        public boolean verify(String urlHost, SSLSession sslSession){
            return true;
        }
    }

set this verifier to both HttpsURLConnection at appropriate place.


    MyHostnameVerifier verifier = new MyHostnameVerifier();
    javax.net.ssl.HttpsURLConnection.setDefaultHostnameVerifier(verifier);
    com.sun.net.ssl.HttpsURLConnection.setDefaultHostnameVerifier(verifier);

Thread dumps...

2005-11-24T00:55:00.000-08:00

Many a times our customers complain of Coldfusion Server(CF) hanging, becoming non-responsive, taking 100% cpu or some request taking too much time. In order to understand the reasons why this happens, only way is to analyze the "Thread dumps". Not many of our CF users are java geeks and therefore when we ask them to send a thread dump, they have a very little clue about it. I hope they wouldn't be clueless anymore ;)

What exactly is "Thread dump"
"Thread dump" basically gives you information on what each of the thread in the VM is doing at any given point of time. This makes "Thread dump" an excellent debugging tool. It can tell you the states of each of the thread in the VM, where exactly each thread is in the execution path at that point, which thread is waiting and where is it waiting and lot more. It also shows you the stack of each thread and can help you track the execution path.

How do I take a Thread dump?
For any Java VM, if you are running it from a command prompt, you can get the thread dump by pressing Ctrl+Break (for windows) or Ctrl+\ (for unix machines) on the terminal running the server.
In case you are running the VM in background or as services, you can send a "SIGQUIT" signal to the process to get the thread dump. To send this signal to VM on unix machine, you can use "kill -SIGQUIT [pid]" where 'pid' is the process id of the server process. This will send the signal to the VM to dump the thread stack on the error stream (On Sun JVM) or to a new file (IBM's JVM).

You can also use the Stacktrace tool to get the thread dump. It is a nice java webstart application that you can run on the machine running the server. All you need is to specify the "process id" and it will nicely show the threaddump in its own window.

To be more specific for CF, if the standalone CF server or CF on J2EE server has a terminal, use Ctrl+Break or Ctrl+\ depending on the platform. If its running as background process or as services, use Stacktrace tool I mentioned earlier.

Here is how the thread dump would look.


Full thread dump Java HotSpot(TM) Server VM (1.5.0_04-b05 mixed mode):

"web-4" prio=5 tid=0x28999418 nid=0x1460 runnable [0x2fbaf000..0x2fbafd18]
       at java.io.WinNTFileSystem.canonicalize0(Native Method)
       at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:374)
       at java.io.File.getCanonicalPath(File.java:531)
       at java.io.FilePermission$1.run(FilePermission.java:218)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.io.FilePermission.init(FilePermission.java:212)
       at java.io.FilePermission.(FilePermission.java:264)
       at java.lang.SecurityManager.checkRead(SecurityManager.java:871)
       at java.io.File.exists(File.java:700)
       at jrunx.resource.FileResource.exists(FileResource.java:98)
       at jrunx.resource.FileResource.getURL(FileResource.java:140)
       at jrun.servlet.JRunServletContext.getResource(JRunServletContext.java:192)
       at jrun.servlet.file.FileServlet.service(FileServlet.java:148)
       at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:91)
       at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
       at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:257)
       at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:541)
       at jrun.servlet.http.WebService.invokeRunnable(WebService.java:172)
       at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:318)
       at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:426)
       at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:264)
       at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

"web-3" prio=5 tid=0x2814e470 nid=0xd70 in Object.wait() [0x2faaf000..0x2faafd98]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x064b5a90> (a jrunx.scheduler.JSemaphore)
       at jrunx.scheduler.JSemaphore.acquire(JSemaphore.java:74)
       - locked <0x064b5a90> (a jrunx.scheduler.JSemaphore)
       at jrun.servlet.network.NetworkService.accept(NetworkService.java:348)
       at jrun.servlet.http.WebService.createRunnable(WebService.java:104)
       at jrunx.scheduler.ThreadPool$DownstreamMetrics.createRunnable(ThreadPool.java:285)
       at jrunx.scheduler.ThreadPool$ThreadThrottle.createRunnable(ThreadPool.java:347)
       at jrunx.scheduler.ThreadPool$UpstreamMetrics.createRunnable(ThreadPool.java:239)
       at jrunx.scheduler.WorkerThread.run(WorkerThread.java:62)

"web-2" prio=5 tid=0x271954c0 nid=0x170c in Object.wait() [0x2ef6e000..0x2ef6fa18]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x02af8eb8> (a java.awt.image.PixelGrabber)
       at java.awt.image.PixelGrabber.grabPixels(PixelGrabber.java:254)
       - locked <0x02af8eb8> (a java.awt.image.PixelGrabber)
       at java.awt.image.PixelGrabber.grabPixels(PixelGrabber.java:209)
       at com.lowagie.text.Image.getInstance(Unknown Source)
       at com.lowagie.text.Image.getInstance(Unknown Source)
       at com.lowagie.text.pdf.PdfGraphics2D.addAltText(Unknown Source)
       at ice.pilots.html4.MacromediaCSSExtension.addonImageProcessing(MacromediaCSSExtension.java:186)
       at ice.pilots.html4.ObjectBox.paintChunk(OEAB)
       at ice.pilots.html4.InlineBox.paintChildren(OEAB)
       at ice.pilots.html4.InlineBox.paintChunk(OEAB)
       at ice.pilots.html4.InlineBox.paintChildren(OEAB)
       at ice.pilots.html4.BlockBox.paint(OEAB)
       at ice.pilots.html4.OutlinePainter.drawBox(OEAB)
       at ice.pilots.html4.BlockBox.paint(OEAB)
       at ice.pilots.html4.OutlinePainter.drawBox(OEAB)
       at ice.pilots.html4.BlockBox.paint(OEAB)
       at ice.pilots.html4.CSSLayout.paint(OEAB)
       - locked <0x02cebb10> (a java.lang.Object)
       at ice.pilots.html4.ThePrinter.printPage(Unknown Source)
       at coldfusion.document.DocumentSection.process(DocumentSection.java:230)
       at coldfusion.document.DocumentSection.print(DocumentSection.java:108)
       at coldfusion.document.DocumentExporter.export(DocumentExporter.java:237)
       at coldfusion.document.DocumentFrame.exportContent(DocumentFrame.java:118)
       at coldfusion.document.DocumentProcessor.processContent(DocumentProcessor.java:130)
       at coldfusion.document.DocumentProcessor.ProcessContent(DocumentProcessor.java:59)
       at coldfusion.tagext.lang.DocumentTag.processContent(DocumentTag.java:1218)
       at coldfusion.tagext.lang.DocumentTag.access$100(DocumentTag.java:84)
       at coldfusion.tagext.lang.DocumentTag$3.run(DocumentTag.java:1179)
       at java.security.AccessController.doPrivileged(Native Method)
       at coldfusion.tagext.lang.DocumentTag.doAfterBody(DocumentTag.java:1174)
       at cfcmyk2ecfm937582361.runPage(E:\CFusionMX7\wwwroot\testfolder\doctest\cmyk.cfm:3)
       at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:152)
       at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:349)
       at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
       at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:209)
       at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:51)
       at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
       at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:69)
       at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
       at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
       at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
       at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
       at coldfusion.filter.RequestThrottleFilter.invoke(RequestThrottleFilter.java:115)
       at coldfusion.CfmServlet.service(CfmServlet.java:107)
       at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:78)
       at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:91)
       at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
       at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:257)
       at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:541)
       at jrun.servlet.http.WebService.invokeRunnable(WebService.java:172)
       at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:318)
       at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:426)
       at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:264)
       at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
.
.
.

If you anayze this thread dump, you can see that "web-2" thread is handling request for "cmyk.cfm" and is waiting for the images to be loaded i.e (actually at Image.getInstance()). At this point, cfdocument has sent another http request for the image file that it has to render and that request is handled by another thread "web-4" which is running and is currently checking if the image file requested exists on the file system. Isn't it very powerful ?

You can checkout a very nice article Thread dump and stack traces written by Rajiv (my ex-colleague and mentor in Pramati) which I feel is a must read for Java developers.

Missing images in CFDocument

2005-11-17T00:31:00.000-08:00

I have seen huge no of postings on CF forum where CFDocument users have complained about seeing red-cross (or red-x ) for images . In this post, I would like to list the reasons why they happen and how they could be resolved .

Before we proceed with different cases of red-x, lets see in brief, how images are rendered in CFDocument.

During the processing of cfdocument tag, CF engine interprets/executes the content inside the cfdocument tag, creates html content out of it and renders it in the memory. While rendering this html content, if any image tag is found, a separate HTTP request is made to retrieve this image content. A separate HTTP request is necessary because image in the generated html can be local as well as remote. For Java geeks out there, we use URLConnection.getContent() to retrieve the image data. A red-x means that CFDocument was not able to retrieve the image.

Now lets see different scenarios one by one

Image name has space in it. In CFMX 7.0, you can get red-x for images if the image file name has any space in it. For example if the image file name is "my picture.jpg", only a red-x will appear in the pdf/flashpaper. It happens because the url created for the image is not encoded. A workaround for this is to either use encoded url for the images i.e replace 'space' in the name with '%20' OR dont have spaces in the file name at all :). This bug has been fixed in Merrimack (Coldfusion 7.0.1). So if you are still on 7.0, upgrade :)
If your server is behind firewall. As we mentioned earlier, CF server needs to send an HTTP request for the images. If the firewall prevents any outgoing connection from the server, CF will not be able to retrieve them and will show a red-x in place of them. You will need to setup your firewall in such a way that server can send an HTTP request to itself.
If your server is behind a proxy. If Coldfusion server is connected to the external world using a proxy, then also CFDocument will not be able to load the images. This is because currently there is no way you can specify proxy configuration for CFDocument tag.
Current solution to solve this is to define the following system properties for the JVM. You can specify these in "runtime/bin/jvm.config" if you are using standlone or on JRun server.

-DproxySet=true -DproxyHost=[hostname] -DproxyPort=[port] -Dhttp.proxyHost=[hostname] -Dhttp.proxyPort=[port]
If you are using HTTPS and your images do not appear in the pdf/flashpaper, you must ensure the following

CF server's certificate is trusted. In other words, certificate of the CA who issued the certificate for you, must be present in the trusted certificate store (runtime/lib/trustStore). You can use keytool to list/view/import/.. certificate in the certificate store.
If CF is using a self signed certificate, CF's certificate must be present in the trustStore.

The certificate is valid and has not expired.

Host name of the server must match the host name to which the certificate was issued.

If the resources on your webserver are protected using some kind of authentication like basic authentication or digest authentication, cfdocument can not retrieve those resources. That is because you can not provide any authentication information to cfdocument tag currently. This means that cfdocument can not retrieve images if it is protected using authentication and you will see a red-x. One solution for this is to replace all image urls with "file" urls.
See this entry for more details on this workaround. The other solution is to write little java code to set a 'java.net.Authenticator'. I will post a separate entry for this.

You get red-x for images and you have verified that its none of the above mentioned cases. Time to check the web server now. We have seen some cases where the web server is configured to allow requests only from a certain set of browsers (User agents to be precise) perhaps to prevent spiders and bots from overloading the server. When CFDocument creates a URLConnection for the image, it sends a "User-Agent" header, that looks like "User-Agent:Java/1.4.2_07", in the HTTP request. If the web server does not recognize "Java" user-agent, it returns a status code of 404 (resource not found) and hence the images can not be displayed. Solution for this case is to either change the configration for the web server or set your own user agent using the following system property on the JVM.
```
    -Dhttp.useragent="ColdFusion"
```
You can give any name here as userAgent in place of "ColdFusion".

Hope this helps people in resolving isues related to missing images in CFDocument. In case you are getting a red-x even after verifying all the cases mentioned above, please let me know.

Related Entry :