Showing posts with label coldfusion. Show all posts
Showing posts with label coldfusion. Show all posts

Friday, July 31, 2009

ColdFusion ORM : Troubleshooting - 'Lazy' does not work

Few days back Manju logged a bug in CF-ORM saying 'lazy' does not work for many-to-one relation and that too on non-Windows machine. At first, I simply rejected the bug because a) ORM can not have anything to do with OS and therefore, if it works on Windows, it works on all the plaform and b) I know it works :-). But he did not agree and I had to take a look at that machine. And apparently he was right - lazy was not working ! The related entity was in-fact getting loaded immediately. (Question for you - how will you know that lazy is working or not?)

Even after seeing this, I did not believe it and asked him to replicate this on another system and he successfully showed that to me on one another system. And he agreed that it works fine on most of the configurations. The problem exists only on a few of the systems.

This got me thinking - Why would a relation get loaded immediately even after it is marked lazy? The only answer would be - if some one is accessing that lazy field and calling some method on it. I checked his code which was loading the entities to see if there could be any case, where the field would get loaded and unfortunately there was none.

And then suddenly it hit me - what if "memory tracking" is swithched on? That would access each of the field of each object recursively to compute the size of an object and that can definitely cause this. I immediately checked the server monitor and the "memory tracking" was right there staring at me in "red"! It was indeed enabled. I asked Manju to check the other system as well (where lazy was not working) and the memory tracking was enabled there as well.

So the lesson - If the 'memory tracking' is enabled on the server, the relationship will no longer remain lazy. And btw, you should enable "Memory tracking" on the server only if you need to track the memory for some troubleshooting. Memory tracking is really really expensive in terms of performance.

Another reason why it might not work for you could be - if you are sending the object to flex. Currently, during serialization, the related objects also get sent irrespective of the lazy attribute set on the relationship. We are still working on it and hopefully by the time we release, this will be fixed.

Monday, July 27, 2009

ColdFusion ORM : How to log SQL

When you use ORM for developing an application, SQLs are generated and executed by the underlying ORM engine (i.e Hibernate for ColdFusion ORM). However, for both troubleshooting and performance optimization, it is crucial to monitor what queries are getting generated. It can help you find out if there is any error in mapping that you have provided as well as it can help you decide various tuning strategies.

ColdFusion can log the SQLs generated by ORM either onto the console or a file. At the same time it leaves enough hook for you to log it anywhere you want.

ColdFusion ORM provides two ways to log the SQLs.

  1. Using application setting to log to console : This is a quick and simple way to log the sql to console. This is enabled by setting "logsql" in ormsettings.
       <cfset this.ormsettings.logsql="true">
    This setting is self sufficient and it will log all the sqls executed by hibernate to the console (or a file where the server output goes). However this is not a very flexible option. The sqls are always written to the console and it will be combined with any other output that goes to console. Also this option will not show the DDL queries used for creating or updating tables. It only logs the SQL for the entity operations.
  2. Using log4J.properties: Hibernate uses log4j for its logging and you can completely control its logging (including SQL) by modifying the log4j.properties. log4j.properties is present under <cf_home>/lib directory. Please note that you don't need to do any application specific setting for this.

I will go in details about using log4j.properties for SQL logging. Here is a snippet from log4j.properties file that is shipped with ColdFusion.

###--------------- Hibernate Log Settings ------ 
### Set Hibernate log
log4j.logger.org.hibernate=ERROR, HIBERNATECONSOLE

### log just the SQL
#log4j.logger.org.hibernate.SQL=DEBUG, HIBERNATECONSOLE
#log4j.additivity.org.hibernate.SQL=false
### Also log the parameter binding to the prepared statements.
#log4j.logger.org.hibernate.type=DEBUG
### log schema export/update ###
log4j.logger.org.hibernate.tool.hbm2ddl=DEBUG, HIBERNATECONSOLE
### log cache activity ###
log4j.logger.org.hibernate.cache=ERROR, HIBERNATECONSOLE

# HibernateConsole is set to be a ColsoleAppender for Hibernate message using a PatternLayout.
log4j.appender.HIBERNATECONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.HIBERNATECONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.HIBERNATECONSOLE.layout.ConversionPattern=%d{MM/dd HH:mm:ss} [%t] HIBERNATE %-5p - %m%n%n
#---------------------------------------------

First we will see the relevant settings for SQL logging:

  • log4j.logger.org.hibernate.SQL : Defines whether the SQL executed for entity operations will be logged and where it will be logged. The second value for this i.e 'HIBERNATECONSOLE' is a appender that controls where the SQLs will be logged. In the above example HIBERNATECONSOLE is a 'console' appender which means it will log the sql to console.
  • log4j.logger.org.hibernate.type : Defines whether parameter values used for parametrized query will be logged.
  • log4j.logger.org.hibernate.tool.hbm2ddl : Defines whether DDL sql statements will be logged.

To enable the SQL logging for console, we just need to uncomment the settings mentioned above. Here is how the hibernate log settings in log4j.properties file would look like

###--------------- Hibernate Log Settings ------ 
### Set Hibernate log
log4j.logger.org.hibernate=ERROR, HIBERNATECONSOLE

### log just the SQL
log4j.logger.org.hibernate.SQL=DEBUG, HIBERNATECONSOLE
log4j.additivity.org.hibernate.SQL=false
### Also log the parameter binding to the prepared statements.
#log4j.logger.org.hibernate.type=DEBUG
### log schema export/update ###
log4j.logger.org.hibernate.tool.hbm2ddl=DEBUG, HIBERNATECONSOLE
### log cache activity ###
log4j.logger.org.hibernate.cache=ERROR, HIBERNATECONSOLE

# HibernateConsole is set to be a ColsoleAppender for Hibernate message using a PatternLayout.
log4j.appender.HIBERNATECONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.HIBERNATECONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.HIBERNATECONSOLE.layout.ConversionPattern=%d{MM/dd HH:mm:ss} [%t] HIBERNATE %-5p - %m%n%n
#---------------------------------------------
Here is the complete log4j.properties for logging SQL for console. Ofcourse after changing this you need to restart the server. If you need to log the parameter values used for queries, you need to uncomment '#log4j.logger.org.hibernate.type=DEBUG' as well.

What if you want to log the SQL to a file and not to console? That is pretty easy. You just need to change the 'Appender' used here (HIBERNATECONSOLE) to point to a 'FileAppender' instead of a ConsoleAppender. Here is how the configuration for HIBERNATECONSOLE should look like after you point it to a File Appender.

log4j.appender.HIBERNATECONSOLE=org.apache.log4j.FileAppender
log4j.appender.HIBERNATECONSOLE.File=../hibernatesql.log
log4j.appender.HIBERNATECONSOLE.Append=true
log4j.appender.HIBERNATECONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.HIBERNATECONSOLE.layout.ConversionPattern=%d{MM/dd HH:mm:ss} [%t] HIBERNATE %-5p - %m%n%n

For standalone ColdFusion installation, the file 'hibernatesql.log' will be created in the /logs directory.You can also specify a full path of the file for property 'log4j.appender.HIBERNATECONSOLE.File' and the log will be written to that.

That was easy. Isn't it? What if you want a rolling log file where you dont want the log file size to grow infinitely. That is fairly easy too. All you need to do is to use an appropriate appender. The appender definition for that will look like

log4j.appender.HIBERNATECONSOLE=org.apache.log4j.RollingFileAppender
log4j.appender.HIBERNATECONSOLE.File=../hibernatesql.log
log4j.appender.HIBERNATECONSOLE.Append=true
log4j.appender.HIBERNATECONSOLE.MaxFileSize=500KB
log4j.appender.HIBERNATECONSOLE.MaxBackupIndex=3
log4j.appender.HIBERNATECONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.HIBERNATECONSOLE.layout.ConversionPattern=%d{MM/dd HH:mm:ss} [%t] HIBERNATE %-5p - %m%n%n

Here is the complete log4j.properties for logging SQL to a file that will be rolled automatically once it reaches 500KB.

Now that you have seen how easy it is to change one 'Appender' to another, you can pretty much log it anywhich way you want. Here are some of the interesting 'Appender's that come with log4j which you can easily use.

DailyRollingFileAppender, ExternallyRolledFileAppender, JMSAppender, SMTPAppender, SocketAppender, SyslogAppender

See log4J for more details on log4j settings.

Saturday, July 18, 2009

ColdFusion 9 : Implicit/Generated CFC Methods

How many times did you write the accessor methods (better known as getters and setters) for the fields of your CFC? And how many times did you feel that it was really a mundane job and wished there was a better way to do this? With ColdFusion 9, we have done just that! You no longer need to write those accessors. ColdFusion will automatically generate the accessor methods for you in the object. All you need to do is to define the properties using cfproperty.

Consider this CFC Person.cfc

Component Person{
property firstname;
property lastname;
property age;
property city;
property state;
}

When an object is created for this CFC, you can directly call the setters and getters for the properties as


person = new Person();
person.setFirstName("Brad");
person.setLastName("Pitt");
person.setAge(46);

writeOutput("Name : #person.getFirstName()# #person.getLastName()#");
writeOutput("Age : #person.getAge()#");


The implicit setter/getter uses the 'variable' scope, which is a private scope for a object, for storing the data. i.e the setter method puts the data in the variable scope and the getter method gets the data from the variable scope.

What if you want to override the setter/getter method? Ofcourse you can do that. You just need to define those methods in your CFC and ColdFusion will call your method instead of calling the implicit one.

Hmm.. What if you dont want ColdFusion to generate getter or setter or both methods for a particular property? You can disable that by adding attribute getter="false" and setter="false" on the property.

So if you define a property

property name="city" getter="false" setter="false";

ColdFusion will not generate getCity() and setCity() methods for this CFC.

The implicit methods can also do some nice validation if you have given the 'type' for properties. Consider the same Person.cfc with type specified where we have made age as a numeric type.


Component Person{
property string firstname;
property string lastname;
property numeric age;
property string city;
property string state;
}

ColdFusion will do the datatype validation whenever the setter method is called for any property. So if you call


person = new Person();
person.setFirstName("Brad");
person.setLastName("Pitt");
person.setAge("LA");

It will throw a nice error saying "The value cannot be converted to a numeric".

Whats more interesting is that you can do a whole lot of validation using cfproperty. We have added two more attributes on cfproperty named 'validate' and 'validateparams'. These attributes allow you do even more advanced validations of the property data when the setter method is called for a property. (This is similar to cfparam or cfform validation).

The possible values for validate attributes are

  • string
  • boolean
  • integer
  • numeric
  • date
  • time
  • creditcard: A 13-16 digit number conforming to the mod10 algorithm.
  • email: A valid e-mail address.
  • eurodate: A date-time value. Any date part must be in the format dd/mm/yy. The format can use /, -, or . characters as delimiters.
  • regex: Matches input against pattern specified in validateparams.
  • ssn: A U.S. social security number.
  • telephone: A standard U.S. telephone number.
  • UUID: A Home Universally Unique Identifier, formatted 'XXXXXXXX-XXXX-XXXX-XXXXXXXXXXXXXXX', where 'X' is a hexadecimal number.
  • guid: A Universally Unique Identifier of the form "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" where 'X' is a hexadecimal number
  • zipcode: U.S., 5- or 9-digit format ZIP codes

The 'validateparams' attribute available with <cfproperty> takes the parameters required by the validator specified in the 'validate' attribute. This needs to be specified in the implicit struct notation.
  • min: Minimum value if 'validate' is integer/numeric
  • max: Maximum value if the 'validate' is integer/numeric
  • minLength: Minimum length of the string if the 'validate' is string
  • maxLength: Maximum length of the string if the 'validate' is string
  • pattern: regex expression if the validator specified in 'validate' attribute is regex
Consider the Person again where we will add few more properties with some validation

Component Person{
property string firstname;
property string lastname;
property numeric age;
property string city;
/**
* @validate string
* @validateparams {minLength=2, maxLength=2}
*/
property string state;
/**
* @validate zipcode
*/
property numeric zip;
/**
* @validate telephone
*/
proeprty phone;
}

Now when you create an object of Person and call setState(state), before setting the data for state in variable scope, ColdFusion will validate that state value provided is of type 'string' and its length is 2. Similarly setZip() will validate that the input data is a valid zip code and setPhone() will validate that the input data is a valid telephone number.

For String type properties, you can also do regular expression validation allows you to have all sort of validation on a property of string type.

And to top it all, all of these nice goodies are available to you at a much lesser cost than regular UDFs. Yes, you read it right. Implicit methods perform much better than regular UDFs that you write in CFC. To confirm that, lets use this CFC where we will use implicit methods for 'firstName' and write our own accessors for 'lastName'.


component 
{
property String firstName;
lastName = "";

public void function setLastName(string name)
{
lastName = name;
}

public string function getLastName()
{
return lastName;
}
}

Both implicit methods and UDFs are doing exactly the same thing i.e settings and getting the data in/from variable scope. Now lets run this comparison test



person = new Person();
t = getTickCount();
for(i = 0; i < 100000; i++)
{
person.setfirstName("Brad");
fname = person.getFirstName();
}
writeOutput("Total Time in implicit methods " & (getTickCount() - t));
t = getTickCount();
for(i = 0; i < 100000; i++)
{
person.setLastName("Pitt");
lname = person.getLastName();
}
writeOutput("Total Time in UDF methods " & (getTickCount() - t));

Here is the output
Total Time in implicit methods 63
Total Time in UDF methods 485

Thats almost eight times faster than regular UDF. Isn't that sweet? You get those auto-generated methods, whole lot of auto-magical validations and to top that, incredible performance. I am sure writing accessors for your CFC fields will become a thing of past now !

Thursday, July 16, 2009

ColdFusion ORM and CFC Performance

Since the day we started thinking about ColdFusion ORM, people have been raising concerns over it because of the CFC performance. We have heard people saying that ORM will be unusable because of poor CFC performance and it is a stupid idea to implement CFC based ORM. And today I heard Hal Helms and Brian kotek talking about it where Hal says that he shudders at the thought of hibernate integration in CF because of poor performance.

It is true that CFC performance is poor as compared to POJO and it will always be. ColdFusion CFC has lot of cost involved because of the dyanmic nature of it and additionally, it has reflection cost involved. I had shared some of the reasons with few of the community members and I thought it will be interesting to share it witha wider audience. Here are the main reasons :

  1. Unlike java where the class has a fully qualified name and is loaded only once in the classloader, when you create CFC, CF needs to search for the CFC file on the disk, compute the class name for it and then load it. And that happens on each object creation unless you have switched on trusted cache .
  2. CFC  is compiled at runtime, loaded and constructed using reflection. I am sure you will be aware that reflection is much slower (25 times easily) than direct invocation in java.
  3. CFC has two different scopes - this and variables scope which is a struct. So every object creation involves two additional struct creation cost.
  4. CFC creation also involves running the default constructor i.e any code outside cffunction and that happens for each object creation. That involves the complete setup for method invocation i.e setting up localscope, superscope etc (another two structs here), which is also significant
  5. CFC allows UDFs to be added/removed from the object at runtime and hence methods are added to each object on object creation and that too in both the scopes (adding each UDFs in both the scopes also has significant cost). Thats the reason why object creation becomes costlier as you add methods in the CFC. Higher the number of methods, higher the object creation cost.

We have done a lot of performance improvements regarding CFC creation and invocation in ColdFusion 9 which I hope you would have noticed. But even then it can not match POJO performance and thats an unrealistic target.

So given that, lets come to the question - How much does CFC creation performance matters when it comes to ORM. In reality, not much. We have done a number of performance comparison between CFC and POJO with hibernate keeping everything else absolutely same. And all the time, CFC performance was same or better than POJO persistence. It might sound unbelievable and in fact I also could not believe it myself when I saw the performance numbers first. But that is the truth. And here is why it happens - When it comes to persistence, the time taken in DB operation and the persistence layer is much more than the time taken in object creation. And that cost is same for POJO and CFC. So the only difference in cost between CFC and POJO with respect to ORM is object creation cost and cost of populating/fetching the properties data. For instantiating POJO object and for using properties accessors, Hibernate has to use reflection which is always costly and that removes this cost difference of CFC and POJO. And for few of the runs, CFC was able to beat POJO as well and explanation for that is the optimizations that we have done in using accessors for properties.

I would be curious to know about your experience regarding CF-ORM performance and CFC performance in general. How has your experience been so far regarding CF-ORM peformance?

ColdFusion ORM : The basics

Before starting on the advanced topics, I thought it will be better to build some ground and hence I decided to do a post on the ORM basics. In this post, we will build a simple example to get a taste of ColdFusion ORM (CF-ORM) and during that we will also understand some of the basic concepts.

ORM is object relational mapping and in ColdFusion, objects are created using CFC. CFCs that needs to be persisted are called persistent CFC and that is marked by setting 'persistent' attribute to true on the component. We also need to define what persistent fields will be there in a persistent CFC and that is defined using 'cfproperty'. A field/property is marked persistent by setting persistent attribute to true on the cfproperty. By default, if the CFC is persistent, all its properties are considered as persistent unless you mark a property non-persistent. So typically 'persistent' attribute on the property is used only when you need to make that property non-persistent.

Each persistent CFC in ColdFusion application maps to a table in the database and each property in the persistent CFC maps to a column in the table (not exactly true but we will come to that later.. For the time being lets keep it that way). We will use the cfartgallery datasource for this example which has Artists and Art tables.

The first thing you need to do is - enable ORM for the application and define a datasource to be used (What is an ORM without a datasource?). ColdFusion ORM uses Application.cfc to define all the ORM specific settings. (If you haven't started using Application.cfc for your application, its time to start using it!)

Application.cfc


Component {
this.name="ArtGallery";
this.ormenabled="true";
this.datasource="cfartgallery";
}

Note that the datasource setting defined here makes it the default datasource of your application which can be used by tags like cfquery, cfinsert, cfupdate, cfdbinfo. The same default datasource will be used by ORM as well.

There are a whole bunch of ORM related configuration that you can do in application.cfc which you can refer here.

Now that the application is configured, let us build the object and define the persistence information on it. To start with, we will first define the Artist.cfc
Artists.cfc

/**
* @persistent
*/
Component {
property name="artistid" generator="increment";
property firstname;
property lastname;
property address;
property city;
property state;
property postalcode;
property email;
property phone;
property fax;
property thepassword;
}

This is the most simplistic definition of the component where we have defined only the component and its properties names. Since the table for this CFC already exists in the database, we have not added any table specific information in this and we will let ORM infer all the information from the database. The only additional setting that we have added here is the 'generator' attribute which is used to auto-generate the primary key.

After the components are defined, the first request to this application (i.e a page in this application) will make CF-ORM do all the setup necessary (basically generation of hibernate configuration, mapping files, building the session factory etc). Once the setup is done, you are all set to work with the entities.

We will first list all the artists and here is what you need to do for that

listAll.cfm
<cfscript>
artists = EntityLoad("Artists");
writedump(artists);
</cfscript>

To load a particular Artists with its ID, here is what you do
list.cfm


<cfscript>
artist = EntityLoadByPK("Artists", 1);
writedump(artist);
</cfscript>

There are several flavors of EntityLoad functions details of which can be read here

Let us now see how to perform insert and update on it.

save.cfm


<cfscript>
// Insert a new artist
artist = new Artists();
artist.setfirstname("Leonardo");
artist.setlastname("Da vinci");
artist.setcity("Paris");
EntitySave(artist);
writeOutput(artist.getartistid());// Update an artist
artist = EntityLoadByPK("Artists", 2);
artist.setcity("NewYork"); // artist is automatically updated.
</cfscript>

As we see in the above example, EntitySave is used to insert/update an object in the table. There are some important things to note here
  • EntitySave is an intelligent function which automatically finds if a new row needs to be inserted for the given object or whether an existing row needs to be updated.
  • We called EntitySave for the insert here but not for update but even then artist '2' gets updated. So how did that happen? Actually what happens here is when you load an artist object, it becomes associated with the hibernate session which keeps track of any changes in the object and automatically saves it when the session is flushed. We will talk about more about hibernate session in a later post. For the time being lets just say that Hibernate Session is a short-lived object that represents a conversation between the application and the persistence layer and also acts as the first level of cache.
  • We did not write any setter or getter method for artist's properties in Artists.cfc but we are calling them here. That works because ColdFusion 9 automatically generates accessor methods for any property written in a CFC. More details on generated methods in a later post.
  • At line no 6, we called entitySave, but if you check the database, the row is not inserted yet. So when does that happen? Hibernate batches up all the operations till the end of the request or to be exact till hibernate session is flushed. ColdFusion ORM starts up a session when the first ORM method is called in the request and is automatically flushed when the request ends. The batching is done for performance reason so that hibernate executes the sql with the final state of the objects. It will be a huge performance bottleneck if ORM keeps executing sql for each changes in the object.

To delete an Artist, you need to call EntityDelete() passing the object to be deleted.

delete.cfm


artist = EntityLoadByPK("Artists", 15, true);
EntityDelete(artist);

Relationship


So far we have seen how to perform CRUD for a single entity. But in any application, there will be entities which are associated and ORM must load the associated object as well when loading a particular entity. For our example, an Art will have an Artist and when loading the art object, it should also load the associated artist. So lets build the model first after which we will see how to work with the association.

In cfartgallery, the table Artists has a one-to-many relationship with Art table, which are joined using the foreign key column ARTISTSID. This means that each artist has created multiple arts and each art is created by one artist. To represent this in the object model, each ARTIST object would contain an array of ART objects. Each ART object will also contain a reference to its ARTIST object thereby forming a bidirectional relation.

To achieve this, we will need to add an extra property 'arts' to 'Artists' that contains an array of ART objects for that Artist. The modified Artists.cfc would look like
/**
* @persistent
*/
Component {
property name="artistid" generator="increment";
property firstname;
property lastname;
property address;
property city;
property state;
property postalcode;
property email;
property phone;
property fax;
property thepassword;
property name="arts" fieldtype="one-to-many" fkcolumn="artistid" cfc="Art" cascade="all" inverse="true";
}

Here is the Art.cfc

/**
* @persistent
*/
Component {
property name="artid" generator="increment";
property artname;
property price;
property largeimage;
property mediaid;
property issold;
property name="artist" fieldtype="many-to-one" fkcolumn="artistid" cfc="Artists" ;
}

Notice the artist property above which is of many-to-one type. Also notice that both the property use the same value for fkcolumn attribute i.e 'artistid' of Art table that references artistID pk of Artist table.


Since we have added a new persistent CFC (Art.cfc) after the application was loaded, we need to re-initialize the ORM for this application so that mappings for Art.cfc also gets generated. This can be done by calling ORMReload() method. There are some nice ways to do this but for the time being lets keep it simple by putting this in a separate page which we will call to reload ORM.

initializeORM.cfm
<cfset ormReload()>

If you load and dump Artist (using listAll.cfm), you should also see the associated art objects for artists.

Now let us create a new Art and associate it with an existing Artist.
artCreate.cfm

<cfscript>
artist = EntityLoad("artists", 1 ,"true");
art = new Art();
art.setartname("landscape");
art.setPrice(1500);
art.setissold(false);
art.setartist(artist);
artist.addArts(art);
EntitySave(art);
</cfscript>

If you notice line 7-8 above, we associate artist to art by calling art.setArtist(artist) as well as art to artist by calling artist.addArts(art). Hibernate needs us to do this in order to set up the bidirectional relation properly. Since it is a bidirectional relation, you must also decide which side will set the relation in the database. i.e which side of the relation will set the fkcolumn in the table. This is controlled by the "inverse" attribute of proeprty, which if set to true, tells hibernate that this is a inverse of the other relation and this side of relation should be ignored for persistance. If you don't set inverse to true, Hibernate will unnecessarily fire two sqls for the same association.

So there you have it. We have seen how you can use ORM to perform the basic CRUD operations on entities. For more details, you can refer to the ORM doc and Hibernate docs.

Monday, July 13, 2009

ColdFusion ORM - An Evolution in building datacentric application

As we all know, ColdFusion was born as DBML - a script that was very similar to HTML but had certain tags that could perform database operation and it revolutionized the way web applications were built. Tags like cfquery, cfinsert, cfupdate made it very easy to perform database operations while building web application. Even after 14 years of CFML, (which we incidently completed day before yesterday), cfquery is *the* most popular and the most commonly used tag. However there are few downsides to it

  • Database vendor dependency - The queries that are written are DB specific. If you ever need to switch the database or if you need to make the application database agnostic, it becomes too much of pain. You will end up changing the entire application code or lot of conditional code to handle queries for different databases.
  • Very tight coupling with the database - The queries have very tight coupling with the database. This means that if there is any change in the table name or column name, you will end up with changes all over your application code.
  • Repititive and hence low on productivity - For any CRUD (Create, read, update and delete) operation, you need to write queries which are more or less same for all the tables. You must have felt that you are repeating the same code everywhere in the application. There is too much of repititiveness which reduces productivity.
  • Error prone - How many times did you write insert/update queries and then you had to struggle to debug because the order of column and its values were different? And how many times did you get frustrated because there was a datatype mismatch between the value you had specified and the one expected? How many times did you write a query that was prone to sql injection and you were reminded to use query param? I am sure all of this has happened to all of us and its completely natural given the way queries are written. They are error prone.

ColdFusion 9 introduces a new way to build datacentric application using ORM, that handles all the downsides we saw above. It is not a replacement to cfquery/cfinsert/cfupdate tags but its a different approach altogether. It allows you to build your application using objects where you focus on the business logic and all the grunt work of persistence is taken care automatically (In simple words you dont write SQL queries generally).

ORM (Object relational mapping) is a well known strategy/technique to map relational data to the object model. In an object model, business objects are not aware of the database structure. Objects have properties and references to other objects whereas Databases consist of tables with columns that are related to other tables using foreign key. ORM provides a bridge between the relational database and the object model allowing you to access and update data entirely using the object model of an application.

ORM is not new to ColdFusion either - Transfer and Reactor being the most popular ones. ColdFusion ORM (or CF-ORM) is much more extensive and sophisticated ORM solution than these ORMs. ColdFusion ORM is built on top of Hibernate - the best java ORM engine out there, which is quite powerful and extensive. ColdFusion ORM makes it very easy to use persistence with objects and at the same time allows you to leverage the full power of Hibernate.

ColdFusion ORM provides features such as

  • Defining the persistence mapping and managing the object's persistence using very simple methods.
  • Database vendor independence
  • Loose coupling between database and application - The application becomes very adaptive to changes as the configuration is central and changes will be required only there.
  • Auto-generation of schema - ORM lets you auto-generate the schema for your application. So you never need to bother about keeping your object model and the database schema in sync. It can automatically happen.
  • Productivity and manageability - Since ORM takes care of all persistence grunt work, you focus on your application logic and thus your application becomes much more cleaner and manageable. It also makes you lot more productive as application can be built much more faster.
  • Concurrency - ORM inherently takes care of concurrency control. So you dont need to worry much about how things would work when multiple database operations happen in parallel web requests.
  • Performance - ORM provides lot of performance optimizations that can make your application run faster. The optimizations include two levels of caching (including pluggable distributed cache), lazy loading, various settings to tune the sql queries generated by ORM.
  • Secure - Since queries will be executed by ORM, issues like SQL injection no longer exist and thus your application becomes secure.
  • Inbuilt pagination
  • SQL like query (HQL) for a finer control of the data to be loaded.
  • ...

In this series of posts, I will be talking lot more about ORM which I hope you would like. In case you have not downloaded ColdFusion 9 beta, go download it and start playing with it.

Monday, February 18, 2008

Reading MP3 meta-data from ColdFusion

On friday I posted about how to read the meta-data for Swf (flash) file from ColdFusion and on the same note, I think it will be cool to read the meta-data for MP3 file as well. Here is the specification for MP3 header and it is fairly straight forward to read the header using the spec.

ColdFusion 8 knows how to read MP3 file and it can provide a set of meta-data for it. Though this functionality is not exposed via any tag or function, you can still use it. The meta-data it provides are

  • Bit Rate : in kbps
  • Frequency : in Hz
  • Play duration : in second
  • Version : integer
      • 0 - MPEG version 2.5
      • 1 - MPEG version 2
      • 3 - MPEG version 1
  • Copy Right : true/false
  • Channel mode : integer
      • 0 - Stereo
      • 1 - Joint Stereo
      • 2 - Dual Channel
      • 3 - Mono or Single Channel

Here is an example code.

<cfset mp3File = createObject("java", "coldfusion.util.MP3File").init("C:\music.mp3")>
<cfoutput>mp3File.getBitRate() : #mp3File.getBitRate()# kbps</cfoutput><br>
<cfoutput>mp3File.getFrequency() : #mp3File.getFrequency()# Hz</cfoutput><br>
<cfoutput>mp3File.getVersion() : #mp3File.getVersion()#</cfoutput><br>
<cfoutput>mp3File.getDuration() : #mp3File.getDuration()# Sec</cfoutput><br>
<cfoutput>mp3File.isCopyRighted() : #mp3File.isCopyRighted()#</cfoutput><br>
<cfoutput>mp3File.getChannelMode() : #mp3File.getChannelMode()#</cfoutput><br>

Though you can use the above mentioned class (coldfusion.util.MP3File) without any problem, you must keep it in mind that it is unsupported and *might* change in future. Just a disclaimer !!! :-)

This class does not get you the ID3V1 or ID3V2 meta-data, but it should be easy for you to do it. For ID3V1, all you need to do is to read the last 128 bytes of the mp3 file and you have all the information required. Reading ID3V2 tag is little more involved and there are a number of open source java libraries available (common one being jidlib and jaudiotagger) which can be used.

Friday, February 15, 2008

Reading Flash (Swf) metadata from ColdFusion

Wouldn't it be cool if you could read the meta-data of Swf file in ColdFusion? The most common use case I can think of is to find the dimension of swf movie so that you can set the dimension for the same in object tag while you are embedding the flash movie in your page. And the good news is that you can do that currently in ColdFusion with just 2-3 lines of code!! All you need to do is to create a TagDecoder and decoder the header with it. Here is the complete code.

<cfscript>
fis = createObject("java", "java.io.FileInputStream").init("C:\test.swf"); // Create the FileInputStream
decoder = createObject("java", "macromedia.swf.TagDecoder").init(fis); // create TagDecoder
header = decoder.decodeHeader(); // Decode the header.
fis.close();
rect = header.size;
WriteOutput("
Is Compressed : #header.compressed#<br>
Frame Count : #header.framecount#<br>
Frame rate : #header.rate#<br>
Version : #header.version#<br>
Height : #rect.getHeight()#<br>
Width : #rect.getWidth()#");
</cfscript>

The height and width value that you see here would be in twips which is defined as 1/20 of a point or 1/1440 inch. This means, for a 72 dpi screeen, you will have to divide it by 20 to get the height and width in pixels.

Thursday, January 10, 2008

DNS lookup Caching in ColdFusion/Java

Two days back, I got an interesting question from our support team. The Customer in this case was using CFLdap which connected to a particular LDAP server and things were working fine. The problem came when they replaced this LDAP server with another LDAP server and assigned the same dns name to the new ldap server. Ideally any connection made henceforth should have worked with the new ldap server but actually that did not happen. ColdFusion started throwing error and it did not work until the ColdFusion server is restarted. Ever seen something similar?

It is obvious that it happend because the IP address was being cached for the host name as a result of which ColdFusion was still trying to connect to the old IP address even though the host name now pointed to a different IP address. This caching also applies for all other network protocol tags such as CFHTTP, CFFTP, CFFEED etc and is not limited to CFLDAP. It is actually the JVM that does this caching. When JVM is requested to resolve a host name, it does the host name to IP address resolution and caches the result. This result will be used for all subsequent lookups. It also caches the negative results. By that I mean, if the dns reolution fails, it caches the failed result for a certain period so that any lookup for that hostname in that period will not result into another resolution on network and will immediately return the failed result.

For more detail on this caching, check out the Javadoc for InetAddress class.

As per this doc, there are two parameters that control this caching

networkaddress.cache.ttl (default: -1)
Indicates the caching policy for successful name lookups from the name service. The value is specified as as integer to indicate the number of seconds to cache the successful lookup.

A value of -1 indicates "cache forever".

networkaddress.cache.negative.ttl (default: 10)
Indicates the caching policy for un-successful name lookups from the name service. The value is specified as as integer to indicate the number of seconds to cache the failure for un-successful lookups.

A value of 0 indicates "never cache". A value of -1 indicates "cache forever".


Now where do you specify this settying? You can specify this setting in <Java_home>/jre/lib/security/java.security file. For standalone ColdFusion server it will be in <ColdFusion _dir>/runtime/jre/lib/security/java.security file.

As you see, by default networkaddress.cache.ttl caches the result for ever and hence it is configured for best performance. Any change to this mean drop in performance. If you don't want to cache the resolved IP address for ever, as is the case here, you would need to change networkaddress.cache.ttl value to 60 seconds or 300 seconds or any value you feel suitable. You would not want to set it to 0 as that would mean "never cache" the result which might affect the performance significantly.

In which case you would want to change the value for networkaddress.cache.negative.ttl? That would be mostly in case when you want to cache the negative result for a longer time and in turn improving the performance. For example, if you are trying to connect to a hostname which can not be resolved to any ip address, and that happens very frequently, each of the call (as long as they are not in the same 10 sec window) would become very slow. Increasing this value would increase the performance but again you would not want to cache the negative result for ever.

After you change this setting, you will have to restart the ColdFusion server for this change to take effect.

Monday, January 07, 2008

ColdFusion 8 finalist among Dr. Dobb's Jolt Product Excellence Awards

Good to see ColdFusion 8 as a finalist in Web Development tools category among Dr. Dobb's 18th Annual Jolt Product Excellence Awards. Here is the entire story and here is the list of finalists.

Adobe ColdFusion IDE Survey

As part of our research on CF9 (Centaur), the ColdFusion team has come up with a survey to find out what features do you use and would like to have in an IDE for Coldfusion. Please take few minutes to take this quick survey.

http://www.surveymonkey.com/s.aspx?sm=321RrO9_2fWaP_2bdYMnmF9CuQ_3d_3d

Check out this space in few days for another survey to tell us what you would want to see in CF 9 !

Wednesday, January 02, 2008

Encrypted CFML Templates

Coming back from the 10 days shutdown period that we had at Adobe, I am catching up with the blogs and I read this interesting post by Ray where he mentioned that some one got his cfml templates encrypted and then lost the originals. Hmmm.. that is a situation, I would never want to be in. In case you have not figured it out yet, it is not possible to get back the original source. Let me explain.

Why do we need to encrypt cfml templates? To protect the cfml code and your intellectual properties (IP) in case you want to distribute your CF application. Right? In pre-CFMX7 era, ColdFusion used to have "cfencode" utility which used to encrypt the template which you can then distribute to anyone. ColdFusion used to know how to decrypt that and then how to execute that. However, this was not a good solution as people came up with utility to decode this encrypted file. And THAT means that  your code and IP was not protected at all. This had another disadvantage - template execution was slower as it meant decrypting the template first and then compiling/executing it.

With CFMX7, this changed. ColdFusion came up with sourceless deployment to make it nearly impossible to decode it back to the original source. When you use 'cfcompile' with deploy option, your cfml template is compiled to java byte code (class) but the file in which this byte code is written still retains the same template name. To give an example, if you run cfcompile on "hello.cfm", the output would still be "hello.cfm" but it will actually be java class file. To confirm this, just open one of such encrypted file in a hex editor and you would see the first 8 bytes will be CAFEBABE - the magic number for java class files. This file can then be distributed to your customers. For execution, ColdFusion works smart - when request comes for this template, it sees that it is not a cfml template but a class file, skips parsing and compiling the template, and directly proceeds with the execution.

As you see, there is no encryption and no key involved here. The encrypted file is simply a java class file and I have not heard of any utility that can decompile this class file back to cfm source. Though there are many Java decompilers available which can convert the class file to approximate java source file, it will be a huge huge task to write a decompiler which can generate cfm code for class file. One needs to know how CF does the code generation, know a great deal about bytecode instructions and apply lots and lots of heuristic to get the approximate cfm source. This is definitely not an easy thing to do even if some one breaks open the CF engine to see how it generates the java code from cfm code.

The bottom line is - in case you use sourceless deployment, ALWAYS backup your original source.

Wednesday, November 21, 2007

64 bit Support on ColdFusion 8 (Windows, Linux and Leopard)

ColdFusion team is working to get ColdFusion 8 working on 64 Bit platforms(Windows, Linux and Leopard). If you are interested and want to participate in the testing of early releases of 64 Bit support on ColdFusion 8 on these platforms, please enroll yourself for prerelease program here.

Wednesday, October 10, 2007

The Week at Max

Wow ! What an incredible week it was. A week that saw the largest ever MAX, a week that saw the largest get together of ColdFusion community, a week that saw the best ever visibility of CF at MAX general sessions, a week that saw U2's Bono (err.. his fake??) having fun at a tech conference like MAX and a week that saw lot of cool and amazing Adobe technologies shaping up.

It also had moments of a Hollywood thriller for me. I was presenting a session on "Leveraging ColdFusion for AIR application". My first session was on Tuesday, 2nd October at 4 o'clock and at around 2 o'clock my laptop crashed and after that it just refused to boot up. Luckily I had the backup of my ppt and some demo files and hence I decided to use Sanjeev's (Pdf and compiler guy in the CF team) laptop to make the presentation. I just had one and half hour and I needed to setup everything back on his machine - install the air runtime, air sdk, cfeclipse, copy all my files etc and keep the setup ready to run whatever demos I was left with. To add to this, somehow there was no Internet available in the conference area - So where to get AIR runtime and SDK from? Sanjeev suggested that may be AIR guys might have it and so we ran across to the AIR park to get those on my flash drive. I can't tell you what those one and half hour were like :-). I barely managed to get the presentation ready in time and thankfully the presentation went well.

As if this was not enough, the adventure continued for the repeat session as well. No this time the laptop didn't crash.. but this time I could not find Sanjeev so that I could take the laptop from him. I just had half an hour left for the session and I did not have any laptop. And I didn't even have the patience for setting it all up once again on another laptop. Finally found him just 15 minutes before my session while he was chatting with BEA guys. phew..

Anyway, here is my presentation and some of the demo files if you are interested to take a look at that. I could not upload it earlier as I got a temporary laptop just yesterday. I will upload the flex part of demo in a day or two.

If you were there at MAX, you must have seen Hemant talking about ColdFusion and AIR at sneak peek. Though this was just a prototype, it surely gives an idea what can be done in CF for air apps. It would be great if you can share what exactly or what else would you like to see in ColdFusion to help you build AIR application easily. Thoughts??

Thursday, September 20, 2007

CFDocument hotfix for ColdFusion 8 released

We have released a cumulative hotfix for CFDocument related issues in ColdFusion 8. Bugs being addressed in this hotfix are

  • PDF/Flashpaper document generated in CF8 are nearly 70% scaled down than CFMX 7. This was making document appear smaller in CF8 as compared to CF7.
  • Using cfdocumentitem in a loop inside cfdocument causes the error, "routines cannot be declared more than once".
  • StackOverFlowError when a large document with table is created. A simple report with over 1500 table row causes this problem. More complex, but shorter documents have also resulted in this error. This was a pretty nasty one and it deserves a separate blog entry :-).

You can download the hotfix directly from here. You would need to extract the hotfix jar from the downloaded zip file and apply that jar.

Thursday, August 16, 2007

ColdFusion 8 : Changes with File Upload

Till ColdFusion 7, it was not possible to upload a large file. The server could not handle uploading a file of probably anything more than 300-400 MB. And on top of that, if there are multiple people trying to upload large files simultaneously to the server, only thing that you can do is to pray :-)

You might wonder and I agree with you, that its a huge limitation. So why was it this way ? Well.. that was because the server had to keep the entire uploaded file in the memory. Whoa.. Why the heck does the server need to keep the entire uploaded file in memory? - For a single reason of serving the function GetHttpRequestData(). As you know, this function returns a 'struct' containing the metadata for the Http request that comes to the server. Since this struct also includes request body, and the uploaded file is part of the request, the file has to be retained in the memory in order to honor this function.

With ColdFusion 8, we have removed the limitation on file upload size. So now you can upload files of any size without causing any issue with the server. I have successfully tried with 5 GB file because that was the biggest I could get hold of but you can absolutely upload files of any size. ColdFusion nicely handles it, reads it the way it should without requiring much memory.

But hey, hold on a sec.. What about the function GetHttpRequestData()? Wasn't that the reason for this limitation in the first place? Well, the function GetHttpRequestData() works perfectly fine except in this one case - In the multipart request, i.e; request in which a file is uploaded - "content" in the returned struct will be empty. If you need to get the uploaded file content in memory, you can always read the uploaded file and keep it in memory. So that should not be much we are asking for. Are we?

If in case, you absolutely want the content from GetHttpRequestData() and file upload size is of no concern to you, we allow you to revert back to ColdFusion 7 behaviour. All you need to do for this is to set the Java system property "coldfusion.markResetForMultiPart" to "true" and you are back to the old behavior.

Wednesday, August 08, 2007

ColdFusion and WebService : "class file has wrong version"

I can't even count the number of times I have seen this error while QA folks were running webservices regressions for ColdFusion 8. Everytime, someone was running webservices tests on a J2EE configuration they used to get an error :

coldfusion.jsp.CompilationFailedException: Errors reported by Java compiler:
....
class file has wrong version 49.0, should be 48.0

I saw the same error posted in the public forum where the user had deployed ColdFusion 8 on JRun with JDK1.5 and was trying to invoke webservice. And hence this blog entry :-)

This error comes because of incompatibility between the tools.jar and the JVM version. ColdFusion 8 ships Java 1.6's tools.jar (you will find it in cfusion/lib) which is of course incompatible with any other JDK/JRE version. You will see the same behavior with ColdFusion 7 as well, if you run it on a JVM other than 1.4. So the rule of thumb is - if ColdFusion is running on a JVM version other than the shipped one, you must ensure that the tools.jar is in synch. In short, you will need to copy JDK's tools.jar to cfusion/lib.

Friday, August 03, 2007

Using CFFeed with URL sending compressed content

Today someone posted on ColdFusion forum regarding this problem where cffeed was not able to handle a particular URL and it was throwing an error. The URL which he tried was http://movies.msn.com/rss/topcelebs and it failed with an error

"Unable to read the source URL.
unknown compression method
"

The reason it happens is - The URL returns the response in gzip compressed format only. So when ColdFusion sent a request to this URL and asked for uncompressed data, it could not get anything and hence it was unable to read it. A simpe workaround for this is to use cfhttp to fetch the content, write to a temporary file and then use the cffeed tag to read this file. Important thing to keep in mind here is to set an additional header in the cfhttp tag using cfhttpparam to indicate that it can accept compressed data as well.

Here is the modified code where it first tries cffeed with the URL. If that fails, then it tries to use cfhttp to fetch the content and writes to a temporary file and then uses it in cffeed.

<cfset tempDir=GetTempDirectory()>
<cfset tempFile = GetTempFile(tempDir, "myfeed")>
<cfset tempFileName = GetFileInfo(tempFile).name>

<cftry>
<cffeed action="read" source="http://movies.msn.com/rss/topcelebs" name="feedInStruct" >
<cfcatch any>
<cfhttp url="http://movies.msn.com/rss/topcelebs" path="#tempDir#" file="#tempFileName#">
<cfhttpparam type="header" name="Accept-Encoding" value="compress,gzip,deflate">
</cfhttp>
<cffeed action="read" source="#tempFile#" name="feedInStruct" >
</cfcatch>
</cftry>
<cfif FileExists(tempFile)>
<cfset FileDelete(tempFile)>
</cfif>

<cfdump var="#feedInStruct#">

Thursday, August 02, 2007

ColdFusion 8 : IsInstanceOf

If you use lot of CFC inside your ColdFusion application, I am sure you would have come across a situation where you would need to know whether the object is an instance of a particular CFC. This is specially needed when you have components extending other component or you are passing the objects around. ColdFusion 8 introduces a new function IsInstanceOf to do exactly the same. It becomes even more useful after we have interfaces in ColdFusion. And the icing on the cake is that it works even with java objects which means that you can use this function to find out if a particular object is of a particular java class type.

Here is how the function looks.

IsInstanceOf(object, typeName)

where typeName is name of the component/Interface or fully qualified java class name.

It returns 'true' if

  • The object passed is an instance of a component which is same as specified type or inherits it or implements the specified interface. Just to be clear, a component 'A' inherits a component 'B' if A or any of its super component extends 'B'. Similarly a Component 'A' implements an interface 'B' if A or any of its super component, implements interface 'B' or any of the interface that 'A' or its parents implement, extends from the specified interface.
  • The object passed is an instance of a java class (created using cfobject or createObject for java class) which is same as specified class name or inherits the specified class name or implements the specified interface.

Here is an example

Intf.cfc
<cfinterface>
<cffunction name = "foo">
</cffunction>
</cfinterface>

Comp.cfc

<cfcomponent implements="Intf">
<cffunction name = "foo">
<cfoutput>In method foo</cfoutput>
</cffunction>
</cfcomponent>

test.cfm

<cfset obj = CreateObject("Component", "Comp")>
<!--- Create a Java object --->
<cfset javaObj = CreateObject("java", "java.lang.StringBuffer")>

<cfoutput>object is of type Comp : #IsInstanceOf(obj, "Comp")#</cfoutput><br>
<cfoutput>object is of type Intf : #IsInstanceOf(obj, "Intf")#</cfoutput><br>

<cfoutput>java object is of type String : #IsInstanceOf(javaobj, "java.lang.String")#</cfoutput><br>
<cfoutput>java object is of type StringBuffer : #IsInstanceOf(javaobj, "java.lang.StringBuffer")#</cfoutput><br>

Wednesday, August 01, 2007

ColdFusion 8 performance whitepaper

I am glad to see the ColdFusion community so excited with CF 8 performance. We got very encouraging response regarding the performance in our pre-release forums and we are thankful to everyone who gave such invaluable feedbacks. Though people started seeing a very good performance gain in their application on ColdFusion 8, we never talked about the performance benchmark numbers until CFUnited 2007, where Ben showed the performance numbers for ColdFusion 8. No wonder this is the #1reason in Top 8 reasons why you want ColdFusion 8 :-).

Extracting performance up to this extent was not an easy task. We analyzed nearly 2.4 million lines of real life CF application code, to zero-in on the most commonly used tags and functions. Main challenge was after that - analyze the generated java code for all the tags, change the compiler to generate more optimized code, run it through profiler and optimize CF engine for each of the tags and functions and their various combinations. And this went on and on in many many iterations. Overall it was real fun :-)

I still remember the most exciting moment when I ran the load test for CFC, after I had made some code changes, and the result was freaking unbelievable. It looked too good to be true and I was literally running around with the code changes to run it on other machines and verify it . That one small change gave nearly 6x gain :-). It is not that the CF7 code was inefficient or poorly written. It was only matter of extracting juices as much as possible and putting some smart intelligence.

Check out the ColdFusion 8 performance whitepaper which talks in much detail about the performance numbers for different areas, the methodology used for benchmarking etc. Manju Kiran, who was my QA-buddy for most of the features I worked on, did a tremendous job in setting up and running the benchmark and creating the meat of this wonderful document (and of course keeping me on toes).