Till ColdFusion 7, it was not possible to upload a large file. The server could not handle uploading a file of probably anything more than 300-400 MB. And on top of that, if there are multiple people trying to upload large files simultaneously to the server, only thing that you can do is to pray :-)
You might wonder and I agree with you, that its a huge limitation. So why was it this way ? Well.. that was because the server had to keep the entire uploaded file in the memory. Whoa.. Why the heck does the server need to keep the entire uploaded file in memory? - For a single reason of serving the function GetHttpRequestData(). As you know, this function returns a 'struct' containing the metadata for the Http request that comes to the server. Since this struct also includes request body, and the uploaded file is part of the request, the file has to be retained in the memory in order to honor this function.
With ColdFusion 8, we have removed the limitation on file upload size. So now you can upload files of any size without causing any issue with the server. I have successfully tried with 5 GB file because that was the biggest I could get hold of but you can absolutely upload files of any size. ColdFusion nicely handles it, reads it the way it should without requiring much memory.
But hey, hold on a sec.. What about the function GetHttpRequestData()? Wasn't that the reason for this limitation in the first place? Well, the function GetHttpRequestData() works perfectly fine except in this one case - In the multipart request, i.e; request in which a file is uploaded - "content" in the returned struct will be empty. If you need to get the uploaded file content in memory, you can always read the uploaded file and keep it in memory. So that should not be much we are asking for. Are we?
If in case, you absolutely want the content from GetHttpRequestData() and file upload size is of no concern to you, we allow you to revert back to ColdFusion 7 behaviour. All you need to do for this is to set the Java system property "coldfusion.markResetForMultiPart" to "true" and you are back to the old behavior.
39 comments:
We are wanting to implement an upload progress bar using Ajax and CF alone.
One limitation with CF7's uploading is that it is not possible to determine the progress of an upload as we don't know its file size until the upload is complete.
Will this be possible with CF8?
We have not made any other change in file upload. Which means that CF8 by itself does not provide any upload progress functionality. But there might be some other solutions possible. Have you looked at the possibility of doing it using flex/CF?
What if we are using ColdFusion 7 (and can't get 8 yet) and still need to upload larger files? Is there anything we can do to make it work?
Thanks!
Hi Rupesh,
I am working with the new CF 8 cffile action = "upload" and seem to have a problem. I am bumped up the settings in CF admin to allow up to 1.5 gig to be uploaded. When the upload exceeds 1.5 I would like to be able to use a CF try and Catch to give a user friendly error message. Right Now I just get a 500 java thread dump. Do you have any insight on this?
Thanks,
Joe
PS. Thanks again for answering our questions at Adobe Max in Chicago.
Hi Joe,
the way cffile upload works is - When you post the form containing file field, the file is uploaded to the server and kept at a temp location. Then the request goes to the cfm to which the form was posted and when in that cfm, you do cffile action="upload", server merely makes the uploaded file available to you.
So, if the size of the file to be uploaded is more than post size limit, then server will not even allow uploading it. So the request is rejected right there and much before cfm gets the chance.
There are two ways in which you can handle this -
1. Make the post size limit very high so that the server does not reject request. Let the request come to the cfm and then you check the file size and throw an error.
2. Let the post size be there in the server and set an error page for this in the web.xml. Thats because your site error handler and application error handler will also not get a chance to handle it.
Rupesh.
We've had serious issues with CF 7.0 and hoped they would vanish with 8.0, but alas.
On Windows 2003 servers running IIS 6, if the file size exceeds around 50K, the client gets a white screen. No errors, no indication of anything, just a plain white screen. Doesn't happen with Windows 2000 at all. Never. Our first test of a 1.5 meg file was successful, but following up with a 1.0 meg file resulted in the white screen again.
Any ideas where to look? This happens on a moderately busy web server, but also occurs on a test server on which few, if any, users appear.
Has anyone heard of this? Any ideas?
I should have mentioned, that the tests in my earlier posts (1.5 meg, then 1.0 meg), were on our new installation of CF 8.0. I meant to make that clear.
John,
Just to be sure, are you saying that upload works fine on windows 2000 but fails on 2003 with IIS?
This should not have any relation with Windows 2003 or IIS. There must be something else that is playing the part here. Could you check that? Is there any IIS setting for this? Are there any exceptions in the logs? You can mail me offline and we can take it up there.
Thanks, Rupesh,
Yes, the issue ONLY occurs on our (multiple) Windows 2003 Server installations. We've had to retain a few servers running Windows 2000 because they don't exhibit this behavior.
I would email you offline, but haven't been able to located your email address.
Thanks for looking into this!
John H.
John, you could mail me at rukumar at adobe dot com
Has anyone found a workaround for the upload issue on II6 and MX7 or higher. Since we upgraded from CF5/MS2000 server we have had upload problems. We have found that running SSL with Client Certificates enabled will cause the uploads to fail with a blank page or a "page cannot be found" error. We have changed all html tages to cf tags. We have changed the CF and IIS settings to allow larger files and larger timeout times. We have even tried an ASP page with HTML tags only and get the same results... Uploads work fine if client certificates are disabled but we HAVE to run in this mode. Thanks for any help in advance.
Thank you, Kim, for putting the issue so clearly and succinctly! This is the EXACT same issue we've been having and, as Kim and I have said, it doesn't occur with Windows Server 2000.
Rupesh, if you create a setup as Kim has described and are able to duplicate the issue, there may be hope for us yet! As it is, we're holding off on completing our long-planned upgrades to CF 8 and Windows Server 2003 in our production environment until a solution can be found.
We hope you can get to the bottom of this!
Just adding my bit to it, we also have an identical problem. 2000 server worked fine but have moved our system to 2003 server and IIS6 running in iis5 isolation mode and all uploads fail even the small ones. Thought initally it was to do with the write, create and modify rights for the iusr anon logon but these are all set correctly. Just can't see whta the issue is here and could do with some help
Bryn
@Anon - Running in IIS 5 will not work either. The problem is the "form" tag. CF rewrote the tag. The tag has a conflict when running in SSL mode with or without client certificates. You have to change all your code to use "cfform" tags and fields only. Then you can upload large files as long as your running in a non-SSL/https mode.
If you are running in SSL/https mode, you will still experience getting the "blank pages" and "this page can not be found" errors. Sometimes you can refresh the page, clear your cookies and delete your temporary files and the upload will work but not for long. CF has issued it as a BUG. No one has found a work around yet.
wow.. Looks likes lot of people are facing this. But on the face of it, doesn't it look like Windows 2003/IIS problem. If it works on Windows 2000/IIS, it should work on 2003 as well because ColdFusion is running on Java. And as Kim said, he sees the same behaviour with ASP also. So the bug should be more likely on 2003/IIS.
Anyways, we are trying to replicate it at our end here and see what exactly is going on. Will update you guys with our findings.
Hi, Rupesh,
Any luck replicating SSL/Blank Page issue? We're eager to move to CF 8 on Windows 2003.
Thanks!
John H.
Hi John,
Yes, we were able to replicate this. But the interesting thing is that - when we debugged, we found out that request in this case does not even come to ColdFusion. The culprit here is IIS. It sends a blank page or 413 response very often in case of upload over SSL. And it looks like lot of people are facing problem with IIS6 + SSL.
Some links
http://www.issociate.de/board/goto/617935/IIS6.0_+_SSL_Breaks_down!.html
http://www.thescripts.com/forum/thread317125.html
http://www.elp.co.at/IIS+Issues_1199.html
http://www.experts-exchange.com/Software/Server_Software/Web_Servers/Microsoft_IIS/Q_20637561.html
http://thedotnet.com/nntp/69797/showpost.aspx
Let me know if any of these links help.
We still haven't found a workaround. We tried all those IIS changes before and we still get the error when SSL is enabled. We are mandated to use SSL and PKI certificates. We can't go back to IIS5/CF5 because it is not supported by our customer.
We seem to have had some success. We configured those settings in our test environment with CF8 and so far uploads have been successful with no blank pages. Our production servers running CF7 still have the problem, as one might expect.
The Production servers will be upgraded to CF8 on Monday and will be tested by our developers Wed/Thurs of next week. I'll report back on their findings on the 31st. (I'll do a couple of tests myself, but they'll have the final word as to whether we're successful or not.)
On an unrelated note--we need to enable SSL on JRun, but the Admin guide for CF8 says it must be set to "false" in JRun.ini. Is there a workaround to this or is that the final word?
Hi John,
Thats a great news! It will be great if you could share which exact configuration setting helped?
Regarding ssl configuration, which communication do you want on ssl - between client and web server or between webserver and JRun or both? The doc just says that since jrun is any way not exposed to external world and only webserver is, you would typically not configure SSL on jrun and hence would keep ssl flag as false in jrun.ini. If you want the communication between IIS and jrun over SSL, you can enable SSL without any problem.
Thanks & Regards,
Rupesh.
Rupesh
Hi, Rupesh,
Sorry for the long delay... Between the holidays and fires that needed to be put out, I haven't had an opportunity to post a reply.
1. Blank Page Issue: Though we'd tried the UploadReadAheadSize setting in the IIS metadata in the past, we'd never tried it in conjunction with the SSLAlwaysNegoClientCert property. That combination seems to work though there still have been one or two instances of blank pages. (We're not sure that they weren't caused by something else.) I wouldn't say we've fixed it 100% until further testing, but at the moment we've gone from about 15% success rate to something closer to 98%. We'll know in the coming weeks if those are, indeed, the fixes, but it's looking good so far.
2. JRun SSL Issue: The reason we need to enable SSL for ColdFusion is that our CF server makes machine-to-machine connections to a WebLogic machine and we need two way SSL (per security regs) to insure the integrity of the communidations. (The WebLogic server is ours now, but could be a "foreign" machine in the future.) We are using the built in version of JRun, and that is configured with the .ini file, as opposed to using the stand-alone JRun that has a GUI as I understand it, that simply lets you check a box to enable SSL. At any rate, we've tried all matter of configurations and either it (JRun) wouldn't start after enabling it, or would start but throw thousands of errors. We've given up for the time being, but it will be mandatory to have the two-way SSL handshake between the WebLogic and CF serves before it will ever go live in production. One subcontrator who was here working on the WebLogic end of things commented that the version of JRun/Java can't handle two-way SSL, only one-way. So, it's never been as simple as "just enabling" it for us. What I'd love is to have the .INI file of someone who's got two-way SSL working that I could use as a template when trying to sort out exactly which directory/file/etc. that needs to be configured in the .ini.
I'll keep you posted about our testing with the blank page issue and we'll keep looking (and hoping) a solution can be found for two-way SSL between ColdFusion and WebLogic.
Thanks again for your help!
John H.
Re: Blank/White Page Issue
It came back again this morning after a few weeks of error-free uploads. Once the issue arose, we tested documents of 49k, 52k, and 101k and all failed. We've scheduled a reboot of the affected server for this evening and will retest again tomorrow (Tues.). I'll let you know if that makes any difference.
John H.
Re: Blank/White Page Issue
The problem continues even after a reboot and no known configuration changes. We're back to missing more than we're hitting, unfortunately. Does anyone have any other suggestions for us to try?
We're mystified!
John H.
Thanks for making this blog post. It's saved me looking in the wrong place for something. I've been trying to hunt down where ColdFusion holds the original name of a file upload, before the CFFile action="upload" tag is used. Tried the getHttpRequestData().content but of course it's empty on CF8 now plus I didn't really want to split the data apart for the name.
I'll keep hunting ;)
Thanks
Unfortunately, we're experiencing the same problem on our server running CF8, MS Server 2003 R2, IIS6, and SSL. I've made both metabase changes and restarted IIS as well as the server with no luck.
We worked with Microsoft to solve this problem. To summarize what we did to get this issue resolved....
We set SSLALwaysNegoClientCert property to true. We had to do this with the adsutil not metabase explorer. The readaheadsize needs to stay as the default values. We have not had any upload problems for a couple weeks and we have uploaded files as large as 100mg.
Good post... Thanks.
Kim,
Could you point us to the syntax of making that change using Adsutil? We'd like to give that a try.
Thanks for your post!
John H.
We had the same upload problems with IIS 6 and Cf8.01 until this fix that my project lead implemented based on the posts above.
The SSLAlwaysNegoClientCert="TRUE" had to be set in a file called MetaBase.xml found in the directory C:\WINDOWS\system32\inetsrv
In order to modify this file you have to shutdown the IIS services and web sites.
Then modify the file by adding the SSLAlwaysNegoClientCert="TRUE" statement to the xml file in the section IIsWebService.
There was another SSL statement there already so he put it above that statement as he thought it needed to be in order.
Once the modifications were complete, then he restarted the services and web sites.
Hi Rupesh,
When you mentioned uploaded 5GB file through ColdFusion 8, may I know what is the setting you used for "Maximum size of post data" and "Request Throttle Memory"? What is your total physical memory size in your machine?
Thanks!
Thanks a lot everyone specially John H., Kim and Brian for finding out a solution for this crazy IIS problem.
Chee,
In ColdFusion 8, since file upload will not keep it in the memory, you dont need a higher heap size.
Regarding other settings, you can keep all of these to be very high on your server so that file upload does not go through the throttle at all. In my local setting, I had everything to be 20 GB.
Rupesh, thanks for the response. In my ColdFusion 8 standard edition, I cannot put any number greater than 2GB in "Maximum size of post data" or else I will receive this error:
"error Posted content length of 95585 exceeds limit of -2147483648".
So, the maximum value I can put in is only 2047 MB. For the "Request Throttle Memory", if I put in more than 2047 MB, I will receive this error:
"javax.servlet.ServletException: ROOT CAUSE:
coldfusion.util.MemorySemaphore$MemoryUnavailableException: Memory required (925170494 bytes) exceeds the maximum allowed memory."
FYI, I have 2GB of physical RAM on the server. Have you seen the above errors before?
There appears to be a problem with the upload limits on a mutli-sever install. I've tried CF8.1 on WinXP/IIS6 (32-bit) and Ubuntu 64-bit running off localhost:8300 and get the same 500 error if any of the request limits are set above 2048. I have enough physical RAM in both cases so I know it's not that. I sumitted this to Jason at Adobe and he said the engineer(s) confirm the problem. I suspect it is just a multi-server JRE issue, but waiting for final confirmation. If anyone knows of a work-around please post it! Btw, my file upload form is plain vanilla (no flash, no ajax).
Yes. It is indeed a bug - thanks to Chee and others for reporting :). We will release a fix for this shortly. I had my setting as 10GB with which everything was working. So for the time being, you can use that as work around till you get the actual fix.
Rupesh, is there a 2Gb upload limit enforced at the browser level (regardless of whatever bug there is in CF)? Are you saying in the posts above that you are able to upload 5Gb using regular HTTP protocol in Firefox or IE?
I've attempted to upload more than 2Gb using other web application servers and still get the same result (Linux 64-bit & Windows 32-bit), so based on this and what I've read on some forums, maybe the 2Gb limit is in the browser?
Btw, I am not even getting to cffile. I just want to see a file larger than 2Gb get into CF's tmp directory! And this also has nothing to do with SSL.
I tried your suggestion of setting the request limits to 10Gb but it does not help; same 400/500 error occurs (depending on OS). I am very curious now what your local set up is that all this works for you.
Increasing the file upload size in the machine.config file is what ultimately fixed this problem for us.
It's in the link http://thedotnet.com/nntp/69797/showpost.aspx as listed above.
@Rupesh: FYI: you have a problem with SEO Link Spamming in this post's comments... (and probably others).
You may want to remove the Japanese comments with all the links in them...
It's a tricky thing to beat... and it's sad that someone out there is putting that junk on such a wonderfully useful blog post.
Keep doing what you do :D
Hi Rupesh,
I've confirmed that "-Dcoldfusion.markResetForMultiPart=true" did populate getHTTPRequestData().content w/ binary multipart form data in CF8. However, I've also confirmed that CF9 and CF10 ignore that JVM argument. Thus, this CF7 backward-compat only existed for CF8.
Thanks,
-Aaron
Post a Comment