PlanetJava
Custom Search

java-resteasy-user
[Top] [All Lists]

Re: FileUpload with big files failing withOutOfMemory Error

Subject: Re: FileUpload with big files failing withOutOfMemory Error
Date: Fri, 8 Jun 2012 11:23:59 -0500
Thanks for the heads-up on plans to deprecate Apache Client, will help
me plan the future releases of my project better. My motivation to go
with Apache was because of its MutiThreaded Http Connection Pool support
and its support for external memory cache like memcached/Ehcache.
LightweightBrowserCache documentation seemed to indicate that it gets
wiped out after the cache gets full, in my usage scenario that was going
to happen too quickly and too often.
-----Original Message-----
From: Bill Burke [mailto:[email protected]] 
Sent: Friday, June 08, 2012 10:54 AM
To: Sandeep Tikoo
Cc: [email protected]
Subject: Re: [Resteasy-users] FileUpload with big files failing
withOutOfMemory Error
FYI, there's a good chance I'm going to be deprecating Apache Client 4
in Resteasy 3.0, then removing it in 3.1.  We'll be switching to our own
custom code.  Particularly because of some of the headers issues I
discussed before.
Also, we do have a Resteasy client cache that you might want to take a
look at.
On 6/8/12 11:12 AM, Sandeep Tikoo wrote:
> Hi,
>
> I cannot use URLConnectionClientExecutor because I need to use 
> Apache's Http Caching component.
> Not sure if I understood the issue with Apache Client's RequestEntity 
> and how that forces having an in-memory Stream but I would appreciate 
> it very much if you could have a look at the attached files and let me
> know if I could run into any issues down the line.
> I am at the moment extending ApacheHttpClient4Executor.java with my 
> custom class; however, the attached file
> (ApacheHttpClient4Executor_2.1.1.GA_PATCHED.java) presents the fix on 
> top of the original file to facilitate easier comparison. The 
> patch.txt file is generated using WinMerge.
>
> Thank you,
> Sandeep.
>
>
> -----Original Message-----
> From: Bill Burke [mailto:[email protected]]
> Sent: Thursday, June 07, 2012 3:23 PM
> To: [email protected]
> Subject: Re: [Resteasy-users] FileUpload with big files failing 
> withOutOfMemory Error
>
> Yes please elaborate on your fix.
>
> FYI, the problem I had with Apache Client 3/4 RequestEntity was that 
> you couldn't add/modify headers within the RequestEntity
implementation.
> MessageBodyWriters are allowed to add/modify headers before they write
> to the output stream.  So, in order to fulfill this requirement on top
> of Apache Client, the entity needs to be buffered.
>
> I don't think you would see this same OOM problem if you used the 
> URLConnectionClientExecutor because it doesn't have the same silly 
> constraint as AC 3/4.
>
> On 6/7/12 2:54 PM, Sandeep Tikoo wrote:
>> Realized that https://issues.jboss.org/browse/RESTEASY-233 fixed it 
>> on
>
>> the server side only and not on the client-side. I have, for now, to 
>> keep my project moving forward, applied a fix on top of 
>> org.jboss.resteasy.client.core.executors.ApacheHttpClient4Executor
>> using apache common's DefferedFileOutputStream
>>
> (http://commons.apache.org/io/apidocs/org/apache/commons/io/output/Def
> er
> redFileOutputStream.html).
>>
>>
>> If there is interest, I would be more than happy to share the fix.
>>
>> Thank you,
>>
>> Sandeep.
>>
>> *From:*Sandeep Tikoo [mailto:stikoo-QZfty/[email protected]]
>> *Sent:* Tuesday, May 15, 2012 6:31 PM
>> *To:* [email protected]
>> *Subject:* Re: [Resteasy-users] FileUpload with big files failing 
>> withOutOfMemory Error
>>
>> Hi,
>>
>> I am using RestEasy on the client side (version 2.2.1.GA) and trying 
>> to upload a huge file and I get a OutOfMemory Error:
>>
>> *//*
>>
>> */Caused by: java.lang.OutOfMemoryError: Java heap space/*
>>
>> */at java.util.Arrays.copyOf(Arrays.java:2786)/*
>>
>> */at
>> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:71)/*
>>
>> */at
>> org.jboss.resteasy.plugins.providers.multipart.HeaderFlushedOutputStr
>> e
>> am.write(HeaderFlushedOutputStream.java:56)/*
>>
>> */at
>> org.jboss.resteasy.plugins.providers.InputStreamProvider.writeTo(Inpu
>> t
>> StreamProvider.java:60)/*
>>
>> */at
>> org.jboss.resteasy.plugins.providers.InputStreamProvider.writeTo(Inpu
>> t
>> StreamProvider.java:22)/*
>>
>> */at
>> org.jboss.resteasy.plugins.providers.multipart.AbstractMultipartWrite
>> r
>> .writePart(AbstractMultipartWriter.java:62)/*
>>
>> */at
>> org.jboss.resteasy.plugins.providers.multipart.AbstractMultipartFormD
>> a
>> taWriter.writeParts(AbstractMultipartFormDataWriter.java:32)/*
>>
>> */at
>> org.jboss.resteasy.plugins.providers.multipart.AbstractMultipartWrite
>> r
>> .write(AbstractMultipartWriter.java:33)/*
>>
>> */at
>> org.jboss.resteasy.plugins.providers.multipart.MultipartFormAnnotatio
>> n
>> Writer.writeTo(MultipartFormAnnotationWriter.java:78)/*
>>
>> */at
>> org.jboss.resteasy.core.interception.MessageBodyWriterContextImpl.pro
>> c
>> eed(MessageBodyWriterContextImpl.java:117)/*
>>
>> */at
>> org.jboss.resteasy.plugins.interceptors.encoding.GZIPEncodingIntercep
>> t
>> or.write(GZIPEncodingInterceptor.java:98)/*
>>
>> */at
>> org.jboss.resteasy.core.interception.MessageBodyWriterContextImpl.pro
>> c
>> eed(MessageBodyWriterContextImpl.java:123)/*
>>
>> */at
>> org.jboss.resteasy.client.ClientRequest.writeRequestBody(ClientReques
>> t
>> .java:472)/*
>>
>> *//*
>>
>> *//*
>>
>> *//*
>>
>> It seems that this issue was addressed a long time ago using mime4j:
>> https://issues.jboss.org/browse/RESTEASY-233
>>
>> I do have mime4j in my classpath but not sure if there is something I
>> need to do differently to make this work?
>>
>> Here is my code:
>>
>> The JAX-RS annotated interface:
>>
>> @Path("/resource")
>>
>> *public**interface*RestClientAPI {
>>
>> @POST
>>
>> @Path(/UPDATE_PRODUCTS_ASYNC_RESOURCE/)
>>
>> //@Consumes(MediaType.APPLICATION_XML)
>> <mailto://@Consumes(MediaType.APPLICATION_XML)>
>>
>> @Consumes(MediaType./MULTIPART_FORM_DATA/)
>>
>> @Produces(MediaType./APPLICATION_XML/)
>>
>> Response upload(@MultipartForm*final*LoaderMultiPartForm loader);
>>
>> }
>>
>> *The MultiPartForm class:*
>>
>> *public**class*LoaderMultiPartForm {
>>
>> @FormParam("loader")
>>
>> @PartType(MediaType./APPLICATION_XML/)
>>
>> *private*Loader loader;
>>
>> @FormParam("bulkProductTypeStream")
>>
>> @PartType(MediaType./APPLICATION_OCTET_STREAM/)
>>
>> *private*InputStream bulkProductTypeStream;
>>
>> *public*Loader getLoader() {
>>
>> *return**this*.loader;
>>
>> }
>>
>> *public**void*setLoader(Loader loader) {
>>
>> *this*.loader= loader;
>>
>> }
>>
>> *public*InputStream getBulkProductTypeStream() {
>>
>> *return**this*.bulkProductTypeStream;
>>
>> }
>>
>> *public**void*setBulkProductTypeStream(InputStream
>> bulkProductTypeStream) {
>>
>> *this*.bulkProductTypeStream= bulkProductTypeStream;
>>
>> }
>>
>> }
>>
>> The Call:
>>
>> RestClientAPI client = factory.getClient(CatalogResource.*class*);
>>
>> Loader loader = *new*Loader();
>>
>> loader.setXXX();
>>
>> LoaderMultiPartForm loaderForm = *new*LoaderMultiPartForm();
>>
>> loaderForm.setLoader(loader);
>>
>> hugeFileUploadStream =
>> *new*BufferedInputStream(*new*FileInputStream(*new*File("C:\\tmp\\hug
>> e
>> FileToUpload.txt")));
>>
>> loaderForm.setBulkProductTypeStream(hugeFileUploadStream);
>>
>> client.upload(loaderForm);
>>
>> Appreciate any help.
>>
>> Regards,
>>
>> Sandeep.
>>
>>
>>
>> ---------------------------------------------------------------------
>> -
>> --------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and 
>> threat landscape has changed and how IT managers can respond.
>> Discussions will include endpoint security, mobile security and the 
>> latest in malware threats.
>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>
>>
>>
>> _______________________________________________
>> Resteasy-users mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/resteasy-users
>
> --
> Bill Burke
> JBoss, a division of Red Hat
> http://bill.burkecentral.com
>
>
> ----------------------------------------------------------------------
> --
> ------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond.
> Discussions
> will include endpoint security, mobile security and the latest in 
> malware threats. 
> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Resteasy-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/resteasy-users
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
msgmiddle
<Prev in Thread] Current Thread [Next in Thread>
Current Sitemap | © 2012 planetjava | Contact | Privacy Policy