PlanetJava
Custom Search

java-resteasy-user
[Top] [All Lists]

Re: FileUpload with big files failing withOutOfMemory Error

Subject: Re: FileUpload with big files failing withOutOfMemory Error
Date: Mon, 11 Jun 2012 10:26:38 -0500
I agree with Joe, my experience has been the same. When I was researching 
different REST frameworks to select one for my organization, I found RESTEASY 
to be the best for JAX-RS because of the rich feature which it also extended 
out to the client-side, and, like I said earlier, Apache Component library was 
very attractive so as far as native HTTP protocol implementation went (HTTP 
Connection Pools and Connection management, HTTP Cache with distributed cache 
support). 
The most attractive thing about RestEasy is/was that it allowed seamless 
integration with Apache Component (and Apache Commons) library - something I 
found lacking in Jersey, Apache CXF.
BTW, Bill, when you get a chance, please let me know if you see any issues with 
code I sent earlier. I would rather know now than find out the hard way after 
the code goes live next month :). Appreciate your time.
Thanks,
Sandeep. 
-----Original Message-----
From: Joe Littlejohn [mailto:[email protected]] 
Sent: Monday, June 11, 2012 5:45 AM
To: Bill Burke
Cc: Sandeep Tikoo; [email protected]
Subject: Re: [Resteasy-users] FileUpload with big files failing withOutOfMemory 
Error
Bill,
Yes, I tend to see this as an issues/help list, which is probably a bad thing. 
I should be more forthcoming with feedback and suggestions too.
Regarding feedback for LightweightBrowserCache - I think my preference here 
would be to stick with hc4 and httpclient-cache. The cache implementation is 
comprehensive, it supports the cache control extensions (as mentioned) and even 
offers multiple storage options (including persistent backends based on ehcache 
and memcached, which are essential for some use-cases). I'm not suggesting 
you'd need provide something this flexible in RESTEasy, but I worry that 
there's actually a lot of work here and it may not be that valuable to repeat 
it.
I read your recent blog post on logging and I applaud your efforts to favour 
simplicity over obsessive 're-use'. I've had to resolve enough hellish Maven 
dependency problems. I just know how useful these caching features have been 
for us. If they can be provided by a custom RESTEasy implementation then that 
would be great, but also I'm happy to stick hc4 and see the effort spent 
elsewhere.
On 8 June 2012 17:39, Bill Burke <[email protected]> wrote:
> Joe, the problem is, I don't get valuable feedback like you provided 
> below on things like LightweightBrowserCache. ÂSo, it doesn't get improved 
> upon.
>
>
> On 6/8/12 12:19 PM, Joe Littlejohn wrote:
>>
>> Bill, will Resteasy 3.0 include an improved LightweightBrowserCache?
>>
>> The CachingHttpClient provided by the Apache Client 4 is far more 
>> sophisticated. It provides a number of storage options, supports 
>> cache-control extensions (RFC5861) and even when using the 
>> BasicHttpCacheStorage you get a much better eviction policy than the 
>> LightweightBrowserCache.
>>
>> I think it'll be hard to live without Apache Client 4 unless the 
>> built-in caching options in Resteasy are dramatically improved.
>>
>>
>> On 8 June 2012 16:54, Bill Burke<[email protected]> Âwrote:
>>>
>>> FYI, there's a good chance I'm going to be deprecating Apache Client 
>>> 4 in Resteasy 3.0, then removing it in 3.1. ÂWe'll be switching to 
>>> our own custom code. ÂParticularly because of some of the headers 
>>> issues I discussed before.
>>>
>>> Also, we do have a Resteasy client cache that you might want to take 
>>> a look at.
>>>
>>> On 6/8/12 11:12 AM, Sandeep Tikoo wrote:
>>>>
>>>> Hi,
>>>>
>>>> I cannot use URLConnectionClientExecutor because I need to use 
>>>> Apache's Http Caching component.
>>>> Not sure if I understood the issue with Apache Client's 
>>>> RequestEntity and how that forces having an in-memory Stream but I 
>>>> would appreciate it very much if you could have a look at the 
>>>> attached files and let me know if I could run into any issues down the 
>>>> line.
>>>> I am at the moment extending ApacheHttpClient4Executor.java with my 
>>>> custom class; however, the attached file
>>>> (ApacheHttpClient4Executor_2.1.1.GA_PATCHED.java) presents the fix 
>>>> on top of the original file to facilitate easier comparison. The 
>>>> patch.txt file is generated using WinMerge.
>>>>
>>>> Thank you,
>>>> Sandeep.
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Bill Burke [mailto:[email protected]]
>>>> Sent: Thursday, June 07, 2012 3:23 PM
>>>> To: [email protected]
>>>> Subject: Re: [Resteasy-users] FileUpload with big files failing 
>>>> withOutOfMemory Error
>>>>
>>>> Yes please elaborate on your fix.
>>>>
>>>> FYI, the problem I had with Apache Client 3/4 RequestEntity was 
>>>> that you couldn't add/modify headers within the RequestEntity 
>>>> implementation.
>>>> MessageBodyWriters are allowed to add/modify headers before they 
>>>> write to the output stream. ÂSo, in order to fulfill this 
>>>> requirement on top of Apache Client, the entity needs to be buffered.
>>>>
>>>> I don't think you would see this same OOM problem if you used the 
>>>> URLConnectionClientExecutor because it doesn't have the same silly 
>>>> constraint as AC 3/4.
>>>>
>>>> On 6/7/12 2:54 PM, Sandeep Tikoo wrote:
>>>>>
>>>>> Realized that https://issues.jboss.org/browse/RESTEASY-233 fixed 
>>>>> it on
>>>>
>>>>
>>>>> the server side only and not on the client-side. I have, for now, 
>>>>> to keep my project moving forward, applied a fix on top of 
>>>>> org.jboss.resteasy.client.core.executors.ApacheHttpClient4Executor
>>>>> using apache common's DefferedFileOutputStream
>>>>>
>>>> (http://commons.apache.org/io/apidocs/org/apache/commons/io/output/
>>>> Defer
>>>> redFileOutputStream.html).
>>>>>
>>>>>
>>>>>
>>>>> If there is interest, I would be more than happy to share the fix.
>>>>>
>>>>> Thank you,
>>>>>
>>>>> Sandeep.
>>>>>
>>>>> *From:*Sandeep Tikoo [mailto:[email protected]]
>>>>> *Sent:* Tuesday, May 15, 2012 6:31 PM
>>>>> *To:* [email protected]
>>>>> *Subject:* Re: [Resteasy-users] FileUpload with big files failing 
>>>>> withOutOfMemory Error
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am using RestEasy on the client side (version 2.2.1.GA) and 
>>>>> trying to upload a huge file and I get a OutOfMemory Error:
>>>>>
>>>>> *//*
>>>>>
>>>>> */Caused by: java.lang.OutOfMemoryError: Java heap space/*
>>>>>
>>>>> */at java.util.Arrays.copyOf(Arrays.java:2786)/*
>>>>>
>>>>> */at
>>>>> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:71)
>>>>> /*
>>>>>
>>>>> */at
>>>>> org.jboss.resteasy.plugins.providers.multipart.HeaderFlushedOutput
>>>>> Stre
>>>>> am.write(HeaderFlushedOutputStream.java:56)/*
>>>>>
>>>>> */at
>>>>> org.jboss.resteasy.plugins.providers.InputStreamProvider.writeTo(I
>>>>> nput
>>>>> StreamProvider.java:60)/*
>>>>>
>>>>> */at
>>>>> org.jboss.resteasy.plugins.providers.InputStreamProvider.writeTo(I
>>>>> nput
>>>>> StreamProvider.java:22)/*
>>>>>
>>>>> */at
>>>>> org.jboss.resteasy.plugins.providers.multipart.AbstractMultipartWr
>>>>> iter
>>>>> .writePart(AbstractMultipartWriter.java:62)/*
>>>>>
>>>>> */at
>>>>> org.jboss.resteasy.plugins.providers.multipart.AbstractMultipartFo
>>>>> rmDa
>>>>> taWriter.writeParts(AbstractMultipartFormDataWriter.java:32)/*
>>>>>
>>>>> */at
>>>>> org.jboss.resteasy.plugins.providers.multipart.AbstractMultipartWr
>>>>> iter
>>>>> .write(AbstractMultipartWriter.java:33)/*
>>>>>
>>>>> */at
>>>>> org.jboss.resteasy.plugins.providers.multipart.MultipartFormAnnota
>>>>> tion
>>>>> Writer.writeTo(MultipartFormAnnotationWriter.java:78)/*
>>>>>
>>>>> */at
>>>>> org.jboss.resteasy.core.interception.MessageBodyWriterContextImpl.
>>>>> proc
>>>>> eed(MessageBodyWriterContextImpl.java:117)/*
>>>>>
>>>>> */at
>>>>> org.jboss.resteasy.plugins.interceptors.encoding.GZIPEncodingInter
>>>>> cept
>>>>> or.write(GZIPEncodingInterceptor.java:98)/*
>>>>>
>>>>> */at
>>>>> org.jboss.resteasy.core.interception.MessageBodyWriterContextImpl.
>>>>> proc
>>>>> eed(MessageBodyWriterContextImpl.java:123)/*
>>>>>
>>>>> */at
>>>>> org.jboss.resteasy.client.ClientRequest.writeRequestBody(ClientReq
>>>>> uest
>>>>> .java:472)/*
>>>>>
>>>>> *//*
>>>>>
>>>>> *//*
>>>>>
>>>>> *//*
>>>>>
>>>>> It seems that this issue was addressed a long time ago using mime4j:
>>>>> https://issues.jboss.org/browse/RESTEASY-233
>>>>>
>>>>> I do have mime4j in my classpath but not sure if there is 
>>>>> something I need to do differently to make this work?
>>>>>
>>>>> Here is my code:
>>>>>
>>>>> The JAX-RS annotated interface:
>>>>>
>>>>> @Path("/resource")
>>>>>
>>>>> *public**interface*RestClientAPI {
>>>>>
>>>>> @POST
>>>>>
>>>>> @Path(/UPDATE_PRODUCTS_ASYNC_RESOURCE/)
>>>>>
>>>>> //@Consumes(MediaType.APPLICATION_XML)
>>>>> <mailto://@Consumes(MediaType.APPLICATION_XML)>
>>>>>
>>>>> @Consumes(MediaType./MULTIPART_FORM_DATA/)
>>>>>
>>>>> @Produces(MediaType./APPLICATION_XML/)
>>>>>
>>>>> Response upload(@MultipartForm*final*LoaderMultiPartForm loader);
>>>>>
>>>>> }
>>>>>
>>>>> *The MultiPartForm class:*
>>>>>
>>>>> *public**class*LoaderMultiPartForm {
>>>>>
>>>>> @FormParam("loader")
>>>>>
>>>>> @PartType(MediaType./APPLICATION_XML/)
>>>>>
>>>>> *private*Loader loader;
>>>>>
>>>>> @FormParam("bulkProductTypeStream")
>>>>>
>>>>> @PartType(MediaType./APPLICATION_OCTET_STREAM/)
>>>>>
>>>>> *private*InputStream bulkProductTypeStream;
>>>>>
>>>>> *public*Loader getLoader() {
>>>>>
>>>>> *return**this*.loader;
>>>>>
>>>>> }
>>>>>
>>>>> *public**void*setLoader(Loader loader) {
>>>>>
>>>>> *this*.loader= loader;
>>>>>
>>>>> }
>>>>>
>>>>> *public*InputStream getBulkProductTypeStream() {
>>>>>
>>>>> *return**this*.bulkProductTypeStream;
>>>>>
>>>>> }
>>>>>
>>>>> *public**void*setBulkProductTypeStream(InputStream
>>>>> bulkProductTypeStream) {
>>>>>
>>>>> *this*.bulkProductTypeStream= bulkProductTypeStream;
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> The Call:
>>>>>
>>>>> RestClientAPI client = factory.getClient(CatalogResource.*class*);
>>>>>
>>>>> Loader loader = *new*Loader();
>>>>>
>>>>> loader.setXXX();
>>>>>
>>>>> LoaderMultiPartForm loaderForm = *new*LoaderMultiPartForm();
>>>>>
>>>>> loaderForm.setLoader(loader);
>>>>>
>>>>> hugeFileUploadStream =
>>>>> *new*BufferedInputStream(*new*FileInputStream(*new*File("C:\\tmp\\
>>>>> huge
>>>>> FileToUpload.txt")));
>>>>>
>>>>> loaderForm.setBulkProductTypeStream(hugeFileUploadStream);
>>>>>
>>>>> client.upload(loaderForm);
>>>>>
>>>>> Appreciate any help.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Sandeep.
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------
>>>>> ----
>>>>> --------
>>>>> Live Security Virtual Conference
>>>>> Exclusive live event will cover all the ways today's security and 
>>>>> threat landscape has changed and how IT managers can respond.
>>>>> Discussions will include endpoint security, mobile security and 
>>>>> the latest in malware threats.
>>>>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Resteasy-users mailing list
>>>>> [email protected]
>>>>> https://lists.sourceforge.net/lists/listinfo/resteasy-users
>>>>
>>>>
>>>> --
>>>> Bill Burke
>>>> JBoss, a division of Red Hat
>>>> http://bill.burkecentral.com
>>>>
>>>>
>>>> -------------------------------------------------------------------
>>>> -----
>>>> ------
>>>> Live Security Virtual Conference
>>>> Exclusive live event will cover all the ways today's security and 
>>>> threat landscape has changed and how IT managers can respond.
>>>> Discussions
>>>> will include endpoint security, mobile security and the latest in 
>>>> malware threats. 
>>>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>> _______________________________________________
>>>> Resteasy-users mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/resteasy-users
>>>
>>>
>>> --
>>> Bill Burke
>>> JBoss, a division of Red Hat
>>> http://bill.burkecentral.com
>>>
>>>
>>> --------------------------------------------------------------------
>>> ----------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and 
>>> threat landscape has changed and how IT managers can respond. 
>>> Discussions will include endpoint security, mobile security and the 
>>> latest in malware threats. 
>>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Resteasy-users mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/resteasy-users
>
>
> --
> Bill Burke
> JBoss, a division of Red Hat
> http://bill.burkecentral.com
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Resteasy-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/resteasy-users
msgmiddle
<Prev in Thread] Current Thread [Next in Thread>
Current Sitemap | © 2012 planetjava | Contact | Privacy Policy