Thumbnail creation takes huge amount of memory

guido-quiet-root · July 28, 2023, 10:06am

Hi,
we see that the thumbnail creation takes a huge amount of memory in our application. Here is one stack trace

http-nio-8080-exec-7
  at sun.java2d.pipe.DrawImage.scaleImage(Lsun/java2d/SunGraphics2D;Ljava/awt/Image;IIIIIIIILjava/awt/Color;)Z (Unknown Source)
  at sun.java2d.pipe.DrawImage.scaleImage(Lsun/java2d/SunGraphics2D;Ljava/awt/Image;IIIIIIIILjava/awt/Color;Ljava/awt/image/ImageObserver;)Z (Unknown Source)
  at sun.java2d.SunGraphics2D.drawImage(Ljava/awt/Image;IIIIIIIILjava/awt/Color;Ljava/awt/image/ImageObserver;)Z (Unknown Source)
  at sun.java2d.SunGraphics2D.drawImage(Ljava/awt/Image;IIIIIIIILjava/awt/image/ImageObserver;)Z (Unknown Source)
  at net.coobird.thumbnailator.resizers.ProgressiveBilinearResizer.resize(Ljava/awt/image/BufferedImage;Ljava/awt/image/BufferedImage;)V (:143)
  at net.coobird.thumbnailator.resizers.Resizers.resize(Ljava/awt/image/BufferedImage;Ljava/awt/image/BufferedImage;)V (:98)
  at net.coobird.thumbnailator.makers.ThumbnailMaker.makeThumbnail(Ljava/awt/image/BufferedImage;II)Ljava/awt/image/BufferedImage; (:218)
  at net.coobird.thumbnailator.makers.FixedSizeThumbnailMaker.make(Ljava/awt/image/BufferedImage;)Ljava/awt/image/BufferedImage; (:304)
  at net.coobird.thumbnailator.Thumbnailator.createThumbnail(Lnet/coobird/thumbnailator/tasks/ThumbnailTask;)V (:112)
  at net.coobird.thumbnailator.Thumbnails$Builder.toOutputStream(Ljava/io/OutputStream;)V (:2753)
  at com.mgmtp.a12.dataservices.attachment.internal.ThumbnailService.saveInternal(Ljava/io/InputStream;Lcom/mgmtp/a12/dataservices/attachment/ThumbnailType;)Ljava/util/Optional; (ThumbnailService.java:122)
  at com.mgmtp.a12.dataservices.attachment.internal.ThumbnailService.lambda$saveInternal$7(Ljava/io/InputStream;Lcom/mgmtp/a12/dataservices/attachment/ThumbnailType;Lcom/mgmtp/a12/dataservices/attachment/AttachmentHeader;Lcom/mgmtp/a12/dataservices/attachment/persitence/AttachmentPersister;)Ljava/util/Optional; (ThumbnailService.java:104)
  at com.mgmtp.a12.dataservices.attachment.internal.ThumbnailService$$Lambda$5526+0x0000000803576670.apply(Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source)
  at java.util.Optional.flatMap(Ljava/util/function/Function;)Ljava/util/Optional; (Unknown Source)
  at com.mgmtp.a12.dataservices.attachment.internal.ThumbnailService.saveInternal(Lcom/mgmtp/a12/dataservices/attachment/AttachmentHeader;Lcom/mgmtp/a12/dataservices/attachment/ThumbnailType;Ljava/util/function/Supplier;)Ljava/util/Optional; (ThumbnailService.java:104)
  at com.mgmtp.a12.dataservices.attachment.internal.ThumbnailService.lambda$save$1(Lcom/mgmtp/a12/dataservices/attachment/events/AttachmentThumbnailBeforeSaveEvent;Ljava/util/function/Supplier;)Ljava/util/Optional; (ThumbnailService.java:67)
  at com.mgmtp.a12.dataservices.attachment.internal.ThumbnailService$$Lambda$5525+0x0000000803576428.apply(Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source)
  at java.util.Optional.flatMap(Ljava/util/function/Function;)Ljava/util/Optional; (Unknown Source)
  at com.mgmtp.a12.dataservices.attachment.internal.ThumbnailService.save(Lcom/mgmtp/a12/dataservices/attachment/AttachmentHeader;Lcom/mgmtp/a12/dataservices/attachment/ThumbnailType;)Ljava/util/Optional; (ThumbnailService.java:67)
  at com.mgmtp.a12.dataservices.attachment.internal.ThumbnailService$$FastClassBySpringCGLIB$$805191fd.invoke(ILjava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source)
	[... spring aop ...]
  at com.mgmtp.a12.dataservices.attachment.internal.ThumbnailService$$EnhancerBySpringCGLIB$$838c807e.save(Lcom/mgmtp/a12/dataservices/attachment/AttachmentHeader;Lcom/mgmtp/a12/dataservices/attachment/ThumbnailType;)Ljava/util/Optional; (Unknown Source)
  at com.mgmtp.a12.dataservices.attachment.internal.AttachmentService.saveInternal(Lcom/mgmtp/a12/dataservices/attachment/DataServicesAttachment;)Lcom/mgmtp/a12/dataservices/attachment/DataServicesAttachment; (AttachmentService.java:180)
  at com.mgmtp.a12.dataservices.attachment.internal.AttachmentService.save(Ljava/lang/String;Ljava/io/InputStream;Ljava/lang/String;Ljava/lang/String;Ljava/util/List;)Lcom/mgmtp/a12/dataservices/attachment/DataServicesAttachment; (AttachmentService.java:82)
  at com.mgmtp.a12.dataservices.attachment.internal.AttachmentService$$FastClassBySpringCGLIB$$d7c49cbe.invoke(ILjava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source)
	[... spring aop ...]
  at com.mgmtp.a12.uaa.authorization.config.DelegatedMethodSecurityInterceptor.invoke(Lorg/aopalliance/intercept/MethodInvocation;)Ljava/lang/Object; (DelegatedMethodSecurityInterceptor.java:37)
	[... spring aop ...]
  at com.mgmtp.a12.dataservices.attachment.internal.AttachmentService$$EnhancerBySpringCGLIB$$6351f5af.save(Ljava/lang/String;Ljava/io/InputStream;Ljava/lang/String;Ljava/lang/String;Ljava/util/List;)Lcom/mgmtp/a12/dataservices/attachment/DataServicesAttachment; (Unknown Source)
  at com.mgmtp.a12.dataservices.server.document.rest.AttachmentControllerImpl.uploadAttachment(Ljava/io/InputStream;Ljava/lang/String;Ljava/lang/String;)Lcom/mgmtp/a12/dataservices/document/UploadedAttachmentDetail; (AttachmentControllerImpl.java:151)
  at com.mgmtp.a12.dataservices.server.document.rest.AttachmentControllerImpl.uploadAttachment(Lorg/springframework/web/multipart/MultipartFile;)Lcom/mgmtp/a12/dataservices/document/UploadedAttachmentDetail; (AttachmentControllerImpl.java:118)

This thread uses about 320 MB to create a thumbnail from a 4MB JPEG file:

Class Name                                                                                   | Shallow Heap | Retained Heap | Percentage
-----------------------------------------------------------------------------------------------------------------------------------------
org.apache.tomcat.util.threads.TaskThread @ 0x77d90c9c8  http-nio-8080-exec-7 Thread         |          120 |   326,959,976 |     39.31%
|- <Java Local> java.awt.image.BufferedImage @ 0x7ff8028e8                                   |           40 |   181,040,448 |     21.76%
|- <Java Local> java.awt.image.BufferedImage @ 0x78e9351a0                                   |           40 |   135,780,560 |     16.32%
|- <Java Local> com.mgmtp.a12.dataservices.attachment.DataServicesAttachment @ 0x78e607aa8   |           24 |     5,687,304 |      0.68%
|- byte[4265398] @ 0x789600000  ......JFIF..............Exif..MM.*..............-t...........|    4,265,416 |     4,265,416 |      0.51%
|- threadLocals java.lang.ThreadLocal$ThreadLocalMap @ 0x781788c80                           |           24 |       164,192 |      0.02%
|- <Java Local> java.awt.image.BufferedImage @ 0x7ff8029e0                                   |           40 |         7,872 |      0.00%
|- <Java Local> net.coobird.thumbnailator.tasks.io.InputStreamImageSource @ 0x78e6077f8      |           32 |         4,184 |      0.00%
'- java.lang.String @ 0x78e608308  eyJlbmMiOiJBMjU2R0NNIiwiYWxnIjoiZGlyIn0..m5NTeBJbOXZig0ry.|           24 |         2,536 |      0.00%
-----------------------------------------------------------------------------------------------------------------------------------------

Are other projects experience the same problem and what do you do about it? Our load test seems to fail because of this - after about 10-20 users uploading images we run in OOM exceptions.

Thanks for any help,
Guido

trung-async-stone · July 31, 2023, 2:37am

Hi Guido,

This is a known issue of thumbnailator which is being used by DS. I have reported it via A12-14517 (sorry I could not post the Jira link)
You can add a workaround by adding JVM setting -Dthumbnailator.conserveMemoryWorkaround=true but this will not solve the problem completely but reduce the likeliness of OOM issue

tomas-thin-gale · July 31, 2023, 4:02pm

Hi @guido-quiet-root ,

There are 3 options for problems like these:

Solution added by @trung-async-stone is a short term solution
There is an DS Event AttachmentThumbnailBeforeSaveEvent which can be used to overwrite DS thumbnail generation. This poses a lot of work for client project and there will be no long term benefits for the client project when there will be A12 improvements for thumbnail generation in the future.
There is new requirement ticket created: A12-14639 which will be used to find an async solution to thumbnail generation (separate micro services or queued thumbnail generation). So far there has been very little push in this direction from the client projects

guido-quiet-root · August 16, 2023, 10:44am

@trung-async-stone and @tomas-thin-gale thanks for your replies.

We are thinking about disabling the thumbnail creation at all.
Does the switch

mgmtp.a12.dataservices.attachments.thumbnail.preview.enabled=false

disable the thumbnail creation completely?

tomas-thin-gale · August 20, 2023, 11:23am

Hi @guido-quiet-root ,

This property will not help you because it’s purpose it is only to not serve thumbnails that are already generated. We are investigating some weird behavior of thumbnailator library (DS default thumbnail library) in which it is consuming too many resources. Currently it is not possible to disable generation of thumbnails because the client expect the thumbnails for images if selected in overview. We can start working on a switch to disable the generation of thumbnails or alternatively you can provide your own implementation of thumbnail generation by listening to the event AttachmentThumbnailBeforeSaveEvent. We have an example in the examples directory in bitbucket how to implement easy thumbnail replacement (use the same image for all thumbnails to not break UI)

I hope that will help you solve these issues for now

guido-quiet-root · August 21, 2023, 9:13am

@tomas-thin-gale
Hi Tomas,
in the meantime we updated to A12 2023.06. Is the above comment still true for that version? We looked through the code at the DefaultAttachmentService

	private AttachmentHeader saveAttachmentInternal(InputStream is, AttachmentHeader attachmentHeader, String documentModelName) {
[...]
		if (dataServicesCoreProperties.getAttachments().getThumbnail().getPreview().isEnabled()) {
			saveContentInputStreamIntoMemory(attachment);
		}
		ContentPersistenceResult persistenceResult = attachmentRepository
			.saveAttachment(attachmentHeader, attachment.getContent().get());
		if (dataServicesCoreProperties.getAttachments().getThumbnail().getPreview().isEnabled()) {
			handleThumbnails(attachment, persistenceResult.getContentType());
		}
[...]

it looks like it will not be generated if

mgmtp.a12.dataservices.attachments.thumbnail.preview.enabled=false

Thanks,
Guido

tomas-thin-gale · August 21, 2023, 11:01am

Hi @guido-quiet-root ,

I am sorry for the confusion you are right. The message above is valid for DS versions 32.2 - 35.0 in version 36.0.0 we use this configuration key to skip thumbnail generation