Exception "Out of space in CodeCache" in conjunction with CDMs

raffael-broad-linden · January 30, 2024, 10:47am

Hey Guys,

I am using A12 version 2023.06-ext4 w/ data-services 36.3.1.

We enabled CDM/CDD support in our project a while back.
Using the following properties on the server:

spring.artemis.mode=embedded
spring.artemis.embedded.enabled=true
mgmtp.a12.dataservices.cdd.dirtyDocuments.async.producer.enabled=true
mgmtp.a12.dataservices.cdd.dirtyDocuments.async.consumer.enabled=true

Since then, we started to monitor exceptions on the server. They seem to be appearing irregularly during server startup and sometimes also at random during runtime.

2024-01-25 08:58:38,676 [n-operation.thread-9][ERROR][l.operation.unsafe.UnsafeRaftReplicateOp][u:] - [<INTERNAL_IP>]:5701 [dev] [5.3.2] Out of space in CodeCache for adapters
java.lang.VirtualMachineError: Out of space in CodeCache for adapters
	at com.hazelcast.cp.internal.session.RaftSessionRegistry.createNewSession(RaftSessionRegistry.java:66)
	at com.hazelcast.cp.internal.session.RaftSessionService.createNewSession(RaftSessionService.java:222)
	at com.hazelcast.cp.internal.session.operation.CreateSessionOp.run(CreateSessionOp.java:57)
	at com.hazelcast.cp.internal.operation.unsafe.AbstractUnsafeRaftOp.call(AbstractUnsafeRaftOp.java:67)
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:299)
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:270)
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:219)
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:180)
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:144)
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.loop(OperationThread.java:134)
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:115)
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111)
2024-01-25 08:58:38,705 [ndpointContainer#0-1][WARN ][listener.DefaultMessageListenerContainer][u:] - Execution of JMS message listener failed, and no ErrorHandler has been set.
org.springframework.jms.listener.adapter.ListenerExecutionFailedException: Listener method 'public void com.mgmtp.a12.dataservices.CddDirtyDocumentsJmsConsumer.process(com.mgmtp.a12.dataservices.document.DocumentReference)' threw exception
	at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:118)
	at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:84)
	at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:736)
	at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:696)
	at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:674)
	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:330)
	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:269)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1239)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1229)
	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1122)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.IllegalMonitorStateException: Current thread is not owner of the Lock[cddUpdate-Item-composed/Item-document/7]
	at com.hazelcast.cp.internal.datastructures.lock.proxy.AbstractFencedLockProxy.newIllegalMonitorStateException(AbstractFencedLockProxy.java:366)
	at com.hazelcast.cp.internal.datastructures.lock.proxy.AbstractFencedLockProxy.unlock(AbstractFencedLockProxy.java:212)
	at com.mgmtp.a12.dataservices.cdd.internal.CddDirtyDocumentsConsumer.createAndUpdateCdd(CddDirtyDocumentsConsumer.java:112)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:411)
	at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734)
	at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
	at com.mgmtp.a12.dataservices.cdd.internal.CddDirtyDocumentsConsumer.lambda$process$0(CddDirtyDocumentsConsumer.java:81)
	at com.mgmtp.a12.uaa.authentication.backend.BackendAuthenticationService.executeWithBackendAuthentication(BackendAuthenticationService.java:52)
	at com.mgmtp.a12.dataservices.cdd.internal.CddDirtyDocumentsConsumer.process(CddDirtyDocumentsConsumer.java:64)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:391)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703)
	at com.mgmtp.a12.dataservices.cdd.internal.CddDirtyDocumentsConsumer$$SpringCGLIB$$0.process(<generated>)
	at com.mgmtp.a12.dataservices.CddDirtyDocumentsJmsConsumer.process(CddDirtyDocumentsJmsConsumer.java:29)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:391)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:751)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703)
	at com.mgmtp.a12.dataservices.CddDirtyDocumentsJmsConsumer$$SpringCGLIB$$0.process(<generated>)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:169)
	at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:119)
	at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:110)
Exception in thread "http-nio-127.0.0.1-8082-exec-1" java.lang.VirtualMachineError: Out of space in CodeCache for adapters
	at org.springframework.web.servlet.FrameworkServlet.publishRequestHandledEvent(FrameworkServlet.java:1148)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1031)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:527)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at com.mgmtp.a12.dataservices.server.autoconfigure.ThreadCleanupFilter.doFilter(ThreadCleanupFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231)
	at org.springframework.security.web.ObservationFilterChainDecorator$FilterObservation$SimpleFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:479)
	at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:340)
	at org.springframework.security.web.ObservationFilterChainDecorator.lambda$wrapSecured$0(ObservationFilterChainDecorator.java:82)
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:128)
	at com.mgmtp.a12.uaa.authentication.security.internal.UAAHeaderFilter.doFilterInternal(UAAHeaderFilter.java:39)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
	at com.mgmtp.a12.uaa.authorization.autoconfigure.security.AuthorizationContextCleanerFilter.doFilterInternal(AuthorizationContextCleanerFilter.java:33)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at com.mgmtp.a12.uaa.authentication.security.internal.DelegatingPathFilter.doFilterInternal(DelegatingPathFilter.java:40)
	at com.mgmtp.a12.uaa.authentication.filter.AbstractPathFilter.doFilter(AbstractPathFilter.java:40)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
	at com.mgmtp.a12.uaa.authentication.filter.AbstractPathFilter.doFilter(AbstractPathFilter.java:38)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
	at com.mgmtp.a12.uaa.authentication.log.internal.LoggingFilter.doFilterInternal(LoggingFilter.java:42)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
	at org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:100)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
	at org.springframework.security.oauth2.server.resource.web.authentication.BearerTokenAuthenticationFilter.doFilterInternal(BearerTokenAuthenticationFilter.java:145)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
	at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
	at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
	at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
	at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82)
	at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
	at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
	at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
	at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$0(ObservationFilterChainDecorator.java:323)
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:224)
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:352)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:268)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:109)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:340)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1744)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:840)
	... 10 common frames omitted

When they appear, we are also experiencing inconsistencies in our data inside documents, for example, sometimes some status flags are not calculated as they should.

In our understanding, the exception is related to the CDD JMS Producer/Consumer, but we are not sure.

Does someone also experienced this kind of exception, or is this a known problem and is there a solution to it? And is it possible that they can cause the data inconsistencies mentioned earlier?

kien-crisp-oak · February 1, 2024, 3:08am

Hi, CDD requires hazelcast cache for the distributed lock. From the log, I suspect that something is wrong with the cache configuration. Could u please help to recheck it or post the cache configuration here?

raffael-broad-linden · February 1, 2024, 9:31am

Hi @kien-crisp-oak

As far as we know, there weren’t any changes made to the cache configuration by us, and we should be using the default provided by the project template and a12 modules.

Also, we don’t define any custom Spring configuration components.

But here is our full configuration:

# Logging
logging.level.root=INFO
logging.file.name=logs/server.log

# Database
spring.datasource.hikari.connectionTimeout=30000
spring.quartz.properties.org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.PostgreSQLDelegate

# Cache
mgmtp.a12.dataservices.search.index.initialization.mode=REBUILD_INDEX

# Search
mgmtp.a12.dataservices.search.analysis.fullText.ngrams.enabled=true

# Security
mgmtp.a12.dataservices.server.context-path=/api
mgmtp.a12.dataservices.jsonRpc.enabled=true
mgmtp.a12.dataservices.jsonRpc.allowedOperations=ADD_DOCUMENT,GET_DOCUMENT,MODIFY_DOCUMENT,VALIDATE_DOCUMENT,LIST_DOCUMENTS,DELETE_DOCUMENT,LIST_LINKS,LIST_CANDIDATES,ADD_LINK,DELETE_LINK,LOAD_ATTACHMENT_URL,LOAD_ATTACHMENT_HEADER,LOAD_THUMBNAIL_URL,LIST_CDDS,LOAD_DOCUMENT_GRAPH
mgmtp.a12.dataservices.attachments.thumbnail.preview.enabled=true

# Resources
mgmtp.a12.dataservices.initialization.import.models.path=classpath:/models/
mgmtp.a12.dataservices.json-rpc.spel.enabled=true

# UAA
mgmtp.a12.uaa.authentication.user.access-rights-resource=classpath:/data/roles.yaml
mgmtp.a12.uaa.authentication.backend.enabled=true
mgmtp.a12.uaa.authentication.backend.grant-super-user-privileges.enabled=true
mgmtp.a12.uaa.authorization.authorization-definition=classpath:/uaa/authorizationDefinition.json
mgmtp.a12.uaa.authorization.child-authorization-definitions=classpath:/data/granularAuthorizations.json

# Spring
server.port=8082
server.address=127.0.0.1

# Database
spring.datasources.dataservices.url=jdbc:postgresql://localhost:5432/testproject
spring.datasources.dataservices.driverClassName=org.postgresql.Driver
spring.datasources.dataservices.name=testproject
spring.datasources.dataservices.username=postgres
spring.datasources.dataservices.password=postgres
spring.jpa.database=POSTGRESQL

# Contentstore
mgmtp.a12.dataservices.contentstore.storage.content-storage=DB
spring.datasources.contentstore.url=jdbc:postgresql://localhost:5432/testproject
spring.datasources.contentstore.driver-class-name=org.postgresql.Driver
spring.datasources.contentstore.name=testproject
spring.datasources.contentstore.username=postgres
spring.datasources.contentstore.password=postgres
spring.datasources.contentstore.jpa.database=POSTGRESQL

# CDMs & CDDs
spring.artemis.mode=embedded
spring.artemis.embedded.enabled=true
mgmtp.a12.dataservices.cdd.dirtyDocuments.async.producer.enabled=true
mgmtp.a12.dataservices.cdd.dirtyDocuments.async.consumer.enabled=true

# Security
mgmtp.a12.uaa.authentication.context-path=/api
mgmtp.a12.uaa.authentication.cors.allowed-origins=http://localhost:8081
mgmtp.a12.uaa.authentication.unsecured.urls=/api/uaa-authentication/selfconfigure,/actuator/health,/actuator/info,/api/attachment/thumbnail/**

# Import data on startup / Resources
#mgmtp.a12.dataservices.initialization.scripts.jsonRpc.enabled=true
#mgmtp.a12.dataservices.initialization.scripts.jsonRpc.path=classpath:/data/request/*.json

# UAA
mgmtp.a12.uaa.authentication.types=OAUTH2
spring.security.oauth2.resourceserver.jwt.jwk-set-uri=http://localhost:8089/realms/A12Realm/protocol/openid-connect/certs
spring.security.oauth2.resourceserver.jwt.issuer-uri=http://localhost:8089/realms/A12Realm

# Configuration for downloading self-configure in front-end
mgmtp.a12.uaa.authentication.client-selfconfiguration.application-base.url=http://localhost:8082/
mgmtp.a12.uaa.authentication.client-selfconfiguration.uaa-base.url=http://localhost:8082/
mgmtp.a12.uaa.authentication.client-selfconfiguration.oauth2.public-client.client-id=a12-spa-client
mgmtp.a12.uaa.authentication.client-selfconfiguration.oauth2.public-client.realm-name=A12Realm
mgmtp.a12.uaa.authentication.client-selfconfiguration.oauth2.public-client.idp-base.url=http://localhost:8089
mgmtp.a12.uaa.authentication.client-selfconfiguration.oauth2.public-client.login-relative.url=
mgmtp.a12.uaa.authentication.client-selfconfiguration.oauth2.public-client.login-redirect-relative.url=introduction
mgmtp.a12.uaa.authentication.client-selfconfiguration.oauth2.public-client.logout-redirect-relative.url=
mgmtp.a12.uaa.authentication.client-selfconfiguration.oauth2.public-client.silent-redirect-relative.url=silent_renew.html
mgmtp.a12.uaa.authentication.client-selfconfiguration.oauth2.public-client.token-exchange-relative.url=token
mgmtp.a12.uaa.authentication.client-selfconfiguration.oauth2.public-client.sso-configuration.username-xpath=//input[@name='username']
mgmtp.a12.uaa.authentication.client-selfconfiguration.oauth2.public-client.sso-configuration.password-xpath=//input[@name='password']
mgmtp.a12.uaa.authentication.client-selfconfiguration.oauth2.public-client.sso-configuration.login-button-xpath=//input[@name='login']

# Actuator
management.endpoints.enabled-by-default=true
management.endpoints.web.exposure.include=*

kien-crisp-oak · February 2, 2024, 2:31am

Hi @raffael-broad-linden , the configuration looks no problems here.

I did some research and found these pages:

There may be some problems with JVM configuration.

Regarding the question if this error could cause data inconsistency. From the configuration, you are enabling the CDD in the async mode which means the CDD process should not have any impact on the documents process, there could be something else that causes the problem/

Best,
Kien

denise-narrow-token · March 1, 2024, 8:13pm

Dear @raffael-broad-linden ,

was the response helpful and is the issue now solved or are there some additional questions? If it is solved, please, use the checkbox to mark the solution to your problem so that other users also know what helped to your case.

Denise from the Discourse team