Improving the application starts up phase

nhat-round-cloud · June 11, 2024, 3:14am

Hi there,

Is there any way to improve or optimize the application start-up faster?

With the heavy models in our application: the model contains ~28k lines and it takes a lot of time and memory of the cluster to start up (sometimes it crashes on Kubernetes)

We have many enum values in the fields (~15k lines for enum) and quite some computations and validations.

I tried to move those enums to TypeDefinitions and the time is faster about ~10s.

May new requirements come to our project and the model can be increased to ~80k lines then I have no idea how to optimize the model to make the application faster.

Logs in local:

2024-06-11 09:51:43,209 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of ContractCDM finished in 73687 ms
2024-06-11 09:52:08,623 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of VehiclesCoverageCategory_relationship_Vehicle____generated finished in 25386 ms
2024-06-11 09:52:11,328 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of VehicleContract_relationship_Contract____generated finished in 2701 ms
2024-06-11 09:52:13,129 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of PartnerData finished in 1800 ms
2024-06-11 09:52:16,095 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of ContractCoverageCategory_relationship_Contract____generated finished in 2961 ms
2024-06-11 09:52:17,161 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of CoverageCategory finished in 1065 ms
2024-06-11 09:52:36,419 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of Vehicle finished in 19236 ms
2024-06-11 09:52:37,444 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of MetaData finished in 1023 ms
2024-06-11 09:52:39,983 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of ClientContract_relationship_contract____generated finished in 2536 ms
2024-06-11 09:52:42,941 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of Client finished in 2953 ms
2024-06-11 09:53:03,758 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of VehicleContract_relationship_Vehicle____generated finished in 20796 ms
2024-06-11 09:53:06,373 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of Broker finished in 2611 ms
2024-06-11 09:53:29,458 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of VehicleCDM finished in 23060 ms
2024-06-11 09:53:32,030 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of BrokerClient_relationship_Client____generated finished in 2566 ms
2024-06-11 09:53:34,240 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of BrokerContract_relationship_contract____generated finished in 2208 ms
2024-06-11 09:53:49,839 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of CoverageCategoryCDM finished in 15581 ms
2024-06-11 09:53:52,150 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of ClientContract_relationship_client____generated finished in 2306 ms
2024-06-11 09:53:54,303 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of BrokerContract_relationship_broker____generated finished in 2150 ms
2024-06-11 09:53:55,293 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of DocumentData finished in 988 ms
2024-06-11 09:53:58,297 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of ClientCDM finished in 2999 ms
2024-06-11 09:54:02,671 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of Client_with_contracts_CDM finished in 4368 ms
2024-06-11 09:54:03,571 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of ContractCoverageCategory_relationship_CoverageCategory____generated finished in 897 ms
2024-06-11 09:54:04,442 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of VehiclesCoverageCategory_relationship_CoverageCategory____generated finished in 869 ms
2024-06-11 09:54:06,469 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of BrokerClient_relationship_Broker____generated finished in 2023 ms
2024-06-11 09:54:08,030 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of PartnerAddress finished in 1559 ms
2024-06-11 09:54:09,773 [task-1              ][DEBUG][rvices.server.util.KernelCachesPreloader][u:] - Pre-loading of Contract finished in 1740 ms
2024-06-11 09:54:09,779 [task-1              ][INFO ][com.mgmtp.cosmo.BapServerApplication    ][u:] - 

================================================================================================

dpos started: http://localhost:7766/

================================================================================================

2024-06-11 09:54:09,779 [task-1              ][INFO ][a12helper.A12DataServicesHealthIndicator][u:] - A12 data services done
2024-06-11 09:58:57,297 [http-nio-7766-exec-1][INFO ][e.ContainerBase.[Tomcat].[localhost].[/]][u:] - Initializing Spring DispatcherServlet 'dispatcherServlet'
2024-06-11 09:58:57,298 [http-nio-7766-exec-1][INFO ][gframework.web.servlet.DispatcherServlet][u:] - Initializing Servlet 'dispatcherServlet'
2024-06-11 09:58:57,303 [http-nio-7766-exec-1][INFO ][gframework.web.servlet.DispatcherServlet][u:] - Completed initialization in 5 ms
OpenJDK 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=
OpenJDK 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=
OpenJDK 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=
OpenJDK 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=
OpenJDK 64-Bit Server VM warning: Initialization of C1 CompilerThread11 thread failed (no space to run compilers)
OpenJDK 64-Bit Server VM warning: Initialization of C1 CompilerThread9 thread failed (no space to run compilers)
OpenJDK 64-Bit Server VM warning: OpenJDK 64-Bit Server VM warning: Initialization of C1 CompilerThread8 thread failed (no space to run compilers)
Initialization of C1 CompilerThread10 thread failed (no space to run compilers)
[598.029s][warning][codecache] CodeCache is full. Compiler has been disabled.
[598.029s][warning][codecache] Try increasing the code cache size using -XX:ReservedCodeCacheSize=
[598.029s][warning][codecache] CodeCache is full. Compiler has been disabled.
[598.030s][warning][codecache] Try increasing the code cache size using -XX:ReservedCodeCacheSize=
[598.030s][warning][codecache] CodeCache is full. Compiler has been disabled.
[598.030s][warning][codecache] Try increasing the code cache size using -XX:ReservedCodeCacheSize=
[598.030s][warning][codecache] CodeCache is full. Compiler has been disabled.
CodeCache: size=49152Kb used=24041Kb max_used=49135Kb free=25110Kb
 bounds [0x00000204f25a0000, 0x00000204f55a0000, 0x00000204f55a0000]
 total_blobs=10909 nmethods=9427 adapters=1400
 compilation: disabled (not enough contiguous free space left)
              stopped_count=1, restarted_count=0
 full_count=0
[598.030s][warning][codecache] Try increasing the code cache size using -XX:ReservedCodeCacheSize=
CodeCache: size=49152Kb used=24041Kb max_used=49135Kb free=25110Kb
 bounds [0x00000204f25a0000, 0x00000204f55a0000, 0x00000204f55a0000]
 total_blobs=10909 nmethods=9427 adapters=1400
 compilation: disabled (not enough contiguous free space left)
              stopped_count=1, restarted_count=0
 full_count=1
CodeCache: size=49152Kb used=24041Kb max_used=49135Kb free=25110Kb
 bounds [0x00000204f25a0000, 0x00000204f55a0000, 0x00000204f55a0000]
 total_blobs=10909 nmethods=9427 adapters=1400
 compilation: disabled (not enough contiguous free space left)
              stopped_count=1, restarted_count=0
 full_count=2
CodeCache: size=49152Kb used=24042Kb max_used=49135Kb free=25109Kb
 bounds [0x00000204f25a0000, 0x00000204f55a0000, 0x00000204f55a0000]
 total_blobs=10910 nmethods=9428 adapters=1400
 compilation: disabled (not enough contiguous free space left)
              stopped_count=1, restarted_count=0
 full_count=3

Thanks & best regards!

petr-high-peak · June 13, 2024, 3:31am

Hi @nhat-round-cloud , this looks like a question for the kernel, because the time is spent compiling the validation and computation code and filing it into the cache.

tobias-static-slope · June 14, 2024, 7:10am

Maybe it help:
https://docs.spring.io/spring-framework/reference/core/aot.html
But I dont know if the kernel is supporting it.

gildardo-cached-canyon · June 17, 2024, 9:36am

Hi @nhat-round-cloud, if the document models don’t change at runtime, you could have a look at the static server-side validation/computation which was improved with A12-14576. With this way, Java-code should be generated at build-time and this generated code should be available in the classpath, so Kernel can have access when executing validation/computation. You can also have a look at the Javadoc of the interface IDocumentStaticServiceConfig.
For a detailed analysis, we would need a ticket with the affected DMs to have a look at your specific situation.