When we start data-services, we need to trigger our different cron jobs, and enable rabbitMQ synchronization jobs only after all documents are indexed. As there can be 1000+ documents, if a cron job is triggered before end of indexing, it can either lead to an exception or return wrong results of queries.
Is there any way to know when solr has finished indexing all documents ?
Hello @alexis-raw-iron,
According to the Initialization Sequence, I think we can handle the use-case by leveraging DataServicesCustomInitializationEvent.
Please let us know if it helps. Thanks
Thank you I’ll have a look at that, but I have another question in the meantime:
Does it include also CDM indexing when we initialize data ? Because in that case it takes a very long time for only 300 documents initilized. The CDD properties are set to be async:
mgmtp.a12.dataservices.cdd.dirtyDocuments.sync.enabled=false
mgmtp.a12.dataservices.cdd.dirtyDocuments.async.consumer.enabled=true
mgmtp.a12.dataservices.cdd.dirtyDocuments.async.producer.enabled=true
I can confirm that the first solution about listening to the event DataServicesCustomInitializationEvent works. If the Solr indexing is not finished (therefore if this event is not published), I prevent our cron jobs for being executed.
Unfortunately, I’ve investigated on the second matter, and I can’t find anything to use with the JMS to know if there are still some messages in the Artemis queue.
Because this is what happens, right ? During initilization, each document is created and a message is published on ArtemisMQ dirtyDocumentsQueue queue?
Moreover, even if I succeed in this checks, I still have a short time where index is built and dirtyDocumentsQueue is not yet prepared: the initialization time, in this case I also need to prevent my cron jobs from executing.
Hello @alexis-raw-iron,
Your concerns are valid.
I’m thinking of a simple fail-and-retry approach in which we can wait for the number of root documents to be the same as the number of corresponding add documents and then handle your logic afterward.
That’s my suggestion, please get in touch with DS for an advisory as well.
Hello,
We came up with a temporary solution.
For RabbitMQ listeners, we can set the spring.rabbitmq.listener.simple.auto-startup to false and trigger the start later in the code (we chose to trigger it after all print models are loaded, meaning after Solr indexing and data initialization (but not CDD indexing unfortunately).
For Quartz scheduler, same idea, we set the spring.quartz.auto-startup to false and trigger the start at the same moment as above.
So we would still need A12 to provide an event publishing when CDD indexing is done if possible.
Hello,
@alexis-raw-iron can you please create A12 Requirement for your need of providing an event publishing when CDD indexing is done?