If just one document is invalid in the database (for whatever reasons) dataservices will refuse to start (at least when using mgmtp.a12.dataservices.search.index.initialization.mode=rebuild_index), so the application is not available, making it impossible to access those documents that are valid.
I am curious why the decision was taken to abort the startup.
To me it would make more sense to let the application start, so that users can at least access the valid documents.
Hi @thomas-soft-grove ,
The DS is a storage for only valid documents and valid models. If the invalid documents are present in our system we cannot guarantee that they can even be deleted from the system (i.e.: for deletion of the document we need to check authorization which requires IDocument instance. if the instance cannot be created the document cannot be deleted). Furthermore DS index should be a searchable representation of the DB table. If the document cannot be created in the index it cannot be found by the search service therefore it cannot be loaded by the overview engine so it cannot be worked on (or event to be deleted).
All of this is aimed to fail fast if we detect problem so that we can avoid weird behavior in search/crud. If you think that you can manage different version of index compared to the DB tables we can introduce a configuration key that will change this behavior. Please create a ticket and we will gladly add this option for your project.
This answer is valid for DS 32.2 - 36.1
I understand that the goal is to store valid documents only.
But bugs happen. Refusing to start DS (which means a complete outage of the application) is like shutting down your laptop because you have one Word Document that can’t be loaded.
Hello @thomas-soft-grove ,
was the ticket for the issue created as suggested by @tomas-thin-gale? Can we close this thread or is there some other clarification needed?
Best regards,
Katerina from the Discourse team
You can look at this in this way but we cannot use database to ensure the data integrity. All of this comes from the application because the data consistency is ensured by validation rules in document model and referential integrity is ensured by relationship model. So if the document becomes invalid and it has been assigned to the link, the client project might experience weird behavior (we do not have any tests on how to behave in this situation).
Furthermore we highly recommend usage of init-app to separate migration from production aspects of your application. During migration you can find problematic document very quickly and fix what needs fixing and you will have time to do it because it is migration. For production use-case you already have data migrated therefore nothing can fail that could cause the server to crash. If the document creation fails because of validation issues now the server will not crash, but rather a single request will fail which is ok.