Migration Exception for Document fields used in CMDs

anon7436550 · January 24, 2024, 7:13am

Hi everyone,

We are currently addressing a specific scenario in our application. Our system features an overview model that includes a field named ‘PuchaseDate’ (typo). This field is computed using a CMD operation structured as follows:

"operation": "...[/Item/Product/PuchaseValue]....

Our objective is to correct this typo by renaming the ‘PuchaseDate’ tags in the documents xml during the execution of a migration script. However, we are encountering issues as shown by the following logs and exception:

...
INFO [template.server.migrations.BaseMigration][u:] -Running document migration for Item-document
INFO [12.dataservices.search.SearchIndexLoader][u:] - Loading [1] document ids for model [Item-document] took [ 11 ms ]
INFO [12.dataservices.search.SearchIndexLoader][u:] - [1] Documents of model [Item-document] has been updated in [86] ms
INFO [aservices.migration.impl.MigrationRunner][u:] - TASK method [migrationTask] on class [com.mgmtp.a12.template.server.migrations.v102.RenamePurchaseMigration] has been executed in [159ms]..
...
INFO [12.dataservices.search.SearchIndexLoader][u:] - Loading [0] document ids for model [Item-composed] took [ 8 ms ]
...
INFO [12.dataservices.search.SearchIndexLoader][u:] - Loading [1] document ids for model [Item-document] took [ 2 ms ]
INFO [12.dataservices.search.SearchIndexLoader][u:] - [1] Documents of model [Item-document] has been updated in [15] ms
...
INFO [12.dataservices.search.SearchIndexLoader][u:] - Index rebuilt in 208 ms
INFO [12.dataservices.search.SearchInitializer][u:] - Index has been rebuilt with data from DB. Following models have been affected [...,Item-composed,Item-document,...]
...
WARN [.support.internal.DefaultDocumentSupport][u:] - Deserialization of document has failed. Reason: For the entity instance '/Item[1]/PuchaseDate[1]', the corresponding entity was not found in the corresponding document model. [ERROR,L0,s0,e0],
...
Caused by: com.mgmtp.a12.dataservices.utils.internal.DocumentModelException: The validation of document of document model 'Item-document' failed. For the entity instance '/Item[1]/PuchaseDate[1]', the corresponding entity was not found in the corresponding document model. [ERROR,L0,s0,e0],

The exception arises exclusively upon restarting the server for the first time, where the database still holds the documents with the typo in it, which triggers the migration script. During this attempt, the server still appears to build and start normally with the exception, except for the associated overview model, which relies on the migrated document. Unfortunately, the list within this overview model no longer displays any items.

Upon restarting the application for the second time, the previously mentioned exception no longer occurs. Upon inspecting the frontend, it operates seamlessly and accurately displays items in the list.

We assume that after the successful execution of the migration on the first attempt, during the step indicating ‘Index has been rebuilt with data from DB. Following models have been affected […, Item-composed, Item-document, …]’, the index may still retain some outdated values for ‘Item-composed.’ Consequently, when the system attempts to verify ‘Item[1]/PuchaseDate[1]’ (old), it understandably does not exist.

Is it possible that A12-Migrations are currently unsupported when using CDMs? Am i missing something?

loi-risen-dale · January 26, 2024, 11:17am

Hi @anon7436550,

Maybe you need to rebuild your Item-composed cdm model index by SearchIndexLoader at the end of your migration.

anon7436550 · February 6, 2024, 8:43am

Hi @loi-risen-dale ,
thank you for the response!

Regrettably, the exception persists even after updating the index of the composed document. To provide clarity, I have recreated the migration using the identical format as demonstrated in the example template:

@MigrationStep(version = "1.0.3", name = "RenamePurchaseMigration")
public class RenamePurchaseMigration{

    private static final String MODEL_TO_MIGRATE = "Item-document";
    private static final String OLD_FIELD = "PuchaseDate";
    private static final String RENAMED_FIELD = "PurchaseDate";
    private final IDocumentRepository documentRepository;
    private final SearchIndexLoader indexLoader;
    private final MigrationConfiguration config;
    public RenamePurchaseMigration( final IDocumentRepository documentRepository,
                            final SearchIndexLoader indexLoader,
                            final MigrationConfiguration config) {
        this.documentRepository = documentRepository;
        this.indexLoader = indexLoader;
        this.config = config;
    }

    @Transactional
    @MigrationTask(name = "Fix PurchaseDate Typo in document")
    @Authenticated(username = "superUser")
    public void migratePuchaseDateField(){
        config.setEnabled(true);
        List<DocumentReference> documentReferences = documentRepository.findAllDocRefsForModel(MODEL_TO_MIGRATE);

        documentReferences.forEach(docRef -> {
            Optional<DataServicesDocument> optDocument = documentRepository.getByDocumentReference(docRef);

            optDocument.ifPresent(documentRepository::update);
        });

        config.setEnabled(false);
        indexLoader.rebuildIndexForModel(MODEL_TO_MIGRATE, 500);        
        indexLoader.updateIndex("Item-composed", 500);
    }

    @DataServicesEventListener(condition = "@migrationConfiguration.isEnabled() && #afterRepositoryLoadEvent.documentReference.documentModelName.equalsIgnoreCase('Item-document')")
    public void listenOnDocumentLoadFromRepository(DocumentAfterRepositoryLoadEvent afterRepositoryLoadEvent)
            throws ParserConfigurationException, IOException, SAXException, TransformerException {
        String documentContent = afterRepositoryLoadEvent.getDocumentContent();

        documentContent = migrateDocument(documentContent);

        afterRepositoryLoadEvent.setDocumentContent(documentContent);
    }

    private String migrateDocument(String documentContent) throws ParserConfigurationException, IOException, SAXException, TransformerException {
        DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
        DocumentBuilder db = dbf.newDocumentBuilder();
        Document document = db.parse(new InputSource(new StringReader(documentContent)));

        NodeList nodes = document.getElementsByTagName(OLD_FIELD);
        for (int i = 0; i< nodes.getLength();i++) {
            Node node = nodes.item(i);
            document.renameNode(node, null, RENAMED_FIELD);
        }

        return convertXMLDocumentToString(document);
    }

    private String convertXMLDocumentToString(Node document) throws TransformerException {
        DOMSource domSource = new DOMSource(document);
        StringWriter writer = new StringWriter();
        StreamResult result = new StreamResult(writer);
        TransformerFactory tf = TransformerFactory.newInstance();
        Transformer transformer = tf.newTransformer();

        transformer.transform(domSource, result);
        return writer.toString();
    }
}

After conducting further analysis, I attempted to create a separate migration script specifically designed to update the index of the composed document. I ensured that this script runs after the renameMigration by simply setting the version of the @MigrationStep to ‘1.0.4’ with the intention of resolving the issue. This approach didi also not work and throw the known exception.

@MigrationStep(version = "1.0.4", name = "UpdateIndexMigration")
...
  @MigrationTask
  public void migrationTask() {
    searchIndexLoader.updateIndex("Item-composed",500);
  }

However, when I modify the version numbers of UpdateIndexMigration to 1.0.3 and RenamePurchaseMigration to 1.0.4, causing the update to occur before the rename, the approach surprisingly executes without any exceptions. Nonetheless, this doesn’t seem logically sound because the index problems should still persist as the rename operation occurs after the index update.

Here is a Log snipped:


2024-02-06 09:33:04,629 [main                ][INFO ][aservices.migration.impl.MigrationRunner][u:] - TASK method [migrationTask] on class [com.mgmtp.a12.template.server.migrations.v102.PersonToAssignmentMigration] Was already executed at [Do. Feb. 01 10:53:41 MEZ 2024].
2024-02-06 09:33:04,651 [main                ][INFO ][12.dataservices.search.SearchIndexLoader][u:] - Loading [0] document ids for model [Item-composed] took [ 20 ms ]
2024-02-06 09:33:04,653 [main                ][INFO ][aservices.migration.impl.MigrationRunner][u:] - TASK method [migrationTask] on class [com.mgmtp.a12.template.server.migrations.v102.UpdateIndexMigration] has been executed in [23ms]..
2024-02-06 09:33:04,673 [main                ][INFO ][ion.backend.BackendAuthenticationService][u:] - Using user [superUser] for backend authentication with strategy [InheritableThreadLocalSecurityContextHolderStrategy]
2024-02-06 09:33:04,730 [main                ][INFO ][12.dataservices.search.SearchIndexLoader][u:] - Loading [1] document ids for model [Item-document] took [ 9 ms ]
2024-02-06 09:33:04,819 [main                ][INFO ][12.dataservices.search.SearchIndexLoader][u:] - [1] Documents of model [Item-document] has been updated in [82] ms
2024-02-06 09:33:04,819 [main                ][INFO ][aservices.migration.impl.MigrationRunner][u:] - TASK method [migratePuchaseDateField] on class [com.mgmtp.a12.template.server.migrations.v102.RenamePurchaseMigration] has been executed in [151ms]..
2024-02-06 09:33:04,821 [main                ][INFO ][aservices.migration.impl.MigrationRunner][u:] - TASK method [migrateLinkDocuments] on class [com.mgmtp.a12.dataservices.migration.RelationshipLinkMigrationStep] Was already executed at [Do. Feb. 01 10:53:41 MEZ 2024].
2024-02-06 09:33:04,821 [main                ][INFO ][aservices.migration.impl.MigrationRunner][u:] - TASK method [migrateAttachments] on class [com.mgmtp.a12.dataservices.migration.AttachmentsV2MigrationStep] Was already executed at [Do. Feb. 01 10:53:41 MEZ 2024].
2024-02-06 09:33:04,821 [main                ][INFO ][aservices.migration.impl.MigrationRunner][u:] - TASK method [validateRelationshipDocRefs] on class [com.mgmtp.a12.dataservices.migration.RelationshipDocRefValidationMigrationStep] Was already executed at [Do. Feb. 01 10:53:41 MEZ 2024].
2024-02-06 09:33:04,821 [main                ][INFO ][aservices.migration.impl.MigrationRunner][u:] - TASK method [migrateAttachmentsToContentStore] on class [com.mgmtp.a12.dataservices.migration.AttachmentsMigrateToContentStoreMigrationStep] Was already executed at [Do. Feb. 01 10:53:41 MEZ 2024].
2024-02-06 09:33:04,821 [main                ][INFO ][ion.backend.BackendAuthenticationService][u:] - Using user [superUser] for backend authentication with strategy [InheritableThreadLocalSecurityContextHolderStrategy]

Am i missing something?

kien-crisp-oak · April 1, 2024, 4:35am

Hi, the searchIndexLoader#updateIndex only reindexes for a specified document model, it does not work with CDD.
For your case, please update the documents first then call the method searchIndexLoad#updateSearchIndex to reindex all documents including its CDD documents.

Best,
Kien