Indexing a document fails

Data Services
1

We are storing PNG images (base64 encoded) that are needed during printing inside our documents (due to a limitation of the print engine).

We have just upgraded to 2023.02 and have also changed some of the pictures that are stored in the document.

For one picture, A12 fails to store the document with the following error message:

Caused by: java.lang.IllegalArgumentException: category path exceeds maximum allowed path length: max=8191 len=11032 path=[Cockpit.Bescheid.Wappenbild_F...
        at org.apache.lucene.facet.taxonomy.FacetLabel.checkComponents(FacetLabel.java:98)
        at org.apache.lucene.facet.taxonomy.FacetLabel.<init>(FacetLabel.java:78)
        at org.apache.lucene.facet.FacetsConfig.processFacetFields(FacetsConfig.java:380)
        at org.apache.lucene.facet.FacetsConfig.build(FacetsConfig.java:344)
        at com.mgmtp.a12.dataservices.search.internal.lucene.LuceneFacetSupport$FacetAnalysis.build(LuceneFacetSupport.java:86)
        at com.mgmtp.a12.dataservices.search.internal.lucene.LuceneIndexDocument.getDocument(LuceneIndexDocument.java:164)
        at com.mgmtp.a12.dataservices.search.internal.lucene.AbstractLuceneSearchService.updateDocumentsInIndex(AbstractLuceneSearchService.java:137)
        ... 194 common frames omitted

Note that there is no field in our model named Cockpit.Bescheid.Wappenbild_F..... - only a field Cockpit.Bescheid.Wappenbild so it’s apparently something outside of our control

The old encoded length of the picture was 4156 the new one is 10976.

However we have several documents in our database that use a different picture where the encoded length is 55982 bytes - but those were stored in the database with 2022.06

My questions are:

  • was there a change in the behaviour between 2022.06 and 2023.02 regarding indexing of “long” strings?
  • how can we disable the indexing of that field? Indexing it makes no sense to begin with.
  • is there a difference between using Lucene and a Solr for this situation? Because the document with the 50k picture was stored in the database on a system using Solr.

My local Lucene is configured like this:

mgmtp.a12.dataservices.search.lucene.homeDir=c:/Projects/<PROJECT_NAME>/work/lucene/bap
mgmtp.a12.dataservices.search.service=lucene
mgmtp.a12.dataservices.search.index.initialization.mode=rebuild_index
2

Hi @thomas-soft-grove,

The encoded content should not be part of the validation and also not relevant for indexing. The fields of type IStringType have a boolean property noValueValidation defined which is used to omit the a12 document field from indexing once it is set to true. This is primarily used to not index embedded attachments, but it can be used also for other purposes once the field content is string and not meant for indexing.

Alternatively, there is also DocumentBeforeIndexEvent event that can be used to remove properties from the document before indexing if the field is needed for validation.

We have not changed the any restrictions for the size but we have implemented the faceting support in 2023.02 which have now different behavior for Lucene and for Solr, To avoid these problems please consider using embedded Solr option for development mgmtp.a12.dataservices.search.solr.embedded.enabled. We will drop support for Lucene completely in future releases. So adapting right now will make it easier once we do that.

If the problems persist please create a ticket with document models and the troublesome document and we will try to debug this.

This response is valid for DS 35.0.0 - 36.0.0

3

Thanks. It works when using Solr.

As we have to use ERU for modeling, we need to check how/if we can set this noValueValidation property