UAA Backend Authentication is not threadsafe

UAA
1

Hey ho,

lately i got following exception on my local instance during the startup:

025-01-20 11:45:30,122 [restartedMain       ][WARN ][model.bulkload.BulkImportProblemReporter][u:] - Error while importing models
java.lang.NullPointerException: Cannot invoke "org.springframework.security.core.Authentication.getPrincipal()" because "authentication" is null
	at com.mgmtp.a12.uaa.authorization.AuthorizationService.createNewExecutionEnvironment(AuthorizationService.java:161)
	at com.mgmtp.a12.uaa.authorization.AuthorizationService.lambda$resolveExecutionEnvironment$4(AuthorizationService.java:152)
	at java.base/java.util.Optional.orElseGet(Optional.java:364)

The Authentication in the Spring SecurityContextHolder is null. This happens within my Eventlisteners which all listen to the event DataServicesInitializationFinishedEvent. They are using the BackendAuthenticationService with the superUser.

With debugging i can see, that the superUser will be set into the SecurityContextHolder. Unfortunately i can’t debug more, because this problem is non-deterministic and even marking a debug-point will “fix” the problem.

But following the code i can see, that the BackendAuthenticationService will set the Authentication to null in the finally block. I’m not quite sure if the SecurityContextHolder is threadsafe. If not then this can explain my problem, because the BackendAuthenticationService sets the Authentication to null, after another Eventlistener set it to the superUser but did not run their function. Then the function will be run with Authentication = null and the Exception happens. When disabling one of my three Eventlisteners, then the application starts up normally. This means these Eventlisteners are disturbing others, which supports my suggestion.

Does somebody else got a similar problem and already made an analysis ? I know there are possibilities to change the SecurityContextHolder behavior, so could this be a solution ? Is this known within the UAA component ?

Happy regards,
Tjorben

2

As far as I know, SecurityContextHolder is based on ThreadLocal, so there should not be any interference with other threads at least.

3

Hey Alex :slight_smile:

You are right that the strategy is by default ThreadLocal and thank you for the hint. I debugged the SecurityContextHolder and saw that DS sets the mode to MODE_INHERITABLETHREADLOCAL

Now let’s start think about that fact. All my EventListeners are listen to DataServicesInitializationFinishedEvent. The parent thread is the DS code which publishes this event. Now i use the BackendAuthenticationService in every of these listener and change the SecurityContext. My knowledge does not reach so far, but i would assume, based on my observations, that the SecurityContext is shared between them, because they all get it from the parent and i think this SecurityContext was not copied.

Maybe it is time to dig deeper into :slight_smile:

Happy regards,
Tjorben

4

Hey ho,

i just want to update my current findings, but they lead to a more confusing state :smiley:

All three Eventlisteners will be executed one after another. Spring uses a synchronous EventListening per default and my analysis confirms it. I put an one minute sleep in each EventListener at the beginning and saw, that other Listener will not be executed during the sleep. Also i saw, that the Stacktrace was still the main Stacktrace.

Means, although the SecurityContext will be inherited, other EventListeners should not have impact on others. So the SecurityContext will be cleaned in the finally-block of the BackendAuthenticationService which should be executed directly after the Try-Block. So in theory it should work, but my observation shows that:

  • Very tiny changes to the runtime leads to a fix
    → Just adding a Breakpoint fixed the problem
  • Removing Listeners fix the problem too, which leads me to the conclusion, that other Listeners had impact somehow

Maybe i find a way to log or debug it, when selecting the right points. Maybe there is an edge case with the finally block which i am not aware of ? Currently it does not come up, because i use Postgres as DB instead of H2, which might makes my systems slower when during DB operations, which all of my listeners do

Edit: I added 5 very fast EventListeners and using the BackendAuthenticationService and i can’t reproduce the problem. I will provide a example where you hopefully can try it out :slight_smile:

Happy regards and much fun playing detective,
Tjorben

5

Peace out,

sorry for multi posting, but i think it helps to understand the current thread especially for ppl, who are already read this thread and just want to see the new stuff.

First of all, i uploaded my project here →
https://wiki.mgm -tp.com/confluence/pages/viewpage.action?pageId=413359963

I have to make the url invalid for smuggling it through the discourse detection, because it forbids mgm-wiki links which i support, but i can’t upload my zip here somehow, so i uploaded it there. If someone wants this zip too, contact me :slight_smile:

I just removed the client part. You can build it with gradle and start it normally with the IntelliJ run config [DS] run or just use the bootRun gradle task with the env variable spring.profiles.active=dev-env

I can reproduce the problem almost everytime, rarely i have to re-start it, at least if i start it not in debug mode

Now to some findings I’ve made. In my previous post i wrote that Listeners are independent from each other, because they will be executed synchronously. Means spring waits for a Listener before it starts a new one. Buuut there is one Listener:

public KernelCachesPreloader() {
    }

   @Async
    @CommonDataServicesEventListener
    public void listenOnServicesInitializationFinished(DataServicesInitializationFinishedEvent dataServicesInitializationFinishedEvent) {
        this.backendAuthenticationService.executeWithBackendAuthentication(this.dataServicesCoreProperties.getAuthorization().getBackendJob().getPrincipal().getUsername(), () -> {
...

And it uses the annotation @Async which make sense here of course, but leads to the fact that not all Listeners will be executed synchronously. Does it cause the problem ? Theoretically yes, Practically i don’t know. Didn’t find a way to debug it, so i can get a clear answer. It makes sense, that this problem only exists in my H2 DB, because the application starts with one AppModel only, but my PG is already filled with more models of course, which might lead to a longer execution of the KernelCachesPreloader so it will not end the backend task and resets the SecurityContext, but as i wrote, theoretically. I will try to find a way to prove it :slight_smile:

Maybe someone else already made a similar observation with the BackendAuthenticationService ?

Happy regards,
Tjorben

6

hi @tjorben-atomic-moss ,

Thanks a lot for your effort with different try out. Within UAA team, we have not seen any situation like this and we don’t do any asynchronous within our code. Therefore we can’t give you more insight about what’s going on.

Probably you can create a JIRA bug ticket for A12 then we can have a look deeper in the code and hopefully we can fully reproduce the symtom inr our A12 Environment so that we can give advice or fix this.

Thanks,
Tuan Do

7

Hi @tjorben-atomic-moss,

in DS, we have @Asyc annotation used together with BackendAuthorizationService in the same way as you do. As you showed in the post above. We also recommend using this authorization from jobs, so we expect that there will be different thread handling. We also use it in our jobs without any issues. The behavior you describe is very puzzling to us, and we cannot find anything that is wrong with your approach for now.

8

heyho, i could reproduce it with a clean Project-Template-Project and created A12-16845

Happy regards,
Tjorben

9

Hey @tjorben-atomic-moss, the mentioned ticket is now done and will be included within the 2024.06-ext5 release. Therefore the problem should be solved.