How to implement a custom operation/endpoint to download huge amounts of data?

Data Services
1

Context
For our project AASH, we incur the necessity of downloading huge amounts of data into one file (meaning possibly gigabytes of data).

Problem
We’d like to return a “stream like” http response. This is possible in spring boot using something like

    public ResponseEntity<StreamingResponseBody> myEndoint() {

        // here some consumer that writes the response body *asynchronously*
        StreamingResponseBody stream = outputStream -> {
            try (PrintWriter writer = new PrintWriter(new OutputStreamWriter(outputStream, StandardCharsets.UTF_8))) {
                log.info("Backup: Invoked async body handler");
                writer.println("--- 5 GB of data ---");
                writer.flush(); // Ensure all data is sent
            } catch (Exception e) {
                log.error("Error while streaming data: {}", e.getMessage());
            }
        };

        // create response header and return result including our consumer
        return ResponseEntity.ok()
                .headers(headers)
                .body(stream);
    }

However, the JsonRpcBasicServer that executes my custom operation does not apply the consumer, hence my download is always of zero byte size.

Looking for
I’m looking for a way to enable the data service backend to provide an endpoint/operation that allows to stream an arbitrary amount of data for download.

2

There are several approaches to this:

  • use a normal spring mvc endpoint like you described. Using dataservices rpc is not a must.
  • use a reactive endpoint to not unnecessarily block servlet resources while the download is in progress
  • preferred approach: store the file to be downloaded into some owncloud, nextcloud or whatever instance and do the actual download through this by redirecting the user there
3

When using approach one or two, how do I weave the UAA security into that endpoint?

BTW: The StreamingResponseBody is thrown at the AsyncTaskExecutorThread pool.

4

Hi @juergen-smooth-ice,

UAA secures the resources of the server by path. So your endpoint will be secured if it will be in the secured path. Please check UAA configuration (i.e.: everything in api context path is secured by default). If you use secured API from DS, you will also get secured data in this endpoint.

Please keep in mind that security checks will often require deserialized data, which will slow down your processing.

5

Moin @juergen-smooth-ice,
you can define a secured custom REST endpoint as follows:

  • You need an authorization definition that defines the permission you want to check for. UAA describes the potential steps in the following chapters:
    – 3.4. Decision Context Variables
    – 3.5. Getting Started
  • The concept of the authorization definition is explained in 3.6. Authorization Definition. If you don’t need a custom permission you can also check out the already defined permissions by Data Services under 4.14. Authorization Properties.
    – The Project Template also contains an example for a custom childAuthorizationDefinition.json file under the path import/data/.
    – Additionally the property mgmtp.a12.uaa.authorization.child-authorization-definitions has to be defined.
  • Your Restcontroller could then be implemented as follows:
package com.mgmtp.a12.template.server;

import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class HelloController {

    @PreAuthorize("hasUAAPermission('Document Read')")
    @GetMapping("/api/hello")
    public String hello(){
        return "Hello World!";
    }
}
  • When you are using the Project Template or a similar basis, you should be able to send a request to http://localhost:8082/api/hello.

:warning: It is important that your endpoint is under /api or the corresponding server.servlet.context-path property defined in your server configuration. See the 2.1.4. UAA security context configuration for more details.

Kind regards,
Jan

6

Thanks! It might be worth mentioning that context paths like

/api/my/custom/endpoint work fine

while

/custom/endpoint(duh!) or
/api/v2/rpc/custom/endpointwill be stripped of their authorisation.

7

For securing endpoint, please use SecuredController annotation from UAA. If you are using @PreAuthorize("hasUAAPermission('Document Read')") please make sure that the parameters of the annotated method contain resources that Document Read scope need. For scopes and their resources, please see DS documentation: <INTERNAL_LINK>

Additionally, UAA has 2 interesting properties:

mgmtp.a12.uaa.authentication.unsecured.urls
mgmtp.a12.uaa.authentication.context-path

Which overwrite what I said above. unsecured.urls will make any endpoint in particular context path unsecured. If you use secured API in those controllers, it calls will fail with exception.