EURO2024 - API Response Delivery

The Problem

Lambda response is very limited; It only allow you to serve up to 6MB of response 5 to be safe. But to be fair -- this is not a limitation, rather it is more like a design choice. Anyway when we have a report that we would like to deliver that should be really short. (e.g. Simple CSV file). Or sometime JSON with quite large dataset. We will have to adopt upload the file to S3. Return the signed URL to frontend and let frontend invoke another call for download. And to make it graceful we have to clean up those files later on.

In Euro we have 1 report that by requirement it only need to relay the data of maximum 30,000 records so we implemented it with naive response. Later on some fields are added and the data size is no longer fit with 5MB :(

So what happen is that API Gateway no longer accept this response and reply with Bad Gateway (Data too large)

                 Browser's REQ                       Upstream REQ
+--------+                          +--------+                          +--------+
|        | -----------------------> |        | -----------------------> |        |
| Client |                          | API Gw |                          | Lambda |
|        | <--------- 504 --------- |        | <-------- 200 ---------- |        |
+--------+                          +--------+                          +--------+
                 Gateway Resp                        Lambda's Resp

The Solution to fix this is either implement the upload to S3 or make it graceful. Or find alternative. Intuitively this 5MB CSV is annoyingly small in our case so to put this case as rest. We find alternative.

We know that we can compress the content and deliver a zip file instead. But that would require us to notify user that the output will be changed. We still don't like this approach. So we find other compression technique. We know that browser and server can compress the content with compression middleware. But we never really did try. So we explore on that.

The Solution

So we use compression middleware on our server application. But adding it will not magically compress our system. This output stream compression only works if client also knows how to. This compression can happen in 2 layers.

A - Gateway Layer API Gateway can compress our content automatically if we setup the proper threshold. But our problem was that the Lambda's content is too large so gateway will reject our content anyway. So we need to compress our content on our own first.

B - Lambda Layer We can compress our content on our own.

In order to make our compression middleware works Client must send accept-encoding header with the standard the the Client support so that it can decompress our output stream. This means if you curl http://localhost:5100/api/my-report this will basically tell our server that hey give me your response. I only read plain binary. So to trigger our compression you have to curl -H 'accept-encoding: gzip' http://localhost:5100/api/my-report this will trigger server to start compressing.

As a result our compression works and it reduces the payload size from 5MB to around 1.4MB.

Here is the final toplogy

                 Browser's REQ                       Upstream REQ
+--------+                            +--------+                            +--------+
|        | - accept-encoding: gzip -> |        | - accept-encoding: gzip -> |        |
| Client |                            | API Gw |                            | Lambda |
|        | <-- compressed content --- |        | <-- compressed content --- |        |
+--------+                            +--------+                            +--------+
                 Gateway Resp                        Lambda's Resp