R2 Down Too!? (r2d2) 200s And 0 Byte Responses!

By Author: Cole Arendt
Total Articles: 6
Comment this article

"Is this my phone or the site? Lol"

"Same is happening on my phone."

Thus it begins! An outage on 1/10/2024 at 5pm. Somebody (me) shipped some code and left for the day

Code named: R2 Down Too! (Get it... R2D2?)

Why we use R2

Cloudflare R2 is a fantastic service that Cloudflare provides. It is classic blob storage, has an S3 compatible API, and no egress fees!! Their tooling does have some shortcomings, but you can use S3 tooling like `aws s3 sync` to get around that.

Pricing is overall quite reasonable and the integration with Workers is superb (more on Workers later). Moreover, they are private by default and easy to turn into a CDN via attaching a "Custom Domain" that makes item access (but not bucket list access) public.

The Outage

When my colleagues opened the site, _all static resources_ were not showing up. I immediately figured I was the problem... but I had not made any changes lately, and no network calls were returning 404s. What is going ...
... on?

```
curl -i cdn.btst.io/test.html
HTTP/1.1 200 OK
Date: Thu, 11 Jan 2024 03:01:22 GMT
Content-Length: 0
Connection: keep-alive
Access-Control-Allow-Origin: *
Vary: Origin
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type, Vary, Origin, Referer, User-Agent, Authorization
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS, PATCH, HEAD
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=ioFpioSLw6%2B4yTuBiRiId5XFaKbmupwDEngFurqI00Psx6r77PrA68uWDly8vNZSbBuqTfIZpRsTaoOZttQMCoqP0REN3ytXbH4w1KKsckwtB791dE8qq3QuQ6Z0PGg%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 8439ea2cc8ed5ca1-RDU
alt-svc: h3=":443"; ma=86400
```

Cloudflare was returning 200 responses with fully-open CORS and 0 bytes! "Yep! We found your file! Here it is!..." Nothing.

It was doing this for files that exist _and_ for files that did not. So clearly some simple little "hello-world service" is intercepting requests for my CDN. But no images or other static files on our website will load.

The Fix

I immediately hopped onto CloudFlare's Discord server and started searching for threads. Nobody else is having issues? Status is 100% OK? What is going on?

So I wrote up the issue and posted it. Let's see what happens...

But wait! What about my other site that uses a worker as an intermediary to fetch content from its R2 bucket? It is working great!

To mitigate downtime while waiting for a response (now at 11pm at night), I quickly copied the fetching worker, tweaked a few parameters in its `wrangler.toml` (which bucket it is attached to and the name of the worker) and deployed it.

Uploaded cms-download (0.99 sec)
Published cms-download (3.69 sec)

Less than 5 seconds and we are now serving files from the bucket again! But at the wrong DNS... I really do not want to make code changes to the site.

So a few tweaks to the DNS are incoming...

Remove the DNS from the R2 Bucket
Set up the DNS Trigger for the new Worker
Set up a route for the Worker

And voila! We are back up!

Summary

I love using Cloudflare's service because of the high quality products and amazing developer experience that they provide. I was able to self-resolve this issue extremely quickly once I noticed the path to resolution.

Unfortunately, the jury is still out on _why_ we had the issue in the first place. I'm hoping that they can help me trace down the problem. I created an alternate domain (cdn2.btst.io) that is still presenting the issue presently.

Sometimes you've just gotta roll with the punches when R2D2 isn't cooperating!

Need some help managing your site or your tech stack? Give me a call! I would love to help.

Software engineer and entrepreneur helping folks level up in tech and build their web presence. Supporting small businesses through Benetist