[Performance] Utilize a CDN for better throughput on reads

[alukach]

A user has pointed out that they get much better performance downloading the same file from HuggingFace than they do with data.source.coop. Diving into this, we can see that HuggingFace responds with a 302 redirect to a Cloudfront server. HuggingFace can do this because they are not pursuing interoperability with S3-clients (at least, I don't believe that they are advertising that their API is S3-compatible). While redirects are technically supported by S3¹, some clients like boto3 don't support this². NVIDIA uses a patch³ to get around this with their S3-compatible API.

However, Cloudfront would bring benefits, particularly for users that are located far away from us-west-2. As such, we should consider:

1. Setting up one or many Cloudfront distributions to serve our data. The strategy for handling different clouds providers and different buckets is to be determined.
2. Serving redirects from our S3-compliant API when the client supports this (perhaps inferred based on the user-agent header).

logs

▶ wget https://data.source.coop/csaybar/3dclouds/pretraining/__TACOCAT__
--2025-11-10 09:02:40--  https://data.source.coop/csaybar/3dclouds/pretraining/__TACOCAT__
Resolving data.source.coop (data.source.coop)... 54.188.53.230, 54.186.232.91, 34.214.239.109
Connecting to data.source.coop (data.source.coop)|54.188.53.230|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 144434880 (138M) [binary/octet-stream]
Saving to: ‘__TACOCAT__’

__TACOCAT__                     100%[======================================================>] 137.74M  18.2MB/s    in 7.6s

2025-11-10 09:02:50 (18.2 MB/s) - ‘__TACOCAT__’ saved [144434880/144434880]

▶ dig -x 18.172.185.108 +short
server-18-172-185-108.yvr52.r.cloudfront.net.

▶ wget https://huggingface.co/csaybar/playground/resolve/main/__TACOCAT__

--2025-11-10 09:03:01--  https://huggingface.co/csaybar/playground/resolve/main/__TACOCAT__
Resolving huggingface.co (huggingface.co)... 18.64.67.102, 18.64.67.107, 18.64.67.39, ...
Connecting to huggingface.co (huggingface.co)|18.64.67.102|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://cas-bridge.xethub.hf.co/xet-bridge-us/68cecf4a2a5dac94776cd712/847778c085fffa81923c520f7c1eaff1eaec3f2215fb097da8c6e2da50952636?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=cas%2F20251110%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251110T170302Z&X-Amz-Expires=3600&X-Amz-Signature=dca5b5fd1a661a116aaa0703097915c8eabc4a3394bb5fd4bb87226dc59cb3f6&X-Amz-SignedHeaders=host&X-Xet-Cas-Uid=public&response-content-disposition=inline%3B+filename*%3DUTF-8%27%27__TACOCAT__%3B+filename%3D%22__TACOCAT__%22%3B&x-id=GetObject&Expires=1762797782&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc2Mjc5Nzc4Mn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2FzLWJyaWRnZS54ZXRodWIuaGYuY28veGV0LWJyaWRnZS11cy82OGNlY2Y0YTJhNWRhYzk0Nzc2Y2Q3MTIvODQ3Nzc4YzA4NWZmZmE4MTkyM2M1MjBmN2MxZWFmZjFlYWVjM2YyMjE1ZmIwOTdkYThjNmUyZGE1MDk1MjYzNioifV19&Signature=Dc%7ELcACz0t0UC4AHBaH22Cr8AbVmUsKjl4tvltnbPeixwWkN-KiEVueFOcF6Hma91yWYJ-fzoZ85Nxca0-n1W0X8QvQdWT7rWBoTQUDc6RaKpB5M-D6e38iT9CfFVgHsJCQtP0xRMPXkUFXl5vyw9oQFC3t-Od1UdBSaOZTcXDJg5Kb7xc9Hs7XhqkDWzgkPqLu7egu1LTm3Z1rmJBEoxlaEoihDE7KZSLjQMofTmB8FNVe9uSSvgH87HCKmf4Nk6dOwGVQCfCYym9EwyeTFijyNozXmM3NpyqOc4jGbJb408JgHuRfWTCXEm1RFk0zB83-2%7EiKvmDZ1s0htAJezcw__&Key-Pair-Id=K2L8F4GPSG1IFC [following]
--2025-11-10 09:03:02--  https://cas-bridge.xethub.hf.co/xet-bridge-us/68cecf4a2a5dac94776cd712/847778c085fffa81923c520f7c1eaff1eaec3f2215fb097da8c6e2da50952636?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=cas%2F20251110%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20251110T170302Z&X-Amz-Expires=3600&X-Amz-Signature=dca5b5fd1a661a116aaa0703097915c8eabc4a3394bb5fd4bb87226dc59cb3f6&X-Amz-SignedHeaders=host&X-Xet-Cas-Uid=public&response-content-disposition=inline%3B+filename*%3DUTF-8%27%27__TACOCAT__%3B+filename%3D%22__TACOCAT__%22%3B&x-id=GetObject&Expires=1762797782&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc2Mjc5Nzc4Mn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2FzLWJyaWRnZS54ZXRodWIuaGYuY28veGV0LWJyaWRnZS11cy82OGNlY2Y0YTJhNWRhYzk0Nzc2Y2Q3MTIvODQ3Nzc4YzA4NWZmZmE4MTkyM2M1MjBmN2MxZWFmZjFlYWVjM2YyMjE1ZmIwOTdkYThjNmUyZGE1MDk1MjYzNioifV19&Signature=Dc%7ELcACz0t0UC4AHBaH22Cr8AbVmUsKjl4tvltnbPeixwWkN-KiEVueFOcF6Hma91yWYJ-fzoZ85Nxca0-n1W0X8QvQdWT7rWBoTQUDc6RaKpB5M-D6e38iT9CfFVgHsJCQtP0xRMPXkUFXl5vyw9oQFC3t-Od1UdBSaOZTcXDJg5Kb7xc9Hs7XhqkDWzgkPqLu7egu1LTm3Z1rmJBEoxlaEoihDE7KZSLjQMofTmB8FNVe9uSSvgH87HCKmf4Nk6dOwGVQCfCYym9EwyeTFijyNozXmM3NpyqOc4jGbJb408JgHuRfWTCXEm1RFk0zB83-2%7EiKvmDZ1s0htAJezcw__&Key-Pair-Id=K2L8F4GPSG1IFC
Resolving cas-bridge.xethub.hf.co (cas-bridge.xethub.hf.co)... 18.172.185.108, 18.172.185.16, 18.172.185.127, ...
Connecting to cas-bridge.xethub.hf.co (cas-bridge.xethub.hf.co)|18.172.185.108|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 144434880 (138M)
Saving to: ‘__TACOCAT__.1’

__TACOCAT__.1                   100%[======================================================>] 137.74M  23.8MB/s    in 6.5s

2025-11-10 09:03:08 (21.3 MB/s) - ‘__TACOCAT__.1’ saved [144434880/144434880]

▶ wget http://us-west-2.opendata.source.coop.s3.us-west-2.amazonaws.com/csaybar/3dclouds/pretraining/__TACOCAT__
--2025-11-10 09:10:09--  http://us-west-2.opendata.source.coop.s3.us-west-2.amazonaws.com/csaybar/3dclouds/pretraining/__TACOCAT__
Resolving us-west-2.opendata.source.coop.s3.us-west-2.amazonaws.com (us-west-2.opendata.source.coop.s3.us-west-2.amazonaws.com)... 3.5.78.113, 3.5.82.187, 3.5.82.65, ...
Connecting to us-west-2.opendata.source.coop.s3.us-west-2.amazonaws.com (us-west-2.opendata.source.coop.s3.us-west-2.amazonaws.com)|3.5.78.113|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 144434880 (138M) [binary/octet-stream]
Saving to: ‘__TACOCAT__.2’

__TACOCAT__.2                   100%[======================================================>] 137.74M  19.0MB/s    in 7.3s

2025-11-10 09:10:16 (18.9 MB/s) - ‘__TACOCAT__.2’ saved [144434880/144434880]

Footnotes

1. https://docs.aws.amazon.com/AmazonS3/latest/API/UsingRouting.html#RedirectsTemporaryRedirection-response-rest-ex1 ↩

2. https://github.com/boto/boto3/issues/2708 ↩

3. https://github.com/NVIDIA/aistore/blob/v1.4.0/docs/s3compat.md?plain=1#L435-L445 ↩

[diegoripley]

To add to this issue:

In my limited experience with a CDN (CloudFlare), it is not just the CDN that matters, but where the underlying data is stored as well, as the CDN has to fetch it from there.

To be specific:

• I had my data stored in CloudFlare R2 (AWS S3 equivalent), in their "Eastern North America" location.
• I used CloudFlare workers (their server less offering)

Example 1
They were not able to max out their 1Gbit connection (hardwired) from Sweden.

Example 2
I have found the same issue with some of my servers in Europe.

From my understanding, this is due to peering. While there are some products like CloudFlare argo, that target use-cases like this, they are very expensive.

My recommendation? Strategic placement of your data assets in multiple regions to target your target audience.

You can do DNS based routing if you want to get fancy, and that would not have to go through your proxy if you do not want it to.

[alukach]

My recommendation? Strategic placement of your data assets in multiple regions to target your target audience.

Agreed. It appears that us-west-2 has become a defacto standard for storage of open data in the AWS world (not sure how that came to be, but here we are) and as such we are storing the vast majority of the data within that region.

For performant cross-region requests, data mirroring might very well be the way to go and the Source data model is currently built to support this. However, it would be prohibitively expensive to mirror all of our datasets so this would likely be done in an opt-in fashion (who owns those mirror accounts and/or who foots the bill is to be determined.)

In #108, I have been experimenting with running the data proxy on Cloudflare Workers. As you likely know, Workers run on Cloudflare's globally distributed CDN. As such, the latency between the user and the server is typically very low. However, as you suggested, the proxy still needs to get to the underlying data which may be across the globe. For data that is far away, Cloudflare's "backbone" provides an optimized network to use for access. Alternatively, for datasets that are mirrored across many regions, the Workers could inspect the Cloudflare headers (e.g. CF-IPCountry or Cf-Ray) to determine which of the data mirror would likely be the most performant for that user.

Initial testing of the CF Workers Proxy has yielded promising results (20-40% improvement on TTFB, 25% improvement on throughput, even matching/beating direct data access performance).

[diegoripley]

Had no idea us-west-2 was the defactor standard for open data in the AWS world (isn't that "earthquake country"...? 😂 ). I briefly looked through your source.coop architecture so I see where y'all are going.

Cloudflare & CDNs

I am going in the same direction, I just recently deployed a subset of my data to Tigris (not affiliated). I have it mirrored to all of their worldwide locations, so I will be able to target my target users efficiently. I am going to do a hybrid approach: Cloudflare workers for the OGC tiling services, but they will point to my Tigris bucket, and I will also give people access to the Tigris bucket itself, so it will be outside of Cloudflare as well, as I have hit some limits on Cloudflare (P.S. they definitely throttle). I am concerned about potential abuse of the Tigris bucket, so I may do a proxy like Source Cooperative is doing if it gets popular in certain places. You will find that Cloudflare has latency issues when it comes to their underlying storage (see this Protomaps doc), but Tigris supposedly has better performance, I still have to test them.

IP Information

Lastly, I think you're DEFINITELY going in the right direction with the CF-IPCountry/Cf-Ray. I am pursuing getting access to IP Address data such as IPinfo.io Lite, and IP abuse data such as FireHOL IP Lists, from open data sources, and to tie that into community trust, specifically vouch (I would look at the author, he's worked on creating some of my favourite products). I hope to use telemetry in my dissemination strategy in order to quickly identify abuse, and to share that information with the community (ex. via simple Parquet files) so that they can make informed actions on facts. Ideally we could integrate vouch into sharing of such information to avoid abuse in sharing such information as well.