指令碼化 GeoDNS：離開 AWS 轉向 Bunny

To op­ti­mize the per­for­mance met­rics of this blog, I re­lied on Ama­zon CloudFront to serve the as­sets,¹ and Route 53 for the geo-aware DNS² to di­rect read­ers to the clos­est host serv­ing the con­tent. But Route 53 is ex­pen­sive: US$0.50 per zone per month, US$0.70 per mil­lion ge­olo­ca­tion queries, and US$0.75 per healthcheck per month, plus US$1 per month for the KMS key needed to sign zones with DNSSEC. When I dis­cov­ered that Bunny DNS is al­most free for up to 500 do­mains and fea­tures script­able an­swers, I made the jump! Plus, it’s a Eu­ro­pean com­pany based in Slovenia.

• Bunny CDN
• Bunny DNS
  • The HTTPS record solution
  • The hy­brid solution
• ACME DNS with Hickory DNS

Bunny CDN#

Bunny.net started as BunnyCDN in 2015. They added IPv6 sup­port in 2017. Switch­ing was un­event­ful. I don’t bring a cus­tom do­main: I serve the con­tent from the b-cdn.net do­main. As I de­pend on con­tent ne­go­ti­a­tion to serve AVIF and WebP im­ages, I con­fig­ure the CDN to cache each im­age for­mat sep­arately.

I dis­able log­ging to avoid col­lect­ing per­sonal in­for­ma­tion. While the bunny.net GDPR page prom­ises they anonymize data, it is un­clear what CloudFront keeps in its own logs when log­ging is disabled.

The only down­side of the bunny.net CDN of­fer­ing is the lack of HTTP/3. I asked their sup­port: two min­utes later, they an­swered they are work­ing on it but have no ETA to share.

Bunny DNS#

Bunny.net in­tro­duced their DNS of­fer­ing in 2022 and re­cently re­duced its cost to al­most zero. They sup­port ge­o­graph­i­cal rout­ing, but a very cool fea­ture is the script­able DNS: it runs a piece of user-sup­plied JavaScript on each query to com­pute the answer.

I run seven small NixOS vir­tual ma­chines to host this blog around the world: Paris, Nurem­berg, Ash­burn, Sin­ga­pore, San Jose, São Paulo, and Syd­ney. Bunny’s script­able DNS lets me re­turn the IP ad­dresses of the two near­est servers. The browser falls back to the sec­ond server if the first one is down.

const servers = [
  {
    ipv4: "45.90.160.60",
    ipv6: "2a0c:8881:5:100::1",
    latitude: 48.88,
    longitude: 2.22,
  },
  /* […] */
  {
    ipv4: "139.180.162.114",
    ipv6: "2001:19f0:5800:1855:5400:6ff:fef0:4c0a",
    latitude: -33.87,
    longitude: 151.21,
  },
];
const families = { A: "ipv4", AAAA: "ipv6" };
const records = { A: ARecord, AAAA: AaaaRecord };

export default function handleQuery(query) {
  const type = String(query.request.queryType).toUpperCase();
  const family = families[type];
  if (!family) return [];
  // Get user location
  const location = query.request.geoLocation;
  // Build a list of candidates
  let candidates = servers.map(
    (s) => new Server(s[family], s.latitude, s.longitude, 100),
  );
  // Select the two closest alive servers
  const selected = [];
  while (selected.length < 2) {
    const server = RoutingEngine.getClosestServer(candidates, location, true);
    selected.push(server);
    candidates = candidates.filter((s) => s.ip !== server.ip);
  }
  // Return their IP addresses
  return selected.map((s) => new records[type](s.ip, 3600));
}

The above script re­lies on the RoutingEngine.getClosestServer() helper, which re­turns the clos­est alive³ server to the user.⁴ Thanks to Doggo, we can check it works as ex­pected, at least for A:⁵

$ doggo --gp-from="Amsterdam" A vincent.bernat.ch
LOCATION          NAME                TYPE  TTL    ADDRESS
Amsterdam, NL
                  vincent.bernat.ch.  A     3600s  45.90.160.60
                  vincent.bernat.ch.  A     3600s  116.203.18.48
$ doggo --gp-from="Texas" A vincent.bernat.ch
LOCATION          NAME                TYPE  TTL    ADDRESS
Houston (TX), US
                  vincent.bernat.ch.  A     3600s  5.161.44.145
                  vincent.bernat.ch.  A     3600s  149.28.219.155

The most an­noy­ing draw­back of this method is that a user from Buenos Aires gets both São Paulo (1000 mi, 30 ms RTT) and Ash­burn (5200 mi, 140 ms RTT). Most re­solvers ran­dom­ize the or­der, so the user may pull the web page from Ash­burn. We could re­turn only the near­est server—and swap the scripted record with the built-in ge­o­graphic rout­ing in­stead. But if the server be­comes un­avail­able, the browser has no al­ter­na­tive un­til the DNS cache ex­pires. It leaves a short win­dow of un­avail­abil­ity, even with a short TTL.

The HTTPS record solution#

The HTTPS DNS record, de­fined in RFC 9460, solves both the avail­abil­ity and the la­tency is­sues. Mod­ern browsers query this record to get ad­di­tional in­for­ma­tion to reach an HTTP server, in­clud­ing an al­ter­na­tive port and the ap­pli­ca­tion layer protocol—HTTP/2 or HTTP/3. Each record has a pri­or­ity. We can ask browsers to con­tact the near­est server first and fall back to the next one if the first is un­available:

vincent    HTTPS  1 first.geo  alpn="h3,h2"
vincent    HTTPS  2 second.geo alpn="h3,h2"
vincent    HTTPS  3 .          alpn="h3,h2"

; Bunny DNS must provide the two closest servers as "first.geo"
; and "second.geo" with geographic or latency routing.
first.geo  A      45.90.160.60
first.geo  AAAA   2a0c:8881:5:100::1
second.geo A      116.203.18.48
second.geo AAAA   2a01:4f8:1c0c:5eb5::1

vincent    A      45.90.160.60
vincent    A      116.203.18.48
vincent    AAAA   2a0c:8881:5:100::1
vincent    AAAA   2a01:4f8:1c0c:5eb5::1

This looks like the ideal so­lu­tion! Un­for­tu­nately, Chrome ig­nores any record whose tar­get is not ., hence the last record at pri­or­ity 3. More­over, un­like with a CNAME, Bunny DNS does not re­turn the IP ad­dresses for first.geo and second.geo in the ad­di­tional sec­tion. The re­solver needs to send a first re­quest to get the HTTPS record, and a sec­ond to re­solve first.geo.

HTTPS records are still too new for this use case. Back to the draw­ing board! 📐

The hybrid solution#

As a com­pro­mise, the script re­turns the two near­est servers if they are at a sim­i­lar dis­tance from the user. Oth­er­wise, it re­turns only the near­est one with a lower TTL of 2 minutes.

export default function handleQuery(query) {
  const location = query.request.geoLocation;
  let candidates = servers.map(
    (s) => new Server(s.ipv4, s.latitude, s.longitude, 100),
  );

  // Select the two closest servers
  const selected = [];
  while (selected.length < 2) {
    const server = RoutingEngine.getClosestServer(candidates, location, true);
    selected.push(server);
    candidates = candidates.filter((s) => s.ip !== server.ip);
  }

  // Compute the distance to the user for each of them
  const [first, second] = selected.map((s) =>
    GeoDistance.calculate(s, location),
  );

  // If the second one is at twice the distance and at more than 1000 miles
  // from the first, return only the first one with a TTL of 2 minutes.
  if (second > 2 * first && second - first > 1000) {
    return [new ARecord(selected[0].ip, 120)];
  }

  // Otherwise, return both with a TTL of 1 hour.
  return selected.map((s) => new ARecord(s.ip, 3600));
}

We can check it with Doggo. From Chicago, we get only the Ash­burn server with a TTL of 2 min­utes, while from Mex­ico, we get both Ash­burn and San Jose.

$ doggo --gp-from="Chicago" A vincent.bernat.ch
LOCATION          NAME                TYPE  TTL    ADDRESS
Chicago (IL), US
                  vincent.bernat.ch.  A     120s   5.161.44.145
$ doggo --gp-from="Mexico" A vincent.bernat.ch
LOCATION          NAME                TYPE  TTL    ADDRESS
Queretaro, MX
                  vincent.bernat.ch.  A     3600s  5.161.44.145
                  vincent.bernat.ch.  A     3600s  149.28.219.155
$ doggo --reverse 5.161.44.145 149.28.219.155
NAME                          TYPE  TTL    ADDRESS
145.44.161.5.in-addr.arpa.    PTR   7187s  web05.luffy.cx.
155.219.28.149.in-addr.arpa.  PTR   3600s  web07.luffy.cx.

Note

En­able JavaScript to make the map be­low in­ter­ac­tive: point at a lo­ca­tion to get the DNS an­swer, or dis­able servers to see how the an­swers change.

This page was brought to you by ???. Check on the map be­low whether it matches your location!

The com­plete code is a bit more com­plex. It works around RoutingEngine.getClosestServer() sup­port­ing only IPv4, and it han­dles the case where there are not enough alive servers to build an answer.

Bunny.net has a com­pre­hen­sive Ter­raform provider. I lever­age CDK Ter­rain to build the tar­get state in TypeScript and OpenTofu to de­ploy it. Have a look at “Man­ag­ing in­fra­struc­ture with Ter­raform, CDKTF, and NixOS” for more details.⁶

ACME DNS with Hickory DNS#

Route 53 cov­ered one ad­di­tional use case of mine: serv­ing acme.luffy.cx to solve DNS-01 ACME chal­lenges. Un­like Route 53, Bunny DNS can­not re­strict ac­cess to a sin­gle zone with a fine-grained ac­cess pol­icy. As this us­age re­quires nei­ther high avail­abil­ity nor low la­tency, I re­placed Route 53 with Hick­ory, a mem­ory-safe DNS server writ­ten in Rust. It sup­ports au­to­matic DNSSEC sig­na­tures and dy­namic up­dates. Its con­fig­u­ra­tion is simple:

[[zones]]
zone = "acme.luffy.cx"
zone_type = "Primary"

[zones.stores]
allow_update = true
type = "sqlite"
journal_path = "acme.luffy.cx.sqlite"
zone_path = "acme.luffy.cx.zone"

# Private key to sign the zone with DNSSEC. Generated with:
#  openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256
[[zones.keys]]
algorithm = "ECDSAP256SHA256"
key_path = "/etc/hickory-dns.acme.luffy.cx.pem"

# TSIG key to sign zone updates. Generated with:
#  openssl rand 32
[[zones.stores.tsig_keys]]
algorithm = "hmac-sha256"
key_file = "/etc/hickory-dns.acme.luffy.cx.update.acme.luffy.cx.tsig"
name = "update.acme.luffy.cx."

The pro­vided zone file only seeds the data­base on first start. I del­e­gate DNS-01 chal­lenges to this zone with CNAME records:

_acme-challenge          CNAME bernat.ch.acme.luffy.cx.
_acme-challenge.vincent  CNAME vincent.bernat.ch.acme.luffy.cx.

Lego, the ACME client I use to get cer­tifi­cates from Let’s En­crypt, an­swers the DNS-01 chal­lenge by push­ing en­tries with the DNS UPDATE protocol.

Un­for­tu­nately, Bunny DNS can­not pub­lish a DS record in a zone.⁷ I need one to se­cure the acme.luffy.cx zone and cer­tifi­cate de­liv­ery. There­fore, I host the luffy.cx zone on deSEC,⁸ a free DNS host­ing ser­vice run by a non-profit or­ga­ni­za­tion based in Berlin. I de­rive the DS record to pub­lish in luffy.cx from the DNSKEY record of acme.luffy.cx:

$ ZONE=acme.luffy.cx
$ dig @nsupdate.luffy.cx $ZONE DNSKEY +noall +answer \
>   | dnssec-dsfromkey -a SHA-256 -f - $ZONE
acme.luffy.cx. IN DS 55628 13 2 4946E7508F20E15F1B577A231104B029A1A82E04699A80CD46ADCE31227EE81F

Over­all, I am quite happy with the switch to bunny.net. There are still a few rough edges, like the in­abil­ity to pub­lish DS records, the lack of HTTP/3, and the IPv4-only RoutingEngine.getClosestServer() helper. The bill went from US$5 per month to US$1. So far, per­for­mance is on par with AWS, but I only fi­nal­ized the cur­rent setup, in­clud­ing the server in Syd­ney, in late Oc­to­ber. I need a bit more time to gather more data, and I’ll post a short update. 📈