Large-scale IP-based virtual hosting

Yesterday there was a thread on the NANOG list about IPv6 addressing for web sites. There is a great opportunity here to switch back to having an IP address per site, like we did with IPv4 in the mid-1990s :-) Of course the web is a bit bigger now, and there is more large-scale hosting going on, but the idea of configuring a large CIDR block of addresses on a server is not unprecedented - except back then we were dealing with /18s rather than /64s.

Demon's Homepages service predated widespread browser support for name-based virtual hosting. In fact it provided one of the download sites for IE3, which was the first version that sent Host: headers. So Homepages was based on IPv4 virtual hosting and had IIRC 98,304 IP addresses allocated to it in three CIDR blocks. It was a single web server, in front of which were a few reverse proxy caches that took most of the load, and that also had all the IP addresses. Every cache would accept connections on all the IP addresses, and the load was spread between them by configuring which address ranges were routed to which cache.

The original version of Homepages ran on Irix, and used a cunning firewall configuration to accept connections to all the addresses without stupid numbers of virtual interfaces. Back then there were not many firewall packages that could do this, so when it moved to BSD (first NetBSD then FreeBSD) we used a "NETALIAS" kernel hack which allowed us to use ifconfig to bring up a CIDR block in one go.

Sadly I have never updated the NETALIAS code to support IPv6. But I wondered if any firewalls had caught up with Irix in the last 15 years. It turns out the answer is yes, and the key feature to look for is support for transparent proxying. On FreeBSD you want the ipfw fwd rule. On Linux you want the TPROXY feature. You can do a similar thing with OpenBSD pf, though it requires the server to use a special API to query the firewall, rather than just using getsockname().

On Demon's web servers we stuffed the IP address into the filesystem path name to find the document root, or used various evil hacks to map the IP address to a canonical virtual server host name before stuffing the latter in the path. mod_vhost_alias is very oriented around IPv4 addresses and host names, so probably not great for IPv6, so mod_rewrite is a better choice if you want to break addresses up into a hierarchial layout. But perhaps it is not that ugly to run a name server which is authoritative for the reverse ip6.arpa range used by the web server, and map the address to the hostname with UseCanonicalName DNS.