Every elevation profile in Geo Elevation, on the web and in the iPhone, iPad and Mac app, comes from our own server. Draw a route, press Get elevation, and about a hundred milliseconds later the chart is there. This is how that server came to hold two terabytes of the Earth’s surface, and why I stopped renting elevation by the request.

Why not just call Google

For years Geo Elevation asked Google’s Elevation API. It works, it is accurate, and it is metered. A profile is one request with up to a few hundred sampled points, so the bill was never frightening, but the arrangement had two problems that grew on me. The apps could not work without a key I had to protect, and the data was a black box: I could not tell a user which dataset a number came from, or offer them another one.

The open alternative is OpenTopoData, Andrew Nisbet’s open-source elevation server. Its public instance is generous but limited by design: a thousand requests a day, a hundred locations per request, one request a second. That is fine for trying it and useless for an app. The code is MIT-licensed, though, and the datasets it can read are public. So the answer was obvious: run my own.

The server

The server is a Linux box we run for our own AI services: 48 cores, 188 GB of RAM, GPUs for the models, and a 5.8 TB NVMe drive. The elevation API uses none of the GPUs. What it wants is disk and memory, and the second one matters more than you would think, because Linux keeps the tiles people actually ask about in the page cache. Right now that cache holds 91 GB, which is to say most of the populated world.

OpenTopoData runs there as one Docker container, Gunicorn and Flask in front of rasterio and GDAL, with memcached beside it. The data directory is mounted read-only. The whole configuration is this:

max_locations_per_request: 500

datasets:
- name: aster30m
  path: data/aster30m/
- name: mapzen
  path: data/mapzen/

Two datasets, two terabytes

ASTER GDEM version 3 is NASA and METI’s global elevation model at one arc-second, about 30 metres, from 83°N to 83°S. It ships as 22,912 tiles of one degree by one degree, each a zip with two GeoTIFFs inside: the elevation, and a count of how many satellite scenes went into each pixel. I wanted the first and not the second.

Downloading it needs a NASA Earthdata account and a bearer token. I wrote a small downloader that reads OpenTopoData’s list of tile URLs, fetches four zips at a time, unpacks each one, keeps the _dem.tif, drops the _num.tif, and logs everything so it can pick up where it left off. It started at 09:53 on a Friday and finished at 16:53 on Saturday: 31 hours, 22,912 of 22,912, 319 GB of GeoTIFF on disk. I kept the zips, another 379 GB, until the tiles had proved themselves in the apps, then deleted them; every tile is re-downloadable.

Mapzen terrain tiles are the other dataset, and the one Geo Elevation uses by default. They are a global mosaic that the Mapzen team stitched together from SRTM, the USGS National Elevation Dataset, EU-DEM, GMTED and ETOPO1, so they cover the oceans too, and they are on AWS as open data. In the SRTM .hgt format that is 65,341 tiles, each a 3601 by 3601 grid of 16-bit integers, 25.9 MB apiece, 1.6 TB once unpacked. The unpacking was its own job: a script that gunzips in batches with a pool of workers, skips what is already done, and deletes each archive after it is safely extracted, because there was not room for both.

Together that is a little over two terabytes on one drive, and every point on land or sea has a number.

The path a request takes

<pre class="mermaid">flowchart LR B[Browser or app] --&gt;|HTTPS| CF[Cloudflare] CF --&gt; N[nginx on the web server] N --&gt;|Tailscale| D[Docker on the elevation server] D --&gt; O[OpenTopoData] O --&gt; T[(2 TB of tiles)]</pre>

The elevation server is not on the public internet. It sits on our Tailscale network, and the cloud server that runs geosq.com has an nginx site for elevationapi.geosq.com that proxies to its tailnet address. Cloudflare sits in front of that with TLS and a cache it mostly does not use. Three hops, and a typical request, two points with interpolation, comes back in 0.12 seconds including the trip across the Atlantic.

The API is OpenTopoData’s, unchanged. A request for a path with three samples, and its answer:

GET https://elevationapi.geosq.com/v1/mapzen?locations=42.43095,-71.10763|42.44173,-71.12168&samples=3

{
  "results": [
    {"dataset": "mapzen", "elevation": 27.0, "location": {"lat": 42.43095, "lng": -71.10763}},
    {"dataset": "mapzen", "elevation": 45.0, "location": {"lat": 42.43634, "lng": -71.11465}},
    {"dataset": "mapzen", "elevation": 66.0, "location": {"lat": 42.44173, "lng": -71.12168}}
  ],
  "status": "OK"
}

samples is the part that makes a profile: give it a path and a number, and it walks the great-circle line between your points and samples the terrain at that many places. Geo Elevation asks for one sample every hundred metres or so, between 50 and 100 per route.

What it changed in the products

The Geo Elevation app for iPhone, iPad and Mac talks to this server directly, no key in the binary, and so does the web version on geosq.com, straight from the browser, because the server allows cross-origin requests. The web tool got a proper panel this week: the numbers beside the chart, ascent and descent, highest and lowest, a picker for the dataset, and downloads as GPX and CSV.

Because the data is ours, the picker can say where a number came from, and a user who wants USGS’s 10-metre data for the United States, or EU-DEM for Europe, can switch to it; those still go through the public OpenTopoData instance, and they are next on the list to bring home. The Elevation Finder uses the same server for spot heights, and now hands its points to Geo Elevation for a profile between them.

What I would tell you before you do this

• The download is the work. Budget a weekend for ASTER and a few days of background transfer for Mapzen, and keep the logs, because something will need a retry.
• Disk is cheap and RAM is the cache. A 2 TB dataset is not a problem when the tiles anyone asks for fit in memory.
• Put the server behind something. Ours is not reachable from the internet at all; the proxy on the web server is the only door, and Cloudflare is in front of that.
• Decide who it is for, and say so in the config. This one serves our apps and our site: the proxy answers cross-origin requests only for geosq.com, and it rate-limits each client, so a stray script cannot take the profiles away from the people drawing routes.

The repository with the configuration and the download scripts lives next to the geosq.com code now, so the next dataset is a script and a restart away.