Boston Harbor was 15,728 meters high

This week I asked our elevation server for the height of a point on the HarborWalk in Boston’s Seaport, a few steps from the water. It answered 15,728 meters. That is about 51,600 feet, nearly twice the height of Everest, at a spot where the ground sits about a meter above the harbor.

Geo Elevation’s profiles come from that server, on the web and in the app, and the data in question is their default. So do the contour, slope, flood and viewshed maps on geosq.com and the heights GPX Explore adds to a track. The first question was how much of the world looked like this. The answer: about 180,000 points, nearly all of them on American shorelines. They are fixed now. This is what they were, where they came from, and how I repaired them without taking the server down.

Two shaded relief maps of Boston Harbor drawn from the elevation data. Before: dotted red lines trace the shores of the inner harbor, the rivers and the islands, and a point on the Seaport waterfront reads 15,728 m. After: the red is gone and the same point reads 1 m.

The red marks every place where the server’s answer was off by more than 10 meters.

One bad number, and everything around it

The data under the server is Mapzen’s terrain tiles: 65,341 squares of one degree by one degree, each a grid of 3,601 by 3,601 heights stored as 16-bit whole numbers. The pixel nearest the HarborWalk point says 0, sea level, which is right. Its neighbor says 32,767.

The server doesn’t hand back a single pixel. It blends the four around each point, which keeps a profile smooth instead of stepping every 30 meters. Blend 32,767 with three zeros and you get a mountain, so one bad pixel turns everything within about 30 meters of it into a peak of anything from a few hundred meters to 32 kilometers. Here is a line from Bunker Hill to Castle Island, almost six kilometers across the harbor, sampled every five meters:

Two elevation profiles of the same line from Bunker Hill to Castle Island. Before: seven spikes rise from zero, the highest to 27,494 meters. After: the line stays between minus 11 and 24 meters, dipping below zero where it crosses the harbor.

Seven spikes above 100 meters, the highest at 27,494. Whether one of your routes hit a spike depended on where its samples happened to fall. When one did, the chart went off the top and the total climb was wrong by kilometers.

Every bad height on the planet

To see how far it went, I read every tile, all 1.6 terabytes, looking for heights that can’t be true: anything higher than Everest or deeper than the Mariana Trench.

181,287 pixels held exactly 32,767, in 42 tiles, and every one of them is American. The worst was the coast around Savannah and Hilton Head, 72,122 on its own. Then Cape Canaveral, Long Island Sound and Rhode Island, Florida’s Big Bend and the Gulf coast, Cape Cod, New Hampshire and Maine, Boston with 1,781, Lake Superior, a few rivers in New York and Vermont, and the west and north coasts of Alaska. The last 977 are on Saipan, a US territory in the Pacific.

Another 216 pixels were impossible in other ways, 120 of them in the highlands of western Guatemala. And eight tiles weren’t the right size at all. That one was mine; more below.

Why 32,767

32,767 is the largest number a signed 16-bit integer can hold, so a pixel that says it is a number that overflowed. Mapzen built these tiles in 2016 and 2017 by blending public sources: SRTM from the Space Shuttle, the USGS’s elevation data for the United States, ETOPO1 for the seafloor and others. Where two sources of different resolution met, the blend came out as infinity, and infinity stored in 16 bits is 32,767. That is why the bad pixels trace shorelines and riverbanks: that is where the data for land ends and the data for water begins. That nearly all of them are in the United States probably comes from the finer USGS data meeting everything else there.

The cause was worked out in 2017, in an issue on Mapzen’s tile builder that is still open. Mapzen closed at the start of 2018, and the tiles, still hosted on AWS as open data, have not been rebuilt. I checked it wasn’t our download: the Boston tile on AWS is byte for byte the one on our server. Anyone serving these tiles has the same seams. The public OpenTopoData server still answers 15,728 meters for the HarborWalk point today.

And one bug of my own

The eight odd tiles all sat just north of 50° south, and each was a fraction of the 25.9 MB a tile should be. Seven are open ocean. The eighth covers the fjords of Chilean Patagonia. Any point in them got an error instead of a height.

When I wrote about building this server, I said the script that unpacked the tiles “skips what is already done, and deletes each archive after it is safely extracted.” It decided what was done by whether the tile’s file existed. When a run was stopped partway through writing a tile, the next run found the file, called it done and deleted the archive, leaving half a tile and no original. Now the script writes each tile under a temporary name, checks that it is exactly the right size, and only then gives it its real name.

The fix

The eight short tiles came down again from AWS, whole.

For the rest, each impossible height became the average of the good heights around it, working in from the edge of each patch. The seams are one or two pixels wide, 30 to 60 meters, so the ground on either side is the best estimate of the ground in between, and no patch needed more than two passes. Along the coasts the new values run from −40 to 102 meters. Nothing else in any tile changed: 181,503 pixels in 76 tiles were rewritten, out of about 850 billion.

The server opens a tile from disk for every request, so I didn’t have to stop it. Each repaired tile was written next to the live one and renamed over it, which happens all at once, so no request ever read a half-written file. The originals are kept, and any tile can go back with one copy.

Then I asked again. The HarborWalk point says 1 meter, and every point on a fine grid around it reads between 0 and 7. Random points along the coasts at Boston, Savannah, Cape Canaveral and Long Island Sound read like coastline. Patagonia answers again. A terrain map of Boston now runs from −11 to 67 meters.

If you saved heights near these coasts

Heights already saved keep the old numbers. If you have a Geo Elevation profile or a GPX file that got its heights from us near one of these shorelines, ask for them again: Get Elevation in Geo Elevation, Replace Elevation in GPX Explore.

What I’d tell you

  • If you serve Mapzen’s terrain tiles, look for 32,767. Reading every tile once is cheap next to hearing about it from someone whose walk along the harbor climbed 27 kilometers.
  • Never call a file done because it exists. Write it under a temporary name, check it, rename it.
  • Keep the original of anything you repair in place, outside the folder the server reads.
  • Check the data, not just the server. Ours answered every request in about a tenth of a second, and some of those answers were 32 kilometers high.
By: Gavi Narra on: