451: Unavailable for legal reasons

The screenshot that started it showed a confluence ring reporting not enough data. That is a legitimate state — a symbol with too few candles cannot be scored — so it read as thin data rather than as an outage. Behind it, a panel was printing the actual answer: Binance returned 451.

HTTP 451 is Unavailable For Legal Reasons. Binance answers it to any request from a US IP address. Our API runs in Railway's sfo region. So every chart, every screener result and every crypto quote the API served had been failing since the day it went live, and the interface had been describing the failure as a data shortage.

Why nothing caught it

Because nothing we tested against was in the United States. Local development runs in Europe. The Electron desktop app makes its market-data calls from the user's own machine. The test suite hits Binance directly from wherever it runs. Every one of those paths is a non-US IP, and every one of them worked.

The single US-hosted component in the system was the production API — the one thing no test exercised from the outside. This is worth naming as a class rather than an incident: an environment difference that is not code cannot be caught by reading code, and a test suite that runs from your laptop is testing your laptop's network position too.

Two fixes rejected, on purpose

The obvious fix is to move the API to Europe. It was rejected: Postgres also lives in sfo, and moving the API alone would put a transatlantic hop on every database query in the product. That trades one broken feature for a permanently slower everything.

The second obvious fix is to swap Binance for an exchange that answers US IPs — Coinbase, Kraken. Also rejected. That means rewriting the symbol format, the kline schema and the entire screener, and it is a rewrite of the data layer to solve what is fundamentally a routing problem. The data was never wrong. The request was leaving from the wrong place.

What shipped: one service, in the right country

apps/binance-proxy is a small Node service pinned to europe-west4. It is now the only thing in the system that talks to Binance, and the API reaches it over Railway's private network at binance-proxy.railway.internal:8080. Postgres stays where it is. The API stays where it is. Only the requests that Binance refuses go anywhere.

The important design decision is that it is deliberately not a general-purpose relay. A service whose job is “forward things to Binance” is one misconfiguration away from being an open path to authenticated endpoints, so it forwards exactly nine public paths and nothing else:

/** Public market-data endpoints only.
 *  Nothing here can move funds or read an account. */
const ALLOWED_PATHS = new Set([
  "/api/v3/klines",
  "/api/v3/ticker/24hr",
  "/api/v3/ticker/price",
  "/api/v3/ticker/bookTicker",
  "/api/v3/depth",
  "/api/v3/trades",
  "/api/v3/exchangeInfo",
  "/api/v3/avgPrice",
  "/api/v3/time",
]);

GET only. One upstream host. A shared secret compared in constant time on every request. No public domain at all. An authenticated path such as /api/v3/account is refused with a 403 — verified against the deployed service, not assumed from the code.

Two things that were not obvious

The first was caught by a smoke test rather than by reasoning, which is the honest way to report it. The ws library hands every frame over as a Buffer regardless of what it was on the wire. Relaying that without carrying the isBinaryflag turns Binance's text JSON into a binary frame, which arrives at the far end as a Blob and breaks JSON.parse — a live price stream that connects successfully and then delivers nothing usable.

const relay = (target) => (data, isBinary) => {
  if (target.readyState === WebSocket.OPEN)
    target.send(data, { binary: isBinary });
};

The second is a protocol constraint with a security consequence. A WebSocket handshake from a browser-style client cannot carry custom headers, so the shared secret cannot travel in one. It travels as a query parameter instead — and is deleted before anything is forwarded, so it never reaches Binance and never appears in an upstream log.

// The secret travels in the query string for the WebSocket
// path, which cannot carry headers; strip it either way so
// it never reaches Binance.
url.searchParams.delete("s");

A service that reports its own reachability

Here is the awkward property of the design: the proxy has no public domain, which is exactly what makes it safe — and also means there is no way to check from outside whether its egress IP is still one Binance will answer. If it ever drifts out of its region, every chart in the product goes down again, silently, in precisely the way that took a production screenshot to notice the first time.

So it says so itself, on boot:

if (res.status === 451) {
  console.error("EGRESS BLOCKED: Binance returned 451 — " +
    "this service is not in a permitted region");
} else {
  console.log(`egress ok: Binance /api/v3/time returned ${res.status}`);
}

One line in a deploy log now answers the only question this service exists to answer. A monitoring system you have to remember to check is worse than a log line that appears every time the thing restarts.

Off by default

Every call site routes through one switch. With BINANCE_BASE_URL and BINANCE_WS_URL unset, the calls go direct to Binance exactly as they did before — so local development, the desktop app and the test suite are all untouched by the fix.

export const binanceRestBase = () =>
  process.env.BINANCE_BASE_URL || "https://api.binance.com";

Verification was done against production rather than a local build, because a local build is the exact thing that lied last time: /api/market/klines returned real BTCUSDT bars, /api/market/screen returned a ranked USDT book, and a live 1m bar arrived over the socket — proving the REST and WebSocket halves both traverse the proxy.

The part worth keeping

Geography is a dependency. It has no import statement, does not appear in a lockfile, and no amount of reading the code will reveal it — but a service's network position determines which APIs answer it, and moving a deploy between regions can break a feature without changing a line.

The other lesson is smaller and more uncomfortable: the interface reported the outage as not enough data. A failure state that renders as a plausible normal state is a failure state you will not investigate. That is the same principle behind the guardian alerting when it fails, and it earns its place in more of this codebase every month.