The Bot That Lied by Being Logged Out

Every dashboard said fine. The broker connection had been dead for two hours.

The day I moved my bot's broker gateway from a tmux session to a proper systemd service, I created the most educational outage of the whole project, and the scariest part is how good everything looked while it was broken.

The gateway process: systemctl is-active said active. The app container: Up (healthy). The website: rendering price charts, candles moving, beautiful. Every signal an operator normally checks said all good.

And then one sceptical question, not from any monitor, from a human looking at the site: "is this coming from the broker API right now, or is it just what's stored in the database?"

We wrote a five-line check that asks the gateway itself for its state. The answer:

init connect fail: msg=SMS verification code required

Logged out. For two hours. The app had been serving cached candles from PostgreSQL the entire time: real data, faithfully rendered, quietly stale.

Anatomy: how a service migration logs you out

The broker gateway (moomoo's OpenD) authenticates a device fingerprint. Killing the tmux session and starting fresh under systemd, plus a restart test, looked like a new device, so the broker demanded SMS re-verification.

Here is the trap: systemd gives a service no stdin. The gateway had no way to ask anyone for a code. So it did the worst possible thing, from a monitoring perspective: it stayed up. Process running, port listening, unit active, logs unreadable (binary), and zero authentication behind any of it.

This is a general failure class, not a moomoo quirk: running but not authenticated is invisible to every naive health check. Process checks pass (it runs), port checks pass (it listens), end-to-end checks pass (the app serves cached data). Any dependency with an expiring session (broker, OAuth token, VPN tunnel, license server) can fail exactly this way.

The only real check: ask the dependency about itself

The fix for the blindness is a check that interrogates authentication state, not liveness:

from moomoo import OpenQuoteContext
 
ctx = OpenQuoteContext(host="127.0.0.1", port=11111)
ret, data = ctx.get_global_state()
print(data)   # want: qot_logined: True, trd_logined: True, program_status_type: READY
ctx.close()

If it prints qot_logined: False or the SMS error, the dashboard's opinion is irrelevant. (A small trap inside the trap: running this through docker exec WITHOUT -i forwards no stdin, so piping the script in produces empty output and exits clean. An empty result is a broken command, not a healthy system. Verify your verifier.)

Recovery was a one-time device re-trust: stop the service, run the gateway in the foreground where it HAS a terminal, request a fresh code, type it, watch the permission table print (that table is the proof of login), then hand it back to systemd. The final step mattered most: restart under systemd once more and re-run the state check, proving it now logs itself in with no human and no SMS. Migration not done until the unattended path is proven.

The monitoring lesson, promoted to a rule

My monitoring stack gained its first application-level alert that day, and the choice of metric came straight from this outage:

alert: opend_logged_in == false

Not "is the process up" (it was, the whole time). Not "does the site load" (it did, beautifully). The alert watches the exact thing that silently failed: authenticated, live connection to the upstream.

Three portable rules from two blind hours:

1. Uptime checks are not freshness checks. A serving system can be a well-preserved corpse. Somewhere, something must verify data is CURRENT and the session is LIVE.
2. Cache makes outages cosmetic. Whatever makes your app resilient to upstream blips also makes upstream death invisible. The stronger your caching, the more you need an explicit upstream-truth check.
3. The best health check asks the dependency about itself. Every serious dependency has some get_state equivalent. Wire the alert to that, not to the process table.

The question that caught it ("API or database?") cost nothing and needed no tooling, just the refusal to accept a downstream symptom as proof of an upstream truth. Monitoring exists to ask that question every minute so a human does not have to get lucky.

Pictures to add later

1. Terminal: systemctl status showing active (running) next to the failing get_global_state output (the lie, side by side)
2. The Grafana alert rule for opend_logged_in
3. Optional: the permission table printed at successful login