← All posts

Engineering · 2026.10.10 · 7 min

The Reconciliation Alert Fired Twice: Missed and Double-Counted Interest When Bitfinex Positions Switch Sides

I check user earnings with a ratio of interest actually paid to booked accrual; it fired twice, once undercounting and once double-counting, both because Bitfinex splits one position across credits and loans.

Reading
Theme
Size
Width

I check whether the earnings users see are computed correctly with a 7-day ratio of "interest actually paid ÷ booked accrual." This alert has fired twice: once at 0.970, with accrual missing from the denominator, and once at 0.488, with the same stretch of interest counted twice. The errors went in opposite directions, but the root cause was the same. Bitfinex splits a single lending position across credits and loans, and when a position switches sides it gets a new id but keeps its original opening time. Both times the automated triage got the reasoning wrong. What actually found the cause was reconciling the numbers until they converged.

How the reconciliation ratio works

The source of truth for "currently lent" is the position snapshot Bitfinex returns, not the bot's own order table. The order table is a record of what the bot did. Cancellations, partial fills, and manual operations can all leave stale rows behind, and patching every kind of leftover just means maintaining a secondary projection.

Bitfinex pays interest once a day, already net of its 15% fee. On my side, from the position snapshots, I spread each position's amount × rate × seconds across the days it covers, second by second, to get booked accrual, and divide the two:

funding_recon_ratio = total interest actually paid over 7 days / total booked accrual over 7 days

A clean account should land around 0.85. If the ratio stays outside its band for 3 days in a row, the alert fires. Too low means accrual is overcounted, and too high means a position is missing. The ratio doesn't depend on the size of the amounts, and there's one per account per currency, so when an account starts drifting I know where to look.

One position, two homes

Bitfinex exposes lending positions through two endpoints, /funding/credits and /funding/loans. Money the borrower hasn't used yet sits on the loans side, and once they start using it, it moves to the credits side. The two sides have separate id namespaces, so a position that switches sides gets a new id, but it keeps its opening time MTS_OPENING and its amount doesn't change.

Closed positions go into each side's hist. The credits hist is written to my credit history table and the loans hist isn't, because the id namespaces differ and mixing them would pollute the ledger.

After the 120-day bid-take lane launched, most of its fills landed on the loans side. Both of the bugs below have to do with that side.

Structure diagram: one lending position lives on either the loans side or the credits side, and a side switch gives it a new id while keeping MTS_OPENING and the amount; most 120-day bid-take fills land on the loans side. Closed positions go to each side's hist, and only the credits hist is written to the credit history table, since the id namespaces differ.

First time: 0.970, closed positions vanishing from the books

About ten days after bid-take launched, the alert fired for three days on one production account, with a ratio of 0.970. The automated triage pointed at "the daily materialization job computing pro-rata wrong for the new lane." It was half right about the direction and wrong about the mechanism.

The position snapshot mirrors live positions, and when a position leaves the exchange, its row is deleted. On the credits side the position later reaches the history table through hist. On the loans side it doesn't. So the instant a bid-take position closed, the accrual for its entire lifetime disappeared from the books, which undercounted the denominator and pushed the ratio up. There happened to be five such positions in the window. I added them back second by second, and the ratio went from 0.970 to 0.851. The difference was exactly those five, so there was no need to look for a second cause.

The fix was a closed-position snapshot table. Before deleting a departing position, the job records its amount, rate, opening time, and closing time, and the daily materialization job reads that table as a third source. The one thing to get right was that a side switch must not count as a close. When a position moves from loans to credits, the old id disappears and a new one appears, which looks exactly like a close plus a new open, and writing it to the closed table would overlap with the new position. So if one id leaves and another arrives in the same cycle with the same opening time and amount, it's treated as a side switch and no close record is written.

I also considered having the materialization job remember the accrual it had already computed and only ever increase it, but that would also block genuine corrections later. It trades a visible error for an invisible one.

Second time: 0.488, the switched stretch counted twice

Two weeks later the alert fired again, on a different account, with a ratio of 0.488. This time the denominator was overcounted. The triage's theory was "the user paused lending and the snapshot froze," but both configs were active, so that didn't hold up.

I laid out that account's daily accrual day by day. From a certain day on, the number of open positions went from 1 to 2, and accrual exactly doubled. Then I matched up rows in the history table and the closed-position snapshots that shared an opening time and amount. A 120-day position had switched from credits to loans. The old credit's hist covered the stretch from opening to the switch, and the new loan started accruing from the opening time all over again, so every day in the overlap was counted twice. Switches aren't rare, either: of the 6,528 hist credits from the last 60 days, 1,452 came from switches.

The other direction had a problem too. The switch detection I'd added the first time let one new id swallow every old id with the same key. Equal-sized positions from the same batch of fills often switch together, so only the first was recognized as a switch and the rest were recorded as closes.

Bitfinex already tells us which endpoint a position came from. I just hadn't stored it. The fix was to add a from_loans column to both snapshot tables. For rows from the loans side, the materialization job looks for a hist credit with the same opening time and amount, and only starts accruing after that credit ends; if one of them is still open, the whole stretch isn't counted. That single rule is correct in both directions. Switch detection was also changed so that one new id matches only one old id.

Illustration: the old credit id's hist covers opening to the side switch; before the fix, the new loan id accrued from the opening time, so the stretch up to the switch overlapped and was counted twice; after the fix, the new loan only accrues after that hist credit ends.

After recomputing, the 7-day ratios for the three affected accounts:

Dot chart of 7-day reconciliation ratios before and after recomputing: account A from 0.734 to 0.855, account B from 0.488 to 0.914, and account C, not measured before, at 0.877, all back near the clean-account level of about 0.85.

None of the loans close rows backfilled on production collided on key with a hist credit, so production wasn't hit by this bug.

What I learned

  • Treat triage conclusions as leads only. The first time it guessed the right lane but the wrong place where the math broke. The second time it was wrong about what was broken at all. Following its advice would have meant changing code that wasn't broken, both times.
  • For reconciliation bugs, get the numbers to converge before touching anything. Only when adding back five positions landed exactly on 0.851 was I sure there was no second cause.
  • MTS_OPENING is the economic opening time, not the birth time of that id. Anywhere you use the opening time as the start of accrual and also sum across multiple ids, ask whether the same position could show up twice under different ids.
  • Store whatever the exchange already tells you the first time around. Reconstructing it later means guessing from times and amounts.
#System Design#Debugging#Reconciliation#Bitfinex

Letters

Loading comments…

Next

Investing

Backtest Numbers Lie: The Pitfalls I Hit in Strategy Research

A pretty backtest doesn't prove a strategy works; these are the pitfalls I hit, from TradingView understating drawdown to missing random baselines, dirty data, short samples, and lookahead, and how I avoid each.