Pseudoscience
"The only way to test a hypothesis is to look for all the information that disagrees with it."
- Karl Popper
On March 12th, Roman Sterlingov was found guilty on all counts for allegedly operating the Bitcoin Fog mixer, one of the oldest Bitcoin custodial tumblers with a 10 year history.
Over the past year we, at OXT Research, have done our best to assist the defense in reviewing the aspects of the case related to blockchain analytics tooling, like the Reactor platform, used by the government in establishing primary evidence for the case.
This article will focus on the trial’s Daubert hearing. A Daubert hearing is meant to evaluate if evidence meets sufficient standards for admissibility. For weeks, experts of both sides testified on the subject with the court ultimately concluding that evidence provided by Chainalysis Reactor was the product of a scientific approach (link). This "conclusion" is problematic for multiple reasons and its impacts go far beyond the specific case of Roman Sterlingov.
To understand this issue lets recap the main arguments made by government experts during the Daubert hearing:
Argument 1
Heuristics used by Chainalysis Reactor to tag Bitcoin addresses are closed source. They are not peer reviewed and cannot be audited by Defense experts but they're "inspired" by academic works that can be peer reviewed.
Argument 2
Known limitations that apply to peer reviewed academic works, specifically with respect to the co-spend heuristic, do not apply to heuristics used by Reactor because counter measures have been implemented but the details of these countermeasures cannot be disclosed.
Argument 3
Chainalysis Reactor is a deterministic platform producing reproducible results.
Argument 4
The Bitcoin Fog cluster produced by Reactor is the result of 2 heuristics: an improved version of the Common Input Ownership Heuristic (i.e. all addresses used as inputs to a transaction are controlled by the same entity) and a second heuristic clustering addresses composing a peelchain (i.e. a long chain of transaction "peeling" small amounts of UTXOs for payments).
Argument 5
There is no estimates or logs of error rate for the heuristics used by Reactor but Reactor user feedback is generally positive.
Argument 6
A few manual tests (transactions sent to or received from the Bitcoin Fog's cluster) show that the heuristics used by Reactor are both accurate (true positive results) and conservative (false negative results).
Review of Arguments 3 & 4: “Deterministic” Platform Claims
During the first few years of operation the Bitcoin Fog wallets had the following basic behavior:
- users deposited bitcoins to a Bitcoin Fog address
- periodically deposits were consolidated into a single UTXO by a consolidation transaction ("Common Input Ownership Heuristic" is applied here)
- a peelchain processing the withdrawals was initiated from this consolidated UTXO ("Peelchain Heuristic" is applied here)
In the course of our analysis of the Bitcoin Fog cluster produced by Reactor, we have identified at least one instance of a peelchain for which the addresses associated with the 4 firsts transactions are included in the Reactor cluster but the many hundreds of addresses composing the rest of the peelchain are excluded from the cluster.
This result is odd because the description of the heuristics given by government experts (see Argument 4 above) implies that all addresses composing this peelchain should either be included in the cluster or excluded from the cluster.
There are two possible explanations for the excluded addresses:
- During the Daubert hearing, government experts misrepresented the functions of the closed source heuristics used to create the Bitcoin Fog cluster.
- Argument 3 above is false; the Reactor platform isn't deterministic and its results aren't reproducible (causes may include bugs, servers crashes with no sane recovery process, etc).
Because Defense experts haven't been able to review the inner details of the heuristics, it's impossible to say which one of these two hypotheses is correct. However, we expect that the latter is the most likely explanation. In both cases, the consequences on the conclusion of this Daubert hearing are hard to ignore
Review of Arguments 5 & 6: “Accurate” yet “Conservative”
Another very disturbing aspect of this Daubert hearing is that the scientific method has been completely ignored while considering that “hearsay” or anecdotal evidence of past successes are a replacement for tracking false positives and negatives.
In the scientific method, a hypothesis can never be 100% proven, only falsified. But Argument 6 completely disregards falsification which is the entire basis for the scientific method.
According to this argument it is impossible to have a test producing a negative outcome. The result will always be considered as "evidence" of a positive outcome (heuristic being described as "accurate" or "conservative").
Review of Arguments 1 & 2: Peer Review and Heuristic Shortcomings
Even more disturbing, these testimonies have completely ignored the cases that contradict these positive results.
For instance, no detail about Reactor’s Mt. Gox cluster have been provided. No explanation has been provided about how this cluster is either full of false positives (see CoinJoinMess) caused by limitations of the Common Input Ownership Heuristic that government experts have refused to acknowledge or how it has been "fixed" thanks to a so-called "Heuristic 3" that is basically just a code name for manual tagging of Bitcoin addresses by Chainalysis analysts.
Conclusions
In conclusion, what may sound like just another anecdotal announcement from the government has profound consequences for all crypto users.
From your self-custody to your spending preferences, the developments in this case have massive far reaching implications that should cause concerns for every crypto user.
Lastly, we find the silence from academics during these Daubert hearing to be very disturbing.