Censorship attack combines packet injection and Heartbleed
We all know there is censorship online. It happens in China. It happens to "terrorists". But we don't believe it will happen to us.
As Eben Moglen and Kaspersky have pointed out, companies developing crypto are prime targets no matter where they are. So you don't have to be a bad guy for the NSA to attack you. You just have to protect people from the NSA. Even protecting yourself is often enough. NSA prefers their victims to be defenseless.
Detection in the wild
In early 2015 people were still downloading our ISO file for GoodCrypto. But suddenly installations stopped.
After a lot of checking we noticed that the downloads got HTTP 200 result codes, but the lengths were all too short. This isn't supposed to happen. A 200 result means success. These weren't successful downloads, but the web logs said they were. Ordinary log checks didn't show the bug.
Finding the vuln
Downloads from goodcrypto.com to goodcrypto.com worked. Downloads from another site at a different datacenter in the same country worked. A little further away in the network, downloads failed but the server logged a "Success" status code.
The obvious answer was a server misconfiguration. We couldn't find one. A server side packet dump showed the client just dropped the connection in the middle of the download.
We couldn't get a browser to download the whole ISO file. The browser thought it came in fine, but the file was incomplete.
So was it a browser bug? We tried other browsers. They couldn't download either.
The wget program often helps debug downloads. It doesn't have the same malware issues as browsers, because wget doesn't support malware vectors such as javascript, java, and css. It also retries failed downloads, and often tells you why it failed.
When we tried wget, it detected errors, retried, and finally succeeded. It said the error was a bad length field in a TLS packet. That didn't make sense at first because we thought TLS packets were error corrected by TCP.
We searched for other bug reports like this. They were all during session initiation, not in the middle of a long download.
But our searching led to Heartbleed. Modifying SSL/TLS length fields is exactly how it works.
Wasn't Heartbleed fixed in 2014?
In 2014 when we all heard about Heartbleed many servers were vulnerable. But OS providers fixed it fast. Our own servers get regular security updates.
How were we seeing it now?
Servers around the world were fixed fast, but clients were vulnerable too. The Heartbleed news coverage was all about servers. Servers got fixed. Many clients didn't. A client side Heartbleed attack is sometimes called Reverse Heartbleed.
Packet evidence shows MITM
Was our server cracked? We're pretty careful, so that doesn't happen often. But we checked, and checked again. Even though we don't usually have any packet logs, we ran download tests with simultaneous packet logs on both the server and client.
The server packet logs showed an ordinary number of bad packets, all error corrected. During the download the client dropped the connection.
The client packet logs were very different. There was a surprising number of bad incoming packets. Almost none of these bad packets showed in the server logs for the same session. These packets appeared to be injected into the packet stream. This is an MITM/MOTS attack, specifically a packet injection attack.
Finally wget reported that a TLS packet with a bad length field got through and caused the TLS connection to break. Now we knew this was a MITM variant of Reverse Heartbleed.
Working around the censorship
We added simple instructions to our Download page:
Somebody (Hi NSA!) is trying to censor GoodCrypto downloads.
But don't worry. The workaround for their super duper advanced network attack is: Just use wget.
If you're on windows you need wget.exe.
At a command prompt:
wget --no-check-certificate (URL appeared here)
Because you will verify the file hash, "--no-check-certificate" is ok in this rare case, and Windows needs it.
We also strengthened the encouragement to verify the file.
False Flag
The attacker injected packets by forging our site's IP address. If a site visitor notices the packets, they will think that the attack is from us. The attacker didn't just shift blame away from themselves. They framed someone they don't like. Faking the evidence to blame an attack on someone else, especially someone the attacker doesn't like, is a classic False Flag operation.
An MITM attack like this requires impersonation, so the benefits to the attacker of a False Flag are built right in.
One way to censor anyone is to attack their reputation.
Censorship by a nation state
Even though the download broke, the browser didn't complain. It looked like a successful download. The server showed an HTTP 200 result code. Neither the client nor server detected the attack.
Server packet logs didn't show anything unusual except the abrupt client disconnection. The attack didn't show at all during downloads from a client that was close to the server in network space, such as another nearby datacenter. It wasn't an attack on the server or the hosting provider.
We set up another server in a different country and the attack continued. It wasn't an attack by the hosting country. We were nowhere near our transfer limits for either server, so it was not traffic shaping by hosting providers.
The clients were all over the world. It wasn't separate attacks launched from close to individual clients.
The attack appears to be from someone sitting on the net pipes and injecting packets. This requires huge resources. The U.S., U.K., Canadian, and Chinese spooks do this. Ordinary criminals don't have the ability.
Because the goodcrypto.com servers are in Europe, China is an unlikely suspect. GCHQ and CSE are dependent on the U.S. for their QUANTUM capability. That means it was likely an NSA attack, either directly or by proxy.
But which nation really doesn't matter. There is no known way to protect against specific nations. You have to protect against all or none.
Who is vulnerable
Anyone who publishes on the web is vulnerable to this form of censorship. Even if you just use HTTP, the Chinese censorship method of a simple RST works.
Update: The attack is now intermittent. When it's paused the attackers may be avoiding forensics, or just changing tactics.
People have asked what wget reported when it detected the error. It consistently looked like this, with the timestamp, speed, and byte displacement varying:
YYYY-MM-DD HH:MM:SS (N NB/s) - Read error at byte N/334495744 (A TLS packet with unexpected length was received.). Retrying.
If you are comfortable generating a pcap, please capture your iso download. If the file doesn't verify, publish your pcap publicly. Because the connection is TLS your pcap won't have readable content, but can still reveal details such as where you are. So be sure you're comfortable with sharing this information.