DeepMind: The Half of Biology only AlphaFold could touch

For 26 pain-staking years, John Moult had run a contest based on the idea that nobody could predict the 3D structure of a protein solely from its amino-acid sequence - potentially the most mind boggling and profound unsolved problem in biology. This competition to re-define humanity was called CASP; he had founded it in 1994 due to computational structure prediction for proteins being a mess of unverifiable claims. What John wanted was a clinical trial framework for the truth. Unfortunately 13 rounds of his contest had ended in a plateau.

Then at 10 a.m EST, Moult began. However, instead of showing the slide that everyone in the room was waiting for, he started walking them through history. CASP1, 1994. CASP2, 1996. CASP3. Round by round. The audience already knew something was up as the results had quietly dropped onto the CASP14 website hours earlier and Twitter was in meltdown. However nobody had seen the actual graph yet. Moult kept building, tiny increments at a time. Years of plateau. CASP9, CASP10, CASP11 without that line of history moving a single bit. And then he showed it. For 26 years, that line had crept upward in fractions of a point. At CASP14, it stood up vertically off the page. The problem had fully collapsed.

Just to put into perspective how crazy this achievement was, according to the combinatorics of this problem there is no imaginable circumstance where it should have been solvable in the first place. In 1969 there was a molecular biologist called Cyrus Levinthal who pointed out that a protein which had 300 amino acids/residues had roughly 10^300 possible 3d configurations. Think about it like this, if you take every atom in the observable universe. Now imagine each atom in the universe contains its own complete observable universe of atoms. Now a step further, imagine each of those atoms in those universes also gets its own complete universe of atoms. That number is approximately 10^240. Now multiply that by the number of Planck moments (shortest unit of time that physics allows ~5.4 × 10⁻⁴⁴ seconds) that have passed since the beginning of our universe. Then finally that is how many shapes a single protein could fold into. Alpha does this in a matter of seconds, crazy right?

So how did AlphaFold actually do this? Well the first property they exploited was data. There is a public archive of every protein structure that has ever been experimentally solved called the PDB (Protein Data Bank). Each entry is essentially the work of a PhD thesis requiring methods like X - ray crystallography, NMR, cryo - EM etc. By the time AlphaFold had come along, the PDB held roughly 200,000 structures which were all detailed to the atom (ImageNet of proteins). AlphaFold also got their hands on UniProt (a public database holding 250,000,000 raw amino acid sequences). After training, AlphaFold embarked on a mission to predict the structures of around 200,000,000 of those sequences and by 2022 they would accumulate a database that dwarfed the PDB by 3 orders of magnitude.

However, here's the thing, from a biological standpoint 200,000 structures might be massive in biology; but in terms of deep learning it's actually quite small. So that begs the question: how did AlphaFold get away with this? That brings us to the second technique called MSA (Multiple Sequence Alignment). How it works is, over hundreds of millions of years of evolution amino acids that sit in close proximity in the 3d structure tend to co-evolve meaning that if one mutates, its partner will also have to mutate (or the protein would lose its function and then the species would die out). So what AlphaFold does is that it takes the protein they want to find the 3d structure for and then they line its amino acid sequence against the same protein in thousands of other species.

That alignment becomes a 2D tensor (a grid of thousands of sequences on one axis, and amino acid positions in another). This is the raw material AlphaFold’s core, the Evoformer, works on. Alongside the MSA tensor, AlphaFold also has to maintain something called a pair representation (a matrix where every cell has a learned relationship between 2 amino acids in the 3d structure). So how does the Evoformer work? It alternates between two attention operations, row-wise attention within each sequence, and column-wise attention across species at each amino acid position. However the more crucial out of the 2 is column-wise attention. Essentially, how column wise attention works is that it computes Q/K/V (a query, key and value). Q = ("what I want"), K = ("what I have") and V = ("what I share"). Firstly the model takes a dot product between Q and every other species’s K. A dot product, to put simply, is a method to find the similarity between 2 vectors. Those scores then get softmaxed (a function that converts a list of numbers into weights that sum to 1, where the largest score gets the most weight). Each species's new representation then becomes a weighted sum of every other species's V vectors, with the most "relevant" species contributing the most with the result being each species having a richer representation based on what every other species had to share. Over thousands of homologous sequences at a single position, the model learns which positions vary together across evolution. Those co-evolution patterns get distilled into the pair representation, which converges, over many Evoformer blocks (more series of Q/K/V), into a learned map of residue-residue proximities in 3D space. However, here's the kicker, none of this attends to actual structural data. The model extracts structural information from sequence alone, performing unsupervised pre - training with an objective that aligns perfectly with the prediction of 3D coordinates.

So going back to that Zoom webinar on November 30th 2020, to quote Moult "in some sense, the problem is solved." Within months his first 3 words of that quote were unnecessary. By the end of 2022, DeepMind had released the AlphaFold Database with 200 million predicted structures, essentially every protein on Earth, freely available to anyone with a browser. Andrei Lupas, a structural biologist at the Max Planck Institute who had spent ten years trying to crack the structure of one stubborn membrane protein had fed it through AlphaFold and had an answer in thirty minutes. Four years later, John Jumper and Demis Hassabis of DeepMind would share the 2024 Nobel Prize in Chemistry, alongside David Baker (for his computational protein design). The fifty-year-old problem was over, because for once, every property the problem had was exactly the property deep learning needed. Evolution is what trained the model and AlphaFold just had to learn to read it.