AI is bigger than the brain - Thresholds and the singularity in the second quarter of the 21st century

One of my interests in mathematics - at least the everyday sort of mathematics and numerology I think about - is the idea of identifying when a threshold is being passed. I try to look for social proofs of when a threshold is being passed, but also for forward looking indicators. Passing a threshold indicates that the world has changed. Walking around nothing changes, but once you go through a doorway you are in a different place. Another analogy might be a phase change. Most of our thresholds are around our human experience because we are the species who dominate planet earth with our build and agricultural environment. It is clear, to both me and a lot of people, that the world is going though a large number of thresholds and phase changes. they are going to change the world very quickly in the next few years. I think this is the beginning of the so called singularity.

To make a short list - many AI tools -for example chat models, drawing models, self driving cars, Evs and solar energy vs oil production, humanoid robots, but also population growth is reaching limits. The big change is that we are passing the threshold of humans being the smartest things on the planet to being the second smartest things, by a factor of about 100 to 1000 fold. Whilst I am not sure that intelligence is 100% the driving thing in the world, due to physical limits, its also not clear if I would hire a human for intellectual work if I could hire an AI that was 10 times faster at 1/10th the cost only when I needed that role. Of course in reality it will be the AI in the virtual hiring department making the decision (at one 100th the pay of 5 humans who were replaced).

For a first threshold, how many connections does the human cortex have and roughly how many computational operations per second (OPs) is it roughly performing. Recently a 1mm3 cube of human cortex was fully analyzed suggesting ~70x10^12 synapses and 9x10^9 neurons in the human temporal cortex.

Shapson-Coe and colleagues found “57,000 cells, about 230 millimeters of blood vessels, and about 150 million synapses” in 1mm3 of cerebral cortex. For a cortex of 480mm x 480mm x 2mm (nicely described fairly deep in this blogpost from Wait but why on Elon Musks neuralink: https://waitbutwhy.com/2017/04/neuralink.html#part2) would suggest 460,800 mm3, 26,265.6 × 10^6 cells (about 1/3rd neurons) and 69 ×10^12 synapses - perhaps equivalent to a artificial “parameter”? the brain probably runs at ~20hz - with the fastest neurons at ~200hz, ,but generally being less hectic than that.

from Alexander Shapson-Coe et al. A petavoxel fragment of human cerebral cortex reconstructed at nanoscale resolution.Science384,eadk4858(2024).DOI:10.1126/science.adk4858

So how does this compare with say an nvidia gpu or chat GPT 4? There is a particularly prophetic chart in Ray Kurzweil’s 2005 book “The singularity is near”.

From Ray Kurzweil’s 2005 book “The singularity is near”. Based on a meta analysis of studies at the time he suggests that the human brain performs the equivalent of about 1×10^15 calculations per second. The latest Nvidia gpu now exceed this at just about $1000 for 1×10^15 tensor OPs.

A Hardware threshold. Single GPUs ($1000) now surpass the ~1x10^15 OPs of a single human brain processing power. Do these numbers from today’s morphology and 2005 estimates agree? Well 70x10^12 synapses running at 20hz would be 1.4x10^15 OPs, so yes I think roughly in agreement. An Nvida GPU of course runs at 2GHz so problems have to be split up into smaller pieces running 100,000 X faster. Hardware wise, we are well past the capability of the human brain today - and equal to it with the best GPUs in gaming PCs at home. For perspective California’s minimum wage in 2025 is $16.5/hour - or if you get a full time position at that rate, $33,000 direct, probably about $45,000 after health insurance and other taxes and 401k. i.e. Human compute at minimum wage is ~40x more than artificial compute (especially as they were likely working on a computer anyway and it just needs an upgrade)

How close are the software models in scale? - well I don’t claim to really understand, but Metas llama 3 large model (which is open source and thus easier to say something about) has 405 billion parameters (4x10^11) compared with 7x10^13 synapses in the human brain, but running 100,000X faster. It was trained using 30 million gpu hours on H100-80GB Nvidia hardware. 30 million hours is 3,424 years. (or 3 months of 15,000 GPUs) It is probably being used by millions of people (although I haven’t played with that one much yet.) Elon Musk’s and X’s “Grok” 3 (I guess he is a Heinlein fan for the stranger in a strange land reference?) is rumored to be using 100K GPUs and trying to scale to 300K GPUs and trillions of parameters, so bigger than human neural networks and 100,000 faster. (update - I originally wrote this post a little while ago, and llama 4 is now out with 4 trillion parameters (2x10^12)

Model card at Hugging face for Llama 3.1 https://huggingface.co/meta-llama/Llama-3.1-405B,

So the human brain has been surpassed in everything but results…. How long do we have, and how much better than humans will it be?

Well I can’t predict when trusted AGI will come, but my guess is soon - within the new US Trump administration term. (An extra 250K people die in more red states than blue states due to covid vaccine denial and Trump gets in with a vaccine denier for the human health department - maybe we should be working on human stupidity?) but snark aside, I think the rise of AI will be far more revolutionary in terms of societal change than a Trump administration, even if that adds some amount of chaos to the equation (I did have invade Canada on my bingo card for the first Trump term as a joke, but not also Panama and Greenland). Anyway - what are the forces that are going to speed this up?

  1. competition - I can think of a number of companies competing to release the best models annually right now. Open AI, Google (US and UK Deep mind), X, Meta, DeepSeq from China, and perhaps also Nvidia - but more for industrial uses. Other races include self driving cars - which Tesla seems to have won recently passing the trust barrier with FSD 13.2 on hardware 4 - see below.

  2. massive research in going beyond internet data to have models learn how to make there own training data and go beyond what is currently known.

  3. continuing cost reductions on GPUs and algorythms combined with training scale.