Discussion

• @alex_whedon Alexander Whedon on x

  Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - [v…

• @subquadratic @subquadratic on x

  The numbers behind the SubQ announcement: Speed: 52x faster than Flash Attention SWE Bench Verified: 81.8% Ruler (128K): 95% MRCR V2: 65.9% Get early access at https://subq.ai/

• @ashleymayer Ashley Mayer on x

  With so much capital concentrating in so few private companies, and Anthropic and OpenAI breaking all “startup” growth norms, it's easy to forget that we are still incredibly early in this AI wave. On that note, I am THRILLED @subquadratic is now out of stealth. This is a

• @zephyr_z9 @zephyr_z9 on x

  “early access” Scammy vibes If it's really a sub-quadratic sparse attention arch (SSA), then serving this should be really cheap No point in putting this behind early access

• @artemr Artem Russakovskii on x

  A 12-million-token context window at 1,000x less compute capable of coding for weeks at a time and filing hundreds of PRs in the process. 🤯 If you thought AI can run laps around us now, the rate of progress in the next few years will become exponential.

• @willdepue Will Depue on x

  Let's read the technical report. TLDR; No real answers on how their method works. Doesn't make me feel better about it. They seem to understand the problem: “[Attention] is expensive for the same reason: every query compares against every key. The result is an all-pairs

• @daniel_mac8 Dan McAteer on x

  SubQ is either the biggest breakthrough since the Transformer... > 52x faster than FlashAttention at 1mm tok context > 20x cheaper than Opus ...or it's AI Theranos. Requested early access so hopefully can investigate soon. [image]

• @willdepue Will Depue on x

  if youre really subquadratic homie why are you only serving 12M context. if its n log n or n^1.25 let's see some 100M at least for a demo my guy

• @phequals7 @phequals7 on x

  does not pass my smell test.. > a breakthrough like this gets published at ICML/NeurIPS/ICLR - not with a startup launch video - would love to read a preprint atleast (technical report coming soon is v SUS) > usual suspects engagement boosting this tweet was the final straw

• @dorialexander Alexander Doria on x

  Very welcome to see more research in that space but a bit puzzling until the report clarifies: *Clearly a continuous pretrain of an open weight model (totally fair for this but we'll need a before and after). *No actually long evals (>1M) even though RULER could be extrapolated.

• @willdepue Will Depue on x

  nevermind no longer trying to give them the benefit of the doubt here: they claimed O(n) and ‘subq is linear vs quadratic’ which is pretty ridiculous the speedup numbers in their announcement video don't seem to line up with this? and just 12M context with O(n) scaling? this is […

• @willdepue Will Depue on x

  my first take, and a good lesson on good research epistemics here: what can we infer from ~82% SWE-Bench? it's possible they (1) they trained a new model, from scratch, that is unlike a regular transformer but i've never heard of this company before, and checking their funding [i…

• @tenobrus @tenobrus on x

  sub 5% chance we hear anything about this model ever again

• @nielsrogge Niels Rogge on x

  After checking his LinkedIn, the chances of it being a scam went up subquadratically [image]