Percy Liang on X: "🚢 Marin 535B-A23B started training this week! As usual, the whole process is open.<br>Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow.<br>Before kicking off the run, we trained a 4-rung" / X<br>Post

Log inSign up

Post

Percy Liang on X: "🚢 Marin 535B-A23B started training this week! As usual, the whole process is open.<br>Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow.<br>Before kicking off the run, we trained a 4-rung"

Percy Liang

@percyliang

🚢 Marin 535B-A23B started training this week! As usual, the whole process is open.<br>Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow.<br>Before kicking off the run, we trained a 4-rung scaling ladder from 1.6B-A61M (48B tokens) to 27.7B-A1.2B (926B tokens) to debug issues, and to make a forecast of our hero run. This is by far our biggest run, so definitely expecting the unexpected.

span:not(:empty)~span:not(:empty)]:before:content-['·'] [&>span:not(:empty)~span:not(:empty)]:before:px-1 [&>span:not(:empty)~span:not(:empty)]:before:shrink-0">9:44 PM · Aug 21, 2026719.7KViews

95<br>526<br>3.7K<br>2.2K

span:not(:empty)~span:not(:empty)]:before:content-['·'] [&>span:not(:empty)~span:not(:empty)]:before:px-1 [&>span:not(:empty)~span:not(:empty)]:before:shrink-0 min-w-0 overflow-hidden">Percy Liang

@percyliang

Aug 21

In addition to forecasting our final loss, we can also forecast the loss of intermediate checkpoints, so we can see if we’re on track over the voyage:

306<br>35K

span:not(:empty)~span:not(:empty)]:before:content-['·'] [&>span:not(:empty)~span:not(:empty)]:before:px-1 [&>span:not(:empty)~span:not(:empty)]:before:shrink-0 min-w-0 overflow-hidden">Percy Liang

@percyliang

Aug 21

Look at our data composition:<br>storage.googleapis.com/marin-public/h…<br>Watch the run live on wandb:<br>wandb.ai/marin-communit…<br>See all the gory details on GitHub:<br>github.com/marin-communit…<br>Assembling this hero run really required data, architecture, infra, kernels to all come together and was a huge Show more

44<br>587<br>84K

span:not(:empty)~span:not(:empty)]:before:content-['·'] [&>span:not(:empty)~span:not(:empty)]:before:px-1 [&>span:not(:empty)~span:not(:empty)]:before:shrink-0 min-w-0 overflow-hidden">Nathan Lambert

@natolambert

Aug 21

congrats team! Excited that there's compute out there for non profits :) and will look forward to working with the model

101<br>10K

Log in or sign up for X<br>See what’s happening and join the conversation<br>Continue with phoneContinue with AppleContinue with Google<br>or<br>Log in with username or email

Relevant people

Percy Liang@percyliangFollow<br>professor of computer science @Stanford @stanfordnlp, co-founder of @togethercompute, https://t.co/7R5THVnJ6u, @simile_ai, pianist

Trending now