GitHub - soofi-project/Soofi-Pretraining: Scripts, configs and documentation for the pre- and midtraining activities of the Soofi Project. · GitHub
/" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Uh oh!
There was an error while loading. Please reload this page.
soofi-project
Soofi-Pretraining
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star<br>18
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>557 Commits<br>557 Commits
data_ablations
data_ablations
docker
docker
documentation
documentation
examples
examples
hpc-tests
hpc-tests
midtraining
midtraining
notebooks
notebooks
pretraining
pretraining
scripts
scripts
singularity
singularity
src
src
tests/preprocessing
tests/preprocessing
.env.example
.env.example
.gitignore
.gitignore
README.md
README.md
View all files
Repository files navigation
Soofi Pretraining
This repository contains scripts, documentation, debugging, etc. of the pretraining for the Soofi project. The current code release contains the complete codebase used for training Soofi S a 30B model based on Nvidia's Nemotron 3 Nano architecture.c
The relevant training code can be found in:
Preprocessing
src/preprocessing — corpus construction, filtering, tokenization, and mixture assembly
Training
pretraining/08_nemotron_3_nano_on_20T
midtraining/02_nemotron_3_nano_annealing
midtraining/03_nemotron_3_nano_more_annealing
midtraining/04_nemotron_3_nano_final_annealing
midtraining/01_nemotron_3_nano_long_context_training
Open model weights coming soon.
About
Scripts, configs and documentation for the pre- and midtraining activities of the Soofi Project.
Resources
Readme
Uh oh!
There was an error while loading. Please reload this page.
Activity
Custom properties
Stars
18<br>stars
Watchers
watching
Forks
forks
Report repository
Releases
No releases published
Packages
Uh oh!
There was an error while loading. Please reload this page.
Contributors
Uh oh!
There was an error while loading. Please reload this page.
Languages
Shell<br>38.3%
Jupyter Notebook<br>35.1%
Python<br>26.5%
Dockerfile<br>0.1%
You can’t perform that action at this time.