GitHub - deepseek-ai/smallpond: A lightweight data processing framework built on DuckDB and 3FS. · GitHub
/" data-turbo-transient="true" />
Skip to content
Search/
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Uh oh!
There was an error while loading. Please reload this page.
deepseek-ai
smallpond
Public
Notifications<br>You must be signed in to change notification settings
Fork<br>454
Star<br>5k
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>3 Commits<br>3 Commits
.github/workflows
.github/workflows
benchmarks
benchmarks
docs
docs
examples
examples
smallpond
smallpond
tests
tests
.gitignore
.gitignore
LICENSE
LICENSE
MANIFEST.in
MANIFEST.in
Makefile
Makefile
README.md
README.md
pyproject.toml
pyproject.toml
View all files
Repository files navigation
smallpond
A lightweight data processing framework built on DuckDB and 3FS.
Features
🚀 High-performance data processing powered by DuckDB
🌍 Scalable to handle PB-scale datasets
🛠️ Easy operations with no long-running services
Installation
Python 3.8 to 3.12 is supported.
pip install smallpond
Quick Start
# Download example data<br>wget https://duckdb.org/data/prices.parquet
import smallpond
# Initialize session<br>sp = smallpond.init()
# Load data<br>df = sp.read_parquet("prices.parquet")
# Process data<br>df = df.repartition(3, hash_by="ticker")<br>df = sp.partial_sql("SELECT ticker, min(price), max(price) FROM {0} GROUP BY ticker", df)
# Save results<br>df.write_parquet("output/")<br># Show results<br>print(df.to_pandas())
Documentation
For detailed guides and API reference:
Getting Started
API Reference
Performance
We evaluated smallpond using the GraySort benchmark (script) on a cluster comprising 50 compute nodes and 25 storage nodes running 3FS. The benchmark sorted 110.5TiB of data in 30 minutes and 14 seconds, achieving an average throughput of 3.66TiB/min.
Details can be found in 3FS - Gray Sort.
Development
pip install .[dev]
# run unit tests<br>pytest -v tests/test*.py
# build documentation<br>pip install .[docs]<br>cd docs<br>make html<br>python -m http.server --directory build/html
License
This project is licensed under the MIT License.
About<br>A lightweight data processing framework built on DuckDB and 3FS.<br>Topics<br>data-processingduckdb<br>Resources<br>Readme<br>MIT license<br>Activity<br>Custom properties<br>Stars<br>5.0k stars<br>Watchers<br>49 watching<br>Forks<br>454 forks<br>Report repository
Releases
Packages
Used by
Contributors
Languages
You can’t perform that action at this time.