GitHub - el10savio/hnTrends: Explore trends in HackerNews Posts. A one time ETL pipeline using embeddings, Kmeans, DuckDB and exploration using Superset. · GitHub

/" data-turbo-transient="true" />

Skip to content

Search or jump to...

Search code, repositories, users, issues, pull requests...

-->

Search

Clear

Search syntax tips

Provide feedback

--><br>We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Cancel

Submit feedback

Saved searches

Use saved searches to filter your results more quickly

-->

Name

Query

To see all available qualifiers, see our documentation.

Cancel

Create saved search

Sign in

/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up

Appearance settings

Resetting focus

You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

{{ message }}

el10savio

hnTrends

Public

Notifications<br>You must be signed in to change notification settings

Fork

Star

main

BranchesTags

Go to file

CodeOpen more actions menu

Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit

History<br>5 Commits<br>5 Commits

data

data

docs

docs

pipeline

pipeline

view

view

.gitignore

.gitignore

Makefile

Makefile

README.md

README.md

config.yaml

config.yaml

docker-compose.yml

docker-compose.yml

View all files

Repository files navigation

hnTrends: Explore Trends in HN Posts

A one-time backfill of HackerNews posts that discovers the topics based on embeddings and leverages DuckDB. Compared to supervised analytics, here topics are discovered, not predefined. So we are doing exploratory reporting on the posts to detect trends for a given timeframe.

Dashboard

Cluster size over time : Stacked area of cluster volume by day.

Top clusters : Ranked by item count; click a row to cross-filter.

Cluster items : The drill-down of that cluster's actual HN posts.

Window (days) filter: Switch between the 9 windows; applies to all charts.

Setup

Requires Docker. After cloning this repo run

make up # builds images, runs load, then starts view, uses values in config.yaml

It would take a while depending on the number of days set. Once complete, open http://localhost:8088 — the HN Topic Clusters dashboard is available to explore.

It embeds stories, clusters them with KMeans across 9 lookback windows (1 day … 5 years), auto-labels each cluster, and serves it all as a Superset dashboard you can explore over time. Two Docker services share a single DuckDB file load runs the pipeline (ingest → embed → cluster) and exits; view starts only after load succeeds and reads the file.

Refrences

The pipeline is intentionally a discovery-and-reporting tool, not a tuned topic-modeling system. The techniques it uses, and where to learn them:

Concept<br>Where it's used<br>Learn more

Sentence embeddings — mapping text to dense vectors that capture meaning<br>pipeline/steps/embed.py encodes title + text with all-MiniLM-L6-v2<br>Sentence-BERT paper · SBERT docs

Transformer / MiniLM encoder — the distilled BERT-family model producing the vectors<br>SentenceTransformer(all-MiniLM-L6-v2)<br>"Attention Is All You Need" · MiniLM paper · model card

Unsupervised learning — no labels; structure is found in the data<br>The whole clustering step<br>scikit-learn: unsupervised learning

L2 normalization → cosine — unit-norm the vectors so Euclidean distance behaves as cosine (what MiniLM is trained for; ≈ spherical k-means)<br>normalize_embeddings=True in pipeline/steps/embed.py; normalize() in cluster.py::_fit<br>Cosine similarity · Spherical k-means

PCA dimensionality reduction — project 384-d vectors to ~50-d before KMeans to cut the curse of dimensionality (light stand-in for BERTopic's UMAP)<br>PCA(...) in pipeline/steps/cluster.py::_fit (PCA_DIMS)<br>scikit-learn: PCA · Curse of dimensionality

K-Means / MiniBatchKMeans — partition vectors into K groups around centroids<br>pipeline/steps/cluster.py::_fit<br>scikit-learn: K-Means · MiniBatch K-Means paper

Choosing K — here a fixed-target-size heuristic, K = clamp(n // TARGET_CLUSTER_SIZE, 2, MAX_K)<br>pipeline/steps/cluster.py::_fit<br>Elbow & silhouette methods (the tuned alternatives this skips)

Centroids & distance — cluster centers; distance over the unit-norm vectors (so Euclidean ≈ cosine)<br>array_distance in the SQL rollup<br>Cluster centroids · Euclidean distance

Cluster cohesion (tightness) — avg_dist, mean member-to-centroid distance<br>cluster_metadata.avg_dist<br>Clustering evaluation

Nearest-to-centroid representative — the title closest to the centroid as a fallback label<br>arg_min(title, distance)<br>Prototype / medoid selection

Topic modeling — discovering themes across a corpus<br>The end-to-end goal<br>BERTopic (embed→reduce→cluster→c-TF-IDF, the pattern this mirrors)

c-TF-IDF cluster labeling — class-based TF-IDF over each cluster's titles/text...