Adaptive Low-Rank Product Transformers with Dynamic Expert Routing for Online Continual Learning
Skip to main
You are using an outdated browser. Please upgrade your browser to improve your experience.
Published May 7, 2026
| Version v1
Preprint
Open
Adaptive Low-Rank Product Transformers with Dynamic Expert Routing for Online Continual Learning
Authors/Creators
Ballanco, Joshua<br>(Contact person)1
Show affiliations
1.
Manhattan Metric
Description
Inspired by the role of sleep in biological continual learning, we introduce RVW, a trans-
former architecture for online continual adaptation of pretrained models. RVW maintains a
small pool of per-layer experts that grow and prune in response to distribution shift, with
no replay buffer and no explicit task identifier. Applied to TinyLlama-1.1B on a 15,000-
chunk six-domain stream, RVW reaches 40 average held-out PPL, substantially better than
EWC (158), fine-tuning (164), and LoRA (448) on the same parameter-matched base, while
preserving prior-domain performance. Threshold sweeps suggest a combinatorial encoding
reading: domain knowledge appears to be carried by routing patterns across layers rather
than by individual specialized experts.
Files
rvw.pdf
Files<br>(531.1 kB)
Name<br>Size
Download all
rvw.pdf
md5:1e3f2cb810db84e65ba03d85f43a24c2
531.1 kB
Preview
Download
653
Views
154
Downloads
Show more details
All versions<br>This version
Views
Total views
653
653
Downloads
Total downloads
154
154
Data volume
Total data volume
96.1 MB<br>96.1 MB
More info on how stats are collected....
Versions
External resources
Indexed in
OpenAIRE
Communities
Keywords and subjects
MeSH
Machine Learning
Details
DOI
DOI Badge
DOI
10.5281/zenodo.20064618
Markdown
[](https://doi.org/10.5281/zenodo.20064618)
reStructuredText
.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.20064618.svg<br>:target: https://doi.org/10.5281/zenodo.20064618
HTML
Image URL
https://zenodo.org/badge/DOI/10.5281/zenodo.20064618.svg
Target URL
https://doi.org/10.5281/zenodo.20064618
Resource type<br>Preprint
Publisher<br>Zenodo
Languages
English
Rights
License
Creative Commons Attribution 4.0 International
The Creative Commons Attribution license allows re-distribution and re-use of a licensed work on the condition that the creator is appropriately credited.
Read more
Citation
Export
Technical metadata
Created
May 7, 2026
Modified
May 7, 2026
Jump up
This site uses cookies. Find out more on how we use cookies
Accept all cookies<br>Accept only essential cookies