Can we reconstruct a closed-source LLM tokenizer using only two oracles from the chat API? : LocalLLaMAjump to contentmy subreddits<br>edit subscriptions<br>popular<br>-all<br>-users<br>| AskReddit<br>-pics<br>-funny<br>-movies<br>-gaming<br>-worldnews<br>-news<br>-todayilearned<br>-nottheonion<br>-explainlikeimfive<br>-mildlyinteresting<br>-DIY<br>-videos<br>-OldSchoolCool<br>-TwoXChromosomes<br>-tifu<br>-Music<br>-books<br>-LifeProTips<br>-dataisbeautiful<br>-aww<br>-science<br>-space<br>-Showerthoughts<br>-askscience<br>-Jokes<br>-Art<br>-singapore<br>-IAmA<br>-Futurology<br>-sports<br>-UpliftingNews<br>-food<br>-nosleep<br>-creepy<br>-history<br>-gifs<br>-InternetIsBeautiful<br>-GetMotivated<br>-gadgets<br>-announcements<br>-WritingPrompts<br>-philosophy<br>-Documentaries<br>-EarthPorn<br>-photoshopbattles<br>-listentothis<br>-blog

more "

reddit.com LocalLLaMAcomments

Want to join? Log in or sign up in seconds.

limit my search to r/LocalLLaMAuse the following search parameters to narrow your results:<br>subreddit:subredditfind submissions in "subreddit"author:usernamefind submissions by "username"site:example.comfind submissions from "example.com"url:textsearch for "text" in urlselftext:textsearch for "text" in self post contentsself:yes (or self:no)include (or exclude) self postsnsfw:yes (or nsfw:no)include (or exclude) results marked as NSFWe.g. subreddit:aww site:imgur.com dog<br>see the search faq for details.

advanced search: by author, subreddit...

this post was submitted on 11 Jul 2026<br>2 points (56% upvoted)<br>shortlink:

Submit a new link

Submit a new text post

LocalLLaMA<br>joinleave

r/LocalLLaMA

A subreddit to discuss about Llama, the family of large language models created by Meta AI.

Subreddit rules

Search by flair

+Discussion

+Tutorial | Guide

+New Model

+News

+Resources

+Other

a community for 3 years

MODERATORS

message the mods

40 &middot; 75 comments<br>Best Local VLMs - July 2026<br>691 &middot; 99 comments<br>0:46

Qwen3.6 35B-A3B (Q8_0, no KV quant) single prompt in opencode: "Create a beautiful, relaxing flight simulator in a single html file with mountains, clouds, and endless procedural terrain"<br>84 &middot; 66 comments

The U.S. tech industry is increasingly anxious about the rising power and competitive price of open-source AI models from China — and whether the Trump administration will respond with yet another executive order | Politico<br>374 &middot; 81 comments

Training an LLM from scratch on 1800's texts (160GB dataset)<br>29 &middot; 25 comments<br>0:29

Running Qwen3 30B A3B at 50 tok/s on RTX 5060 Ti<br>&middot; 9 comments<br>Why are MoE models so belittled?<br>239 &middot; 119 comments<br>Tencent-HY3 is the real deal on 128GB!<br>781 &middot; 258 comments

2.5x faster Qwen3.6 NVFP4 Unsloth quants<br>&middot; 6 comments<br>Are EPYC CCDs all you need + benchmarks<br>74 &middot; 16 comments

Nostalgia for Bloom

Welcome to Reddit,<br>the front page of the internet.<br>Become a Redditorand join one of thousands of communities.

&times;

Can we reconstruct a closed-source LLM tokenizer using only two oracles from the chat API?Discussion (self.LocalLLaMA)<br>submitted 7 hours ago by bayes-song

From the chat APIs alone, we can extract two useful oracles:

1. Token length oracle: Given any string s, return len(tokenize(s)).

Prefix token oracle: Given a string s and integer n, return the string decoded from the first n tokens of tokenize(s).

The first oracle (token count) is straightforward. The second oracle's main purpose is to resolve ambiguities in merge order. For example, without it we couldn’t distinguish between the segmentations (a, bc) and (ab, c). By leveraging the model’s ability to repeat text in a controlled way, we can construct this second oracle from the regular chat interface.

With only these two oracles, is it possible to reconstruct an equivalent tokenizer — one that produces the exact same token boundaries/segmentation on any input (up to token ID remapping)?

3 comments<br>share<br>save<br>hide<br>report

all 3 comments<br>sorted by: best<br>topnewcontroversialoldrandomq&alive (beta)

Want to add to the discussion?<br>Post a comment!<br>Create an account

[+][deleted] 7 hours ago (1 child)<br>[removed]

[–]bayes-song[S] 2 points3 points4 points 7 hours ago (0 children)<br>I don't think the tokenizer has to be unique. Even for BPE, there can be different merge orders that never change the final tokenization. I'd consider it solved as long as the reconstructed tokenizer produces the exact same token boundaries for every input. Assuming BPE seems reasonable since it's the most common tokenizer family. My guess is this should be possible, the question is whether there's an algorithm that does it with the minimum number of oracle queries (or total queried input length).

permalink<br>embed<br>save<br>parent<br>report<br>reply

about<br>blog<br>about<br>advertising<br>careers

help<br>site rules<br>Reddit help center<br>reddiquette<br>mod guidelines<br>contact us

apps & tools<br>Reddit for iPhone<br>Reddit for Android<br>mobile website

reddit premium

Use of this site constitutes acceptance of our User Agreement and Privacy Policy. &copy; 2026 reddit inc. All rights reserved.<br>REDDIT and the ALIEN Logo are registered trademarks of...