Hacker Newsnew | past | comments | ask | show | jobs | submit | stablepixel's commentslogin

Having studied Japanese at university and living there for a bit my interest naturally flowed in that direction, so I ran a quick check on OpenAI's tokeniser, nothing close to an actual study. I ran two tests of everyday English and its Japanese equivalent and another using more technical vocabulary across both languages. These were my results:

  Tokeniser  Everyday JP/EN   Technical JP/EN
  GPT-3      2.69x (183/68)   3.48x (233/67)
  GPT-4      2.16x (147/68)   2.53x (167/66)
  GPT-5.x    1.49x (101/68)   1.74x (115/66) 
It shows that the gap clearly has shrunk a fair bit with Japanese going from 2.7x English on GPT-3 to 1.5x on GPT-5.x for everyday text. The same passage of text yielded flat results for English across each tokeniser, so all the improvement went into Japanese. I also found that the Japanese polite form used to cost extra on GPT-3 but is now effectively free. The polite version of the Japanese passage was 9 characters longer (133 vs 124), and contained endings such as ました, ません and でした which have all become single tokens in newer tokenisers. Perhaps most interestingly, katakana loanwords seemed to add a lot of extra cost. For example, a word like パスワード(password) is split into four tokens (パ / ス / ワ / ード), while "password" is one token in English. In this example, the exact katakana string likely did not appear frequently enough in the training data, unlike the common hiragana clusters such as the polite endings, and the English word “password”, which earned their own tokens. The newer tokenisers continue to split less common kanji (like 靴 and 濡) into byte pieces, but katakana stands out as even everyday words are broken apart. It'd be interesting to run these checks on the same Qwen, Gemma, Llama tokenisers the study uses.

Can echo this, I've also fallen foul of keychain behaviour. In my case I found a lookup that passed the account name as raw bytes instead of text didn't match my item. On macOS it matched all items with a blank account (strictly, generic password items), which on my computer meant items from GitHub CLI, Malwarebytes, and Adobe Creative Cloud. The app I was building silently used one of those as its key, and would change whenever one of those apps would update its entry. The fix was to make sure the account is passed as a string and that it has a service name so that lookup either matches exactly my item or none. The API takes a loosely typed dictionary, so wrong types throw no errors, just different behaviour. I now always test against the real keychain and not a mock.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: