Ask HN: (Why) Was the LLM breakthrough useful for images, audio, etc.?

jerlendds · 2026-09-06T17:20:12.000000Z

VLM != LLM. Vision language models basically treat text tokens and image tokens the same. Post-training an LLM on images+text can improve its capabilities. Id recommend searching around the keyword VLM to find more resources on how multi-modal AI works.

- https://huggingface.co/blog/vlms

- https://en.wikipedia.org/wiki/Multimodal_learning

kypro · 2026-09-08T21:37:49.000000Z

While some neural network architectures can be designed to better fit certain tasks they are at their core general-purpose learning algorithms which can approximate any target function.

> I feel like I have a decent conceptual grasp of what LLMs are doing with written text

Just think of it as input data. In theory it shouldn't matter what each token represents. They could be xbox controller buttons, image pixels, or text.

The model with enough training data will map those inputs to an expected output.

verdverm · 2026-09-06T18:50:01.000000Z

transformers were first an image understanding technique, the text processing came later, it's all about the training data and gradient descent, and allegedly attention

laruss5 · 2026-09-07T09:18:27.000000Z

It's actually the other way round - the Transformer architecture was introduced for text (machine translation) in "Attention Is All You Need" (2017). Vision Transformers, which apply it to images, came three years later in 2020: https://arxiv.org/abs/2010.11929

verdverm · 2026-09-07T15:40:34.000000Z

right, it was not text generation per-se (completion/contemporary understanding) that came first, vision was before that, translation before that