INFORMAZIONI SU QUESTO EPISODIO
🤗 Upvotes: 40 | cs.LG, cs.AI, cs.CL
Authors:
Hyunjong Ok, Jaeho Lee
Title:
Weight-Space Programming Doesn't Need Transformers
Arxiv:
http://arxiv.org/abs/2611.22055v1
Abstract:
Standard neural architectures such as the Transformer were chosen because they train well with gradient descent and run fast on GPUs. In weight-space programming, a large language model (LLM) writes target weights by hand without gradient-based training, so those reasons no longer apply. Should we keep the Transformer? Given minimal guidance, ten labeled examples, and a sandbox, mainly on multiple-choice questions, we task three frontier LLMs with either generating the weights of a fixed architecture or designing the architecture themselves. When free to design, all three compile plain-text facts into a sparse table queried by a fixed procedure, reaching up to 54.4% test accuracy. An LLM-designed model achieves higher test accuracy than the most accurate released Transformer that the same LLM wrote while using about 1,800x fewer parameters. Without gradients, they choose a table, in which the LLM can state what each weight means and can find and rewrite a wrong one. Each wrong answer points to one row, and removing a row changes about one other answer. From this, we propose an associative-memory-based model up to 17x faster on GPUs than a same-sized Transformer. We also test pretrained weights, other benchmarks, and tasks where models generate answers. Pretrained LLMs remain more accurate; the claim concerns which structure to build when writing weights.
TRASCRIZIONE 🔗
00:00:03,000 --> 00:00:05,400
Evan: Welcome to Daily Paper Cast!
2
00:00:05,448 --> 00:00:13,058
Ashley: Today's paper is selected from the Hugging Face daily paper list of November 18, 2026, with 40 upvotes.
3
00:00:13,104 --> 00:00:21,444
Evan: The paper we're delving into is titled 'Weight-Space Programming Doesn’t Need Transformers' by Hyunjong Ok and Jaeho Lee.
4
00:00:21,504 --> 00:00:31,914
Ashley: Our corresponding author, Jaeho Lee, and his colleague Hyunjong Ok, are affiliated with t…