Audio episod ini mungkin rosak.
How to fix it?
MENGENAI EPISOD INI
Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention. In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens as a ``sink'' even if they are not semantically important. Based on the above analysis, we introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence lengths without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more. In addition, we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup. Code and datasets are provided at https://github.com/mit-han-lab/streaming-llm.
2023: Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis
https://arxiv.org/pdf/2309.17453v1.pdf
2023: Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis
https://arxiv.org/pdf/2309.17453v1.pdf
Bahasa Inggeris
Amerika Syarikat
CARI EPISOD LEPAS
Cari episod lepas bagi Papers Read on AI.
EPISOD LAIN DALAM PODCAST INI
We introduce Diagram of Thought (DoT), a framework that models iterative reasoning in large language models (LLMs) as the construction of a directed acyclic graph (DAG) within a single model. Unlike traditional approaches that represent reasoning as linear chains or trees, DoT organizes proposit ...
The potential effectiveness of counterspeech as a hate speech mitigation strategy is attracting increasing interest in the NLG research community, particularly towards the task of automatically producing it. However, automatically generated responses often lack the argumentative richness which c ...
Large language models (LLMs) often exhibit deficient reasoning or generate hallucinations. To address these, studies prefixed with"Self-"such as Self-Consistency, Self-Improve, and Self-Refine have been initiated. They share a commonality: involving LLMs evaluating and updating themselves. Nonet ...
In a convergence of machine learning and biology, we reveal that diffusion models are evolutionary algorithms. By considering evolution as a denoising process and reversed evolution as diffusion, we mathematically demonstrate that diffusion models inherently perform evolutionary algorithms, natu ...
Large language models (LLMs) often produce errors, including factual inaccuracies, biases, and reasoning failures, collectively referred to as"hallucinations". Recent studies have demonstrated that LLMs' internal states encode information regarding the truthfulness of their outputs, and that thi ...
Penafian: Podcast dan karya seni yang dibenamkan pada halaman ini adalah daripada Rob, yang merupakan hak milik pemiliknya dan tidak bergabung dengan atau disokong oleh Listen Notes, Inc.
SUNTING
Terima kasih kerana membantu memastikan pangkalan data podcast dalam keadaan terkini.
