رونوشت 🔗
1
00:00:03,060 --> 00:00:05,600
Evan: Welcome to Daily Paper Cast.
2
00:00:05,664 --> 00:00:14,144
Ashley: Today's paper is from the Hugging Face daily paper list of October 14, 2026, and it has received 55 upvotes.
3
00:00:14,208 --> 00:00:21,808
Evan: The title of the paper is 'Vela: User Behavior as Supervision Sparks LLM Intelligence'.
4
00:00:21,864 --> 00:00:29,024
Ashley: It’s authored by Yang Xiao, Yusong Sun, and the corresponding author is Yang Xiao from the AllSpark Team.
5
00:00:29,088 --> 00:00:32,468
Evan: Let’s dive right into the Introduction section of the paper.
6
00:00:32,520 --> 00:00:38,770
Ashley: As foundation models grow more capable, they're increasingly being used in real-world applications.
7
00:00:38,770 --> 00:00:51,280
Think of personal assistants that take actions based on users’ long-term behavior histories—such as OpenClaw and WorkBuddy—or recommendation and advertising systems that predict what users will do next.
8
00:00:51,336 --> 00:01:01,696
Evan: Sure, and these applications share three key properties: decisions are individualized, span long durations, and every judgment is grounded in the user’s behavior history.
9
00:01:01,696 --> 00:01:07,636
Together, these properties define what they call the 'user dimension' of LLM intelligence.
10
00:01:07,680 --> 00:01:08,660
Ashley: Exactly.
11
00:01:08,660 --> 00:01:19,880
The ability to infer an individual’s preferences, intent, and likely future actions from real-world user behavior data is crucial for understanding users from their behavior histories.
12
00:01:19,880 --> 00:01:28,700
Models need to learn from context, identify relevant signals amid noise, integrate evidence over time, and track changes in preferences.
13
00:01:28,752 --> 00:01:34,262
Evan: But current large language models remain limited in this user dimension of intelligence, right?
14
00:01:34,262 --> 00:01:40,212
Common training sources include things like web pages, textbooks, code, and synthetic data.
15
00:01:40,212 --> 00:01:47,392
While these might contain user-related content, they don't necessarily capture how individual users' behavior evolves over time.
16
00:01:47,448 --> 00:01:48,458
Ashley: Precisely.
17
00:01:48,458 --> 00:01:54,788
Training solely on those sources is insufficient for developing user dimension of LLM intelligence.
18
00:01:54,788 --> 00:02:03,228
Recent evaluations show that even LLMs with stronger general capabilities might not always perform better in user-facing applications.
19
00:02:03,228 --> 00:02:12,308
There's a tendency for models to drift toward an average-user profile, lose accuracy as behavior histories lengthen, and struggle with personalization.
20
00:02:12,360 --> 00:02:20,830
Evan: Real-world user behavior data, on the other hand, is abundantly available on content platforms and in personal-assistant agents.
21
00:02:20,830 --> 00:02:31,000
These records capture user actions and their contexts over time, making them a natural supervisory signal that has the potential to improve LLM intelligence.
22
00:02:31,056 --> 00:02:32,126
Ashley: That's right.
23
00:02:32,126 --> 00:02:41,486
As human-generated text becomes a bottleneck for scaling LLMs, user behavior data offers a large, still-growing source of training signals.
24
00:02:41,486 --> 00:02:48,736
The challenge, then, is converting heterogeneous behavior records into diverse tasks with well-defined supervision.
25
00:02:48,792 --> 00:02:54,822
Evan: Previous efforts have focused on converting user behavior data into training signals for specific tasks.
26
00:02:54,822 --> 00:03:00,652
For instance, P5 merges multiple recommendation tasks under a language modeling objective.
27
00:03:00,652 --> 00:03:07,452
Similarly, RLPF uses prediction feedback to train LLMs to produce user summaries.
28
00:03:07,512 --> 00:03:11,462
Ashley: However, these studies are limited to specific applications.
29
00:03:11,462 --> 00:03:22,932
They leave open the broader question of whether training on user behavior data can enhance performance across diverse real-world user-related scenarios and general LLM capabilities.
30
00:03:22,992 --> 00:03:25,662
Evan: And this brings us to the core of their research.
31
00:03:25,662 --> 00:03:38,392
They propose 'user behavior as supervision,' a paradigm in which LLMs are trained to understand users from their behavior histories, using supervision derived from observed behaviors.
32
00:03:38,448 --> 00:03:49,528
Ashley: To operationalize this paradigm, they've introduced Vela, a framework that converts raw user behavior data from a large social media platform into trainable tasks and trajectories.
33
00:03:49,528 --> 00:03:59,228
Using anonymized behavior records from 100,000 users, Vela generates 100,000 tasks across 91 real-world task templates.
34
00:03:59,280 --> 00:04:00,880
Evan: That's quite a scale.
35
00:04:00,880 --> 00:04:08,560
Difficulty-based filtering and trajectory acceptance criteria are jointly used to retain approximately 75,000 tasks.
36
00:04:08,560 --> 00:04:14,740
The selected trajectories form two main training corpora: Vela-Instruct and Vela-RFT.
37
00:04:14,784 --> 00:04:29,594
Ashley: Fine-tuning their base model, Qwen3.6-35B-A3B, on these training trajectories yields two models: Vela-35B-A3B and Vela-35B-A3B-RFT.
38
00:04:29,594 --> 00:04:41,204
To evaluate their approach, they used three groups of benchmarks: VelaBench, user-related out-of-distribution benchmarks, and general out-of-distribution benchmarks.
39
00:04:41,256 --> 00:04:43,296
Evan: And the results were quite promising.
40
00:04:43,296 --> 00:04:59,296
Vela-35B-A3B improved over the base model by 11.5 points on VelaBench and up to 11.2 points on user-related out-of-distribution benchmarks, with an average improvement of 4 points on general benchmarks.
41
00:04:59,296 --> 00:05:08,596
Reinforcement learning after fine-tuning further boosted performance, outperforming the base model by up to 7.7 points on user-related tasks.
Load full transcript
42
00:05:08,640 --> 00:05:20,120
Ashley: These findings suggest that using real-world user behavior as supervision can significantly enhance both user-centered capabilities and the broader intelligence of large language models.
43
00:05:20,184 --> 00:05:22,424
Evan: That wraps up the Introduction section.
44
00:05:22,424 --> 00:05:25,104
Let's move on to the next part of the paper shortly.
45
00:05:26,354 --> 00:05:27,774
Evan: Welcome back, everyone.
46
00:05:27,774 --> 00:05:36,834
Now, let's discuss the Method section of this paper, where the authors outline how they implemented their proposed paradigm of 'user behavior as supervision.'
47
00:05:36,920 --> 00:05:37,630
Ashley: Evan.
48
00:05:37,630 --> 00:05:45,190
The heart of this method is the Vela framework, which converts raw user behavior data into trainable tasks and trajectories.
49
00:05:45,190 --> 00:05:53,450
This involves several critical stages: data collection, task synthesis, difficulty-based filtering, and trajectory generation.
50
00:05:53,498 --> 00:05:54,098
Evan: Got it.
51
00:05:54,098 --> 00:05:56,818
So, let’s start with the data collection.
52
00:05:56,818 --> 00:06:00,018
What kind of user behavior data are they dealing with here?
53
00:06:00,074 --> 00:06:09,124
Ashley: The data consists of timestamped sequences of user actions taken on a social media platform, spanning several weeks to years from 100,000 users.
54
00:06:09,124 --> 00:06:19,314
Importantly, they anonymized this data by removing user profile information and direct personal identifiers, ensuring privacy and compliance with ethical standards.
55
00:06:19,430 --> 00:06:21,050
Evan: Okay, that makes sense.
56
00:06:21,050 --> 00:06:23,150
And how do they categorize this data?
57
00:06:23,210 --> 00:06:31,090
Ashley: They structured the data into six categories: Views, Searches, Purchases, Posts, Follows, and Dislikes.
58
00:06:31,090 --> 00:06:38,110
Each category captures a different aspect of user behavior, making the data comprehensive and versatile for training.
59
00:06:38,222 --> 00:06:39,222
Evan: Interesting.
60
00:06:39,222 --> 00:06:45,342
Moving on to task synthesis, how do they convert this user behavior data into trainable tasks?
61
00:06:45,386 --> 00:06:48,646
Ashley: They use a process called user behavior as supervision.
62
00:06:48,646 --> 00:06:56,736
For each task, they generate questions grounded in a single user's behavior history, categorized into Predictive and Rubric questions.
63
00:06:56,736 --> 00:07:04,666
Predictive questions are scored against ground-truth future behavior, while Rubric questions use task-specific criteria for scoring.
64
00:07:04,730 --> 00:07:08,070
Evan: So, could you give some examples of these question types?
65
00:07:08,144 --> 00:07:09,104
Ashley: Of course.
66
00:07:09,104 --> 00:07:17,064
For instance, a Predictive question called 'Objective' asks the model to select products the user will purchase from a list of candidates.
67
00:07:17,064 --> 00:07:22,634
It’s evaluated by comparing the model’s predictions against the actual products the user buys.
68
00:07:22,634 --> 00:07:30,754
Another Predictive question, 'Open-prediction,' asks the model to freely predict the products a user will purchase, without any candidates.
69
00:07:30,878 --> 00:07:33,198
Evan: And what about Rubric questions?
70
00:07:33,242 --> 00:07:41,372
Ashley: For Rubric questions, an 'Open' question might ask the model to summarize a user’s content interests based on their visible history.
71
00:07:41,372 --> 00:07:49,642
This is evaluated using a task-specific rubric to ensure summaries are accurate and adequately supported by evidence from the user's history.
72
00:07:49,642 --> 00:07:58,582
There's also an 'Attribution' Rubric question, which presents an observed future behavior and asks the model to explain it based on the user’s visible history.
73
00:07:58,694 --> 00:07:59,304
Evan: Got it.
74
00:07:59,304 --> 00:08:02,214
How do they handle the difficulty of these tasks?
75
00:08:02,258 --> 00:08:08,198
Ashley: They filter tasks based on difficulty by generating multiple model responses and scoring them.
76
00:08:08,198 --> 00:08:16,838
Tasks with intermediate difficulty scores are retained, as they provide a balanced training signal neither too easy nor too challenging.
77
00:08:16,898 --> 00:08:17,658
Evan: Right.
78
00:08:17,658 --> 00:08:20,598
And then comes the trajectory generation, correct?
79
00:08:20,642 --> 00:08:21,482
Ashley: Correct.
80
00:08:21,482 --> 00:08:25,702
For each retained task, they generate multiple training trajectories.
81
00:08:25,702 --> 00:08:30,592
Only the highest-scoring trajectory is kept, ensuring quality training samples.
82
00:08:30,592 --> 00:08:45,982
Two main corpora are created from these trajectories: Vela-Instruct and Vela-RFT, fine-tuned with different configurations to develop two models, Vela-35B-A3B and Vela-35B-A3B-RFT.
83
00:08:46,034 --> 00:08:47,724
Evan: What about the evaluation?
84
00:08:47,724 --> 00:08:50,414
How do they measure the performance of these models?
85
00:08:50,474 --> 00:08:52,714
Ashley: They use three groups of benchmarks.
86
00:08:52,714 --> 00:09:00,524
First, VelaBench, which consists of 982 held-out tasks formed from the same 91 templates used during training.
87
00:09:00,524 --> 00:09:02,684
This assesses in-domain performance.
88
00:09:02,684 --> 00:09:11,474
Second, user-related out-of-distribution benchmarks, including scenarios like PurchaseIntent, ProfileSummary, PersonaBench, and AdRecommendation.
89
00:09:11,474 --> 00:09:25,474
Lastly, they use general out-of-distribution benchmarks such as AIME 2026 for mathematics, GPQA-Diamond for science, IFBench for instruction following, and both CL-bench and CL-bench Life for context learning.
90
00:09:25,608 --> 00:09:28,358
Evan: And what metrics do they use for evaluation?
91
00:09:28,418 --> 00:09:31,678
Ashley: Evaluation metrics vary depending on the task type.
92
00:09:31,678
--> 00:09:37,548
For Objective questions, they use set-F1, which considers both precision and recall.
93
00:09:37,548 --> 00:09:43,628
For Open-prediction questions, they map predicted items to ground-truth sets based on semantic matching.
94
00:09:43,628 --> 00:09:51,698
Rubric questions are scored according to criteria specific to each task, ensuring that the responses are measured against the visible history.
95
00:09:51,746 --> 00:09:53,296
Evan: That's comprehensive.
96
00:09:53,296 --> 00:09:55,666
Did they perform any additional analyses?
97
00:09:55,730 --> 00:10:00,790
Ashley: Yes, they examined the benefits of difficulty-based filtering and reinforcement learning.
98
00:10:00,790 --> 00:10:09,480
They compared models trained with and without difficulty-filtered tasks, finding an average gain of 1.1 points in out-of-distribution performance.
99
00:10:09,480 --> 00:10:19,490
Additionally, reinforcement learning initialized from the Vela-35B-A3B model further improved performance on VelaBench and ProfileSummary benchmarks.
100
00:10:19,538 --> 00:10:23,098
Evan: Their approach to reinforcing learning certainly sounds interesting.
101
00:10:23,098 --> 00:10:26,698
Does scaling the dataset improve the model's performance further?
102
00:10:26,762 --> 00:10:28,082
Ashley: Yes, indeed.
103
00:10:28,082 --> 00:10:40,002
Scaling experiments reveal that increasing the number of training examples from 4.7K to 75K continues to enhance performance across both user-related and general tasks.
104
00:10:40,002 --> 00:10:53,522
For instance, the VelaBench score rises from 51.5 to 63.0, while the average score on user-related out-of-distribution benchmarks climbs from 43.5 to 47.5.
105
00:10:53,630 --> 00:10:57,590
Evan: It's quite interesting how they managed to balance scaling and quality.
106
00:10:57,590 --> 00:11:01,190
Now, what about the potential broader impacts of their methods?
107
00:11:01,250 --> 00:11:10,690
Ashley: Their findings highlight the potential of user behavior data to vastly improve the general intelligence and user-related performance of large language models.
108
00:11:10,690 --> 00:11:22,410
They suggest using real-world user behavior as supervision can be a scalable approach, proving valuable in diverse real-world scenarios and expanding the scope for LLM applications.
109
00:11:22,466 --> 00:11:25,786
Evan: That brings us to the end of the Method section in this paper.
110
00:11:27,051 --> 00:11:28,391
Evan: Welcome back, everyone.
111
00:11:28,391 --> 00:11:34,311
Now that we've covered the Method section, it’s time to dive into the Experiments and Results section of the paper.
112
00:11:34,371 --> 00:11:38,671
Ashley: Yes, this is where we see how their approach performs in practice.
113
00:11:38,671 --> 00:11:43,331
The authors conducted extensive benchmarking to validate their proposed method.
114
00:11:43,331 --> 00:11:45,611
So, Evan, where should we begin?
115
00:11:45,725 --> 00:11:48,245
Evan: Let’s start with the setup of their experiments.
116
00:11:48,245 --> 00:11:51,775
Ashley, can you walk us through their initial configurations?
117
00:11:51,819 --> 00:11:52,569
Ashley: Sure.
118
00:11:52,569 --> 00:12:08,789
They fine-tuned their base model, Qwen3.6-35B-A3B, on the Vela-Instruct and Vela-RFT corpora to obtain the Vela-35B-A3B and Vela-35B-A3B-RFT models, respectively.
119
00:12:08,789 --> 00:12:21,559
Both models were trained using the Swift framework with the same training settings: two epochs, a packed sequence length of 131,072 tokens, and a learning rate of 1 times 10 to the minus 5.
120
00:12:21,663 --> 00:12:25,043
Evan: And what benchmarks did they use to evaluate the models?
121
00:12:25,107 --> 00:12:27,437
Ashley: They used three groups of benchmarks.
122
00:12:27,437 --> 00:12:34,617
The first group is VelaBench, which includes 982 held-out tasks from the same 91 templates used during training.
123
00:12:34,617 --> 00:12:36,937
This measures in-domain performance.
124
00:12:36,937 --> 00:12:45,897
The second group consists of user-related out-of-distribution benchmarks, including PurchaseIntent, ProfileSummary, PersonaBench, and AdRecommendation.
125
00:12:45,897 --> 00:12:51,527
These benchmarks test the models on real-world, user-facing scenarios outside the training data.
126
00:12:51,527 --> 00:13:05,487
The final group is general out-of-distribution benchmarks, which include AIME 2026 for mathematics, GPQA-Diamond for science, IFBench for instruction following, and both CL-bench and CL-bench Life for context learning.
127
00:13:05,627 --> 00:13:07,857
Evan: Impressive diversity there.
128
00:13:07,857 --> 00:13:10,847
What were the key findings from these evaluations?
129
00:13:10,899 --> 00:13:18,529
Ashley: On VelaBench, Vela-35B-A3B outperformed the base model by 11.5 points.
130
00:13:18,529 --> 00:13:26,499
For user-related out-of-distribution benchmarks, it showed substantial improvements as well, with gains of up to 11.2 points.
131
00:13:26,499 --> 00:13:31,579
The average improvement across general out-of-distribution benchmarks was 4 points.
132
00:13:31,635 --> 00:13:32,675
Evan: Interesting.
133
00:13:32,675 --> 00:13:35,055
Were any additional analyses performed?
134
00:13:35,115 --> 00:13:36,565
Ashley: Yes, indeed.
135
00:13:36,565 --> 00:13:42,055
They explored the impact of difficulty-based filtering and reinforcement learning on performance.
136
00:13:42,055 --> 00:13:46,655
First, they compared models trained with and without difficulty filtering.
137
00:13:46,655 --> 00:13:56,475
They found that filtering improved user-related out-of-distribution performance by an average of 1.1 points without affecting in-domain performance on VelaBench.
138
00:13:56,523 --> 00:13:58,423
Evan: What about reinforcement learning?
139
00:13:58,467 --> 00:14:06,867
Ashley: They used a reinforcement learning approach called Group Sequence Policy Optimization, or GSPO, to further enhance model performance.
140
00:14:06,867 -->
00:14:08,677
The results were quite telling.
141
00:14:08,677 --> 00:14:21,537
Reinforcement learning initialized from Vela-35B-A3B raised performance on VelaBench to 66.2 points, compared to 58.5 points when initialized from the base model.
142
00:14:21,537 --> 00:14:32,847
For ProfileSummary, the improvement was equally substantial—37.6 points from the Vela-35B-A3B compared to 29.9 points from the base model.
143
00:14:32,967 --> 00:14:36,827
Evan: So, scaling the data also played a role here, right?
144
00:14:36,891 --> 00:14:40,811
Ashley: Yes, scaling experiments were a crucial part of their analysis.
145
00:14:40,811 --> 00:14:49,511
They observed that increasing the training set from 4.7 thousand to 75 thousand examples showed clear performance improvements across the board.
146
00:14:49,511 --> 00:14:57,691
For instance, VelaBench scores increased from 51.5 at the base scale to 63 points at 75 thousand examples.
147
00:14:57,691 --> 00:15:09,391
Similarly, the average user-related out-of-distribution performance increased from 43.5 to 47.5 points, and general out-of-distribution benchmarks also saw benefits.
148
00:15:09,485 --> 00:15:15,555
Evan: It's fascinating how scaling, coupled with user behavior data, brings such significant improvements.
149
00:15:15,555 --> 00:15:21,215
Were there any notable insights on general LLM intelligence beyond user-related tasks?
150
00:15:21,267 --> 00:15:22,197
Ashley: There were.
151
00:15:22,197 --> 00:15:30,587
They found that user behavior as supervision enhances not just user-related intelligence but also general LLM capabilities.
152
00:15:30,587 --> 00:15:38,627
For example, their Vela-35B-A3B model showed gains of 4 points on general out-of-distribution benchmarks.
153
00:15:38,627 --> 00:15:47,207
The largest improvements were on IFBench and CL-bench Life, with increases of 6.7 and 5.6 points, respectively.
154
00:15:47,259 --> 00:15:53,539
Evan: It’s amazing how real-world data can push the envelope on both user-specific and general tasks.
155
00:15:53,539 --> 00:15:58,559
Was there any mention of future research directions or applications for this framework?
156
00:15:58,611 --> 00:16:05,931
Ashley: Yes, they suggest that using real-world behavior data for supervision can be scaled up for broader applications.
157
00:16:05,931 --> 00:16:16,291
This could be particularly valuable for enhancing the general intelligence of language models while also improving user-specific capabilities in diverse real-world scenarios.
158
00:16:16,347 --> 00:16:19,237
Evan: Certainly promising avenues for future work.
159
00:16:19,237 --> 00:16:23,277
That brings us to the end of the Experiments and Results section for this paper.
160
00:16:23,277 --> 00:16:26,067
Stay tuned as we move forward in our discussion.
161
00:16:27,317 --> 00:16:28,387
Evan: Welcome back.
162
00:16:28,387 --> 00:16:36,497
Next up, we're diving into the Related Work section of the paper, where the authors situate their research within
the broader context of existing studies.
163
00:16:36,587 --> 00:16:41,827
Ashley: It's important to understand how this work builds on and differs from previous efforts.
164
00:16:41,827 --> 00:16:50,697
Let’s start with how prior research has evaluated LLMs' abilities to remember, understand, and respond to individual users.
165
00:16:50,741 --> 00:16:51,461
Evan: Sure.
166
00:16:51,461 --> 00:16:59,641
One notable study, LongMemEval, examines information retention and reasoning across long conversational histories.
167
00:16:59,641 --> 00:17:06,761
Another, PrefEval, tests whether models can infer and follow explicit and implicit user preferences.
168
00:17:06,821 --> 00:17:07,821
Ashley: That's right.
169
00:17:07,821 --> 00:17:18,251
PersonaMem focuses on evolving user profiles and personalized responses, while Memora looks at how agents update memory and avoid relying on outdated information.
170
00:17:18,251 --> 00:17:26,661
ATM-Bench extends this evaluation to multiple modalities and sources, including resolving personal references and combining evidence.
171
00:17:26,717 --> 00:17:33,617
Evan: So, these studies form a foundation, but they all tackle the problem in specific contexts or applications.
172
00:17:33,617 --> 00:17:41,257
For instance, behavior prediction benchmarks connect understanding users to their observed behaviors within certain scenarios.
173
00:17:41,309 --> 00:17:42,369
Ashley: Exactly.
174
00:17:42,369 --> 00:17:48,909
OPeRA captures online shopping sessions with observations, personas, rationales, and actions.
175
00:17:48,909 --> 00:17:58,669
FineRob evaluates fine-grained social media behavior prediction from long histories, and OmniBehavior studies long-horizon behavior across various scenarios.
176
00:17:58,669 --> 00:18:06,249
Our work complements these efforts by using real-world behavior histories as supervision for a broader set of user-related tasks.
177
00:18:06,353 --> 00:18:07,423
Evan: Interesting.
178
00:18:07,423 --> 00:18:12,913
How have researchers approached learning from user histories beyond behavior prediction then?
179
00:18:12,965 --> 00:18:20,685
Ashley: Learning from user histories has also been explored through user representations, recommendation systems, and behavior simulation.
180
00:18:20,685 --> 00:18:28,805
For instance, User-LLM encodes interaction histories into user embeddings that condition LLM responses.
181
00:18:28,853 --> 00:18:31,143
Evan: P5 also seems relevant here.
182
00:18:31,143 --> 00:18:32,453
How does that fit in?
183
00:18:32,501 --> 00:18:44,511
Ashley: P5 expresses multiple recommendation tasks within a unified text-to-text framework, whereas RLPF uses prediction feedback to optimize user summaries for downstream tasks.
184
00:18:44,511 --> 00:18:50,261
DEEPER refines personas by examining discrepancies between predicted and observed behavior.
185
00:18:50,369 --> 00:18:55,489
Evan: Beyond user representations, what about improved personalized simulations?
186
00:18:55,571 --> 00:18:56,701
Ashley: Good question.
187
00:18:56,701 --> 00:19:02,711
Customer-R1 uses reinforcement learning to improve personalized shopping behavior simulation.
188
00:19:02,711 --> 00:19:09,261
OmniBehavior, accompanying FineRob, fine-tunes models for behavior prediction with long-term memory.
189
00:19:09,357 --> 00:19:15,117
Evan: And how does the source of training signals play a role in these methodologies?
190
00:19:15,173 --> 00:19:23,173
Ashley: Work on recursive training with model-generated data warns of risks like losing information from the original data distribution.
191
00:19:23,173 --> 00:19:30,133
This is why using observed future behavior to derive predictive ground truth, as done in Vela, is crucial.
192
00:19:30,197 --> 00:19:37,187
Evan: So, Vela brings something new to the table by using tasks and trajectories grounded in real user behavior.
193
00:19:37,187 --> 00:19:40,357
How extensive is their evaluation of this approach?
194
00:19:40,421 --> 00:19:46,071
Ashley: They evaluate it across a wide set of user-related tasks and multiple data scales.
195
00:19:46,071 --> 00:19:54,161
They assess the effects on user-related performance, general LLM capabilities, and the benefits for reinforcement learning tasks.
196
00:19:54,281 --> 00:20:00,341
Evan: It's clear that grounding in real-world user behavior offers robust supervision.
197
00:20:00,341 --> 00:20:05,761
Were there any specific studies that inspired or were crucial to the development of Vela?
198
00:20:05,813 --> 00:20:12,423
Ashley: Past research on user embeddings, recommendation frameworks, and behavior simulation were foundational.
199
00:20:12,423 --> 00:20:22,613
Each addressed specific facets that Vela synthesizes into a more expansive framework benefiting both user-related tasks and general LLM performance.
200
00:20:22,661 --> 00:20:24,911
Evan: That wraps up the Related Work section.
201
00:20:24,911 --> 00:20:30,981
Now that we have the context, we can appreciate how Vela positions itself and builds upon previous studies.
202
00:20:30,981 --> 00:20:33,821
Stay tuned as we continue our exploration.
203
00:20:35,070 --> 00:20:37,950
Evan: Welcome back for the final segment of our discussion.
204
00:20:37,950 --> 00:20:42,970
It’s time to summarize the key contributions and takeaways from this fascinating paper.
205
00:20:43,014 --> 00:20:54,774
Ashley: To recap the highlights, the standout contribution of the paper is the introduction of the Vela framework, which uses real-world user behavior data as supervision to enhance large language models.
206
00:20:54,822 --> 00:21:06,602
Evan: Right, and this approach sets itself apart by tapping into the rich, sequential nature of user data, unlike traditional static text sources.
207
00:21:06,654 --> 00:21:07,944
Ashley: Exactly.
208
00:21:07,944 --> 00:21:20,674
The paper showed that converting anonymized user behavior histories into trainable tasks and trajectories allows for a more personalized and context-aware model training.
209
00:21:20,674 --> 00:21:31,854
They successfully retained around 75,000 tasks out of an initial 100,000 using difficulty-based filtering and trajectory generation.
210
00:21:31,902 --> 00:21:34,452
Evan: Those numbers are impressive.
211
00:21:34,452 --> 00:21:38,702
How do these contributions translate into performance gains?
212
00:21:38,766 --> 00:21:43,606
Ashley: Fine-tuning with user behavior data led to significant performance improvements.
213
00:21:43,606 --> 00:21:54,376
For instance, the Vela-35B-A3B model outperformed the base model by up to 11.5 points on user-related tasks according to VelaBench.
214
00:21:54,376 --> 00:22:01,086
The gains weren't just limited to user-centric tasks; there were also noticeable improvements in general benchmarks.
215
00:22:01,194 --> 00:22:02,514
Evan: And that’s not all.
216
00:22:02,514 --> 00:22:09,394
Their experiments indicated that scaling the dataset further boosts performance, making the approach even more robust.
217
00:22:09,438 --> 00:22:20,278
Ashley: Plus, the work extends the benefits to reinforcement learning, showing enhancements when models fine-tuned on their user-centric data underwent further reinforcement learning processes.
218
00:22:20,394 --> 00:22:32,714
Evan: So, in essence, they demonstrated a scalable approach that leverages real-world user behavior data to significantly improve both user-specific capabilities and broader LLM intelligence.
219
00:22:32,766 --> 00:22:41,706
Ashley: Their work opens the door to exciting future research directions, including the application of this approach to more diverse real-world scenarios and tasks.
220
00:22:41,826 --> 00:22:47,746
Evan: It’s fantastic to see such innovative work pushing the boundaries of what LLMs can achieve.
221
00:22:47,790 --> 00:22:50,350
Ashley: We hope you found this episode insightful.
222
00:22:50,350 --> 00:22:56,210
Make sure to tune in next time for more cutting-edge research from the world of AI and machine learning.
223
00:22:56,302 --> 00:23:01,422
Evan: And if you enjoyed today’s episode, don’t forget to subscribe and leave us a review.
224
00:23:01,422 --> 00:23:05,062
Your feedback helps us bring you the most relevant content.
225
00:23:05,062 --> 00:23:08,982
Until next time, thanks for listening to Daily Paper Cast.