अनुलिपि 🔗
WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 4:33:15 PM
Duration: 2734.666
Channels: 1
1
00:00:13.985 --> 00:00:18.530
Hello, and welcome to Podcast Dot in It, the podcast about Python and the people who make it great.
2
00:00:19.410 --> 00:00:26.150
When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it. So take a look at our friends over at Linode.
3
00:00:26.955 --> 00:00:34.975
With the launch of their managed Kubernetes platform, it's easy to get started with the next generation of deployment and scaling powered by the battle tested Linode platform,
4
00:00:35.360 --> 00:00:37.460
including simple pricing, node balancers,
5
00:00:37.760 --> 00:00:38.980
40 gigabit networking,
6
00:00:39.440 --> 00:00:43.220
dedicated CPU and GPU instances, and worldwide data centers.
7
00:00:43.935 --> 00:00:45.235
Go to python podcast.com/linode,
8
00:00:46.895 --> 00:00:56.790
that's l I n o d e, today and get a $100 credit to try out a Kubernetes cluster of your own. And don't forget to thank them for their continued support of this show.
9
00:00:57.489 --> 00:01:03.590
Subsurface Live is the Cloud Data Lake Conference, a virtual conference where data engineers, data scientists,
10
00:01:03.925 --> 00:01:09.145
data architects, and data analysts can gather and hear about cloud data lakes into the data ecosystem.
11
00:01:09.925 --> 00:01:11.945
Subsurface live winter of 2022
12
00:01:12.325 --> 00:01:16.080
includes keynote talks from Bill Inman, the father of the data warehouse,
13
00:01:16.700 --> 00:01:21.600
author of deep work Cal Newport, and several more from companies such as Dremio, AWS,
14
00:01:22.060 --> 00:01:23.440
DBT, and more.
15
00:01:23.885 --> 00:01:31.825
Subsurface will also have many breakout sessions featuring Panda's creator, Wes McKinney, Apache Superset, and Airflow creator, Maxime Beauchmann,
16
00:01:32.330 --> 00:01:43.115
and engineers from Apple, Uber, Adobe, Bloomberg, and more. Meet other data professionals and learn about the data technologies and practices helping companies meet their current and future data needs.
17
00:01:43.575 --> 00:01:46.235
Learn more and register for free today at pythonpodcast.com/subsurface.
18
00:01:49.110 --> 00:02:00.755
Your host as usual is Tobias Macy. And today, I'm interviewing Scott Rogowski about Code 2 Flow, a utility for generating pretty good call graphs for dynamic languages. So, Scott, can you start by introducing yourself?
19
00:02:01.134 --> 00:02:02.595
My name is Scott.
20
00:02:03.055 --> 00:02:07.909
In the past, I've been a full stack web engineer and a data engineer.
21
00:02:08.530 --> 00:02:12.550
Most recently, I led the analytics services team at a midsized
22
00:02:12.850 --> 00:02:14.950
accounting software company called Workiva.
23
00:02:15.765 --> 00:02:22.825
Currently, I'm on a break from programming, and I'm actually talking to you from far eastern Colombia near
Load full transcript
the border with Venezuela.
24
00:02:23.440 --> 00:02:26.660
I'm here with the organization called On the Ground International,
25
00:02:27.440 --> 00:02:31.459
and we're basically just working to address the Venezuelan refugee crisis.
26
00:02:31.925 --> 00:02:46.800
So I've been here for almost 2 months now, but I'll actually be leaving just next week. When I get back, I do plan to return to the software world, and I plan to split my time between CodeDeploy, which is what we are talking about today,
27
00:02:47.260 --> 00:02:51.145
and Mangita, which is my other project of some popularity,
28
00:02:51.925 --> 00:02:52.985
just to briefly
29
00:02:53.285 --> 00:02:56.665
introduce that. So that's a small implementation of MongoDB,
30
00:02:57.260 --> 00:03:01.200
which is meant to be a local dependency free drop in replacement
31
00:03:01.900 --> 00:03:07.954
for when you just don't wanna install MongoDB or can for some reason. But today, we are talking about CodeDeploy.
32
00:03:08.415 --> 00:03:13.394
Yeah. Definitely some interesting work that you're doing there. I could definitely see the appeal of having
33
00:03:14.450 --> 00:03:15.350
a not MongoDB
34
00:03:15.730 --> 00:03:19.830
implementation of MongoDB for local testing, but I'm also curious about
35
00:03:20.530 --> 00:03:25.845
your ventures in Colombia if that's something where you're actually using some of your software development capabilities
36
00:03:26.145 --> 00:03:35.180
to help with that endeavor or if it's some other skill set that you have that you're using to apply down there or if it's just abundance of goodwill and energy?
37
00:03:35.720 --> 00:03:37.820
I think it's the last option there.
38
00:03:38.255 --> 00:03:44.435
I'm certainly not using any of my software skills here. I'm helping them a little bit with some basic web design.
39
00:03:44.895 --> 00:03:52.190
But, no, essentially, I just wanted to take a little break and find a way to connect with people on a 1 to 1 level.
40
00:03:52.490 --> 00:03:58.645
But I am the sort of person who does like dealing with systems and software. So
41
00:03:58.945 --> 00:04:10.040
it is a break, any sort of a career change, but I am thoroughly enjoying my time here, and I really like the work we're doing. Well, it's definitely always great to talk to people who have that kind of drive and
42
00:04:10.340 --> 00:04:22.175
energy to be able to dedicate to helping other folks in whatever fashion that might take. So definitely glad to be able to speak with you and glad that you're able to take the time away from your work down there to speak to me about CodeDeploy.
43
00:04:22.635 --> 00:04:23.535
Yeah. Absolutely.
44
00:04:23.840 --> 00:04:28.260
And do you remember how you first got introduced to programming and how that led you to Python?
45
00:04:28.640 --> 00:04:36.935
Yeah. So I've been programming since a super young age. I remember actually starting on the PI 83 calculators
46
00:04:37.315 --> 00:04:38.615
back in 8th grade.
47
00:04:38.995 --> 00:04:41.975
Basically, we used to trade games on these calculators.
48
00:04:42.650 --> 00:04:48.430
Being what it was, you could actually go in and, like, view the source code of what was in these games.
49
00:04:48.810 --> 00:04:49.310
So
50
00:04:49.745 --> 00:04:54.165
I decided 1 day to basically just I wanted to make my own game.
51
00:04:54.545 --> 00:05:09.415
So I looked at a program that was already on my calculator. I just kinda figured out how to do it through some horrid programming that was a lot of go to's and labels, but it worked. And that was my intro into the programming world.
52
00:05:09.875 --> 00:05:13.815
For me, Python came much later near the end of college.
53
00:05:14.260 --> 00:05:21.800
So I was the treasurer of the university hiking club, and we had this weird system at the time of getting payment for people
54
00:05:22.125 --> 00:05:26.065
who wanted to go on hikes that involved this complicated spreadsheet.
55
00:05:26.765 --> 00:05:27.265
So
56
00:05:27.645 --> 00:05:32.250
me being the person who was the CS major in our group,
57
00:05:32.550 --> 00:05:35.050
I was tasked with fixing this and making it better.
58
00:05:35.430 --> 00:05:41.745
So I had heard that Python was an easy language for these sorts of things, basically, just gluing different things together.
59
00:05:42.285 --> 00:05:50.710
So we had to connect a payment processor to a spreadsheet and also generate some sort of a confirmation page. So all these different APIs working together,
60
00:05:51.185 --> 00:05:53.365
and Python works great for it.
61
00:05:53.745 --> 00:05:58.245
From there, in every job that I've had, I've been using Python ever since.
62
00:05:58.710 --> 00:06:00.090
I still write JavaScript
63
00:06:00.550 --> 00:06:22.699
here and there and c plus plus when I need something really, really fast, but I do maybe 90% of my programming in Python. At some point in that journey, that led you to building this Code to Flow program that we're talking about today. So I'm wondering if you can describe a bit about what it is that you're building there and some of the story behind how it came to be and why you decided that you wanted to put in the effort to bring it into reality.
64
00:06:23.345 --> 00:06:29.525
So CodeDeploy is a way to generate call graphs, which you can also look at as flowcharts
65
00:06:30.225 --> 00:06:31.604
from your source code.
66
00:06:32.110 --> 00:06:32.610
So
67
00:06:32.990 --> 00:06:35.570
when you have a program and you have
68
00:06:36.030 --> 00:06:37.730
functions calling other functions
69
00:06:38.190 --> 00:06:41.625
and you want a map of that to broadly understand
70
00:06:42.005 --> 00:06:46.185
the structure of your program, this is what you use CodeDeploy for.
71
00:06:46.565 --> 00:06:53.110
CodeDeploy works on dynamic languages and currently supports Python, JavaScript, Ruby, and PHP.
72
00:06:53.729 --> 00:06:55.910
The origin story is pretty interesting.
73
00:06:56.365 --> 00:07:03.425
So it was now 9 years ago, and I had to look back through the commit history to figure out exactly when I started on this.
74
00:07:03.759 --> 00:07:09.380
But at the time, I was about 1 year out of college, and I was working on this project called Event Tsunami.
75
00:07:09.759 --> 00:07:21.705
Event Tsunami was 1 of these maybe 1,000 failed attempts at what I view now retrospect as the most common and maybe worst idea in the history of startups, which is an event aggregator.
76
00:07:22.500 --> 00:07:28.759
So this is something that still doesn't exist today if you're curious about what's happening downtown, what cool things there are to do.
77
00:07:29.220 --> 00:07:32.165
There's not really a place to find this sort of thing.
78
00:07:32.885 --> 00:07:41.065
So, anyways, I was young when you're out of college, and, I was writing this thing, and I had this very long JavaScript file.
79
00:07:41.440 --> 00:07:50.980
And there was a point where I looked at this 1 day and said, I don't understand this anymore. I would like to try to understand the structure, how everything is connecting together.
80
00:07:51.665 --> 00:07:55.845
So 1 day, I spent maybe 2 or 3 hours
81
00:07:56.465 --> 00:07:58.805
with a series of regular expressions
82
00:07:59.539 --> 00:08:05.960
writing this thing that pulled out all of the functions and all the function definitions and connected them together.
83
00:08:06.635 --> 00:08:16.789
And I saw a graph at the end of this, and I thought that was really cool. So I showed my roommate. He thought it was really neat as well, and I decided to put it online.
84
00:08:17.250 --> 00:08:25.205
I didn't think that would come of it, but I thought that it was worth a shot, and it proved to be a little more popular than I expected actually.
85
00:08:25.905 --> 00:08:28.485
But either way, that was 9 years ago.
86
00:08:28.919 --> 00:08:33.339
So fast forward to last year, actually, and I'm contacted
87
00:08:33.720 --> 00:08:34.620
out of the blue
88
00:08:35.080 --> 00:08:38.805
by a small Japanese startup called Cider,
89
00:08:39.425 --> 00:08:43.365
and they wanted to use CodeDeploy for what they were doing.
90
00:08:43.665 --> 00:08:47.810
So I thought that this was a good idea. They offered to pay me money for this.
91
00:08:48.610 --> 00:08:54.310
But at the time, everything was horrid with this. I was 1 year out of college when I wrote the original version,
92
00:08:54.850 --> 00:08:55.830
and there were
93
00:08:56.165 --> 00:08:58.504
really strange things that I was doing. Like,
94
00:08:58.805 --> 00:09:03.785
instead of using abstract syntax trees, I was counting the
95
00:09:04.160 --> 00:09:07.060
characters of indentation in order to try to determine
96
00:09:07.920 --> 00:09:12.980
how large a Python function was and try to get the scope of this.
97
00:09:13.325 --> 00:09:18.145
So I was doing all these weird things that you would only do when you're straight out of college,
98
00:09:18.845 --> 00:09:20.145
and I asked them
99
00:09:20.500 --> 00:09:28.680
if they would let me rewrite it as well. And they said okay. So I went ahead and rewrote it, and it's in a very good place now, I think.
100
00:09:28.985 --> 00:09:33.885
But I'm happy about it, and it's been growing pretty steadily ever since then.
101
00:09:34.425 --> 00:09:47.595
And so you mentioned that some of the original impetus for building this project was to be able to understand some of your own code. And I'm curious if you can talk to some of the ways that being able to generate a call graph will help
102
00:09:47.915 --> 00:09:48.574
the software
103
00:09:48.875 --> 00:09:59.140
development process and some of the types of insight that you can gain by being able to convert the raw source code into some visual representation to be able to explore it in that manner?
104
00:09:59.600 --> 00:10:00.420
Yeah. Absolutely.
105
00:10:00.959 --> 00:10:01.459
So
106
00:10:02.080 --> 00:10:08.135
to back up on that question a little bit, I think that humans are primarily
107
00:10:08.755 --> 00:10:09.735
visual people.
108
00:10:10.274 --> 00:10:12.455
And source code is this
109
00:10:13.160 --> 00:10:18.620
weird agreement that we've made with the computers that says that we will write this thing
110
00:10:19.000 --> 00:10:22.060
that you can kind of understand and we can kind of understand.
111
00:10:22.665 --> 00:10:26.285
And this is the agreement we have, and this is how we're gonna communicate together.
112
00:10:26.904 --> 00:10:28.444
But I think that
113
00:10:28.745 --> 00:10:30.899
we, as humans, really
114
00:10:31.279 --> 00:10:34.180
like the idea of being able to understand
115
00:10:35.200 --> 00:10:36.740
these complex programs
116
00:10:37.445 --> 00:10:39.545
through the use of flowcharts,
117
00:10:40.245 --> 00:10:42.425
for example, or other visual representations.
118
00:10:43.525 --> 00:10:49.040
So Code 2flow produces this, and I view it as being most useful
119
00:10:49.660 --> 00:10:50.160
for
120
00:10:50.940 --> 00:10:51.760
new developers.
121
00:10:52.300 --> 00:10:59.285
For people who are just getting started with a large code base when you're coming into a company and you have no idea
122
00:11:00.305 --> 00:11:02.725
what anything does in the codebase.
123
00:11:03.250 --> 00:11:05.750
I think that that's the time to use CodeDeploy
124
00:11:06.130 --> 00:11:06.630
to
125
00:11:06.930 --> 00:11:13.404
try to get this broad, high level overview of how everything fits together before you dive in a little deeper.
126
00:11:13.945 --> 00:11:20.045
In terms of the visual representation of it, I'm wondering if you can talk through some of the
127
00:11:20.680 --> 00:11:22.700
ways that you're able to convert these
128
00:11:23.079 --> 00:11:23.579
semantic
129
00:11:23.959 --> 00:11:28.540
and structural components of the software project into
130
00:11:29.115 --> 00:11:33.775
visual elements to be able to help somebody tie together
what the actual
131
00:11:34.315 --> 00:11:41.560
logical flow is through the program where you say you've got function definitions, you've got class definitions, you've got instantiations of classes,
132
00:11:42.100 --> 00:11:47.134
executions of functions, parameters passed to functions, and being able to convert all of that into
133
00:11:47.595 --> 00:12:02.629
some visual diagram to be able to say, okay. These are the different pieces. These are how they interact. Now I want to be able to follow the logical flow from this function definition through to where it's called, through to what the output might be, and just some of the ways that you've thought about that
134
00:12:03.075 --> 00:12:06.615
sort of mapping between the semantic and the visual
135
00:12:07.235 --> 00:12:08.375
components of it?
136
00:12:08.755 --> 00:12:10.455
So the basic units
137
00:12:11.350 --> 00:12:12.010
of CodeDeploy
138
00:12:12.550 --> 00:12:14.089
is the function.
139
00:12:14.950 --> 00:12:22.305
And within every function, you have a variety of function calls that map to different functions.
140
00:12:23.005 --> 00:12:23.905
There's also
141
00:12:25.005 --> 00:12:28.705
on the outside of every function, there's usually some namespace,
142
00:12:30.110 --> 00:12:34.850
some class representation that you can kind of contain a function into.
143
00:12:35.550 --> 00:12:40.595
So the hope is that by looking at it from this level, from the
144
00:12:40.975 --> 00:12:46.915
sets of 1 function calling another function, you can get the structural representation.
145
00:12:47.900 --> 00:12:56.320
In terms of the sort of utility of a visual flowchart of this call graph for the purposes of being able to
146
00:12:56.775 --> 00:13:03.115
explore and and understand the code base. I'm curious how you think about that in comparison to projects such as
147
00:13:03.415 --> 00:13:04.955
Sourcegraph that is taking
148
00:13:05.450 --> 00:13:17.285
more of an indexing style approach of being able to say, okay. I've got this function. Now I wanna see all the places that it's called and being able to jump around a project from the very sort of semantic and structural aspect of it
149
00:13:17.665 --> 00:13:26.810
versus this visual representation and just some of the differences in terms of the utilities or the use cases that each of those approaches might provide?
150
00:13:27.350 --> 00:13:29.130
On the high level,
151
00:13:29.965 --> 00:13:30.865
they're all
152
00:13:31.485 --> 00:13:32.465
developer tools.
153
00:13:33.085 --> 00:13:36.545
And I think that you wanna use the right tool for the right job.
154
00:13:37.005 --> 00:13:37.505
Sourcegraph
155
00:13:38.260 --> 00:13:40.360
has a job in terms
156
00:13:40.980 --> 00:13:44.600
of jumping around to projects, being able to go quickly
157
00:13:45.300 --> 00:13:47.320
from lines of code to lines of code.
158
00:13:48.325 --> 00:13:53.464
And where Code 2flow fits in is understanding the high level representation.
159
00:13:54.325 --> 00:13:59.430
In terms of being able to understand that high level representation, once somebody has been able
160
00:13:59.890 --> 00:14:16.250
to grasp, okay, these are the different objects that exist in this semantic graph, What are some of the next steps that they might take to be able to say, okay, I now know that, you know, this is a function that is tied to 5 other places based on the lines that are connecting to it in this representation
161
00:14:17.110 --> 00:14:28.654
and then saying, okay. Now I'm going to dig into this function and understand more about the logic that's embedded within it. And maybe, you know, from this visual representation, I can identify some potential opportunities for
162
00:14:28.955 --> 00:14:37.980
refactoring or further exploration and just some of the sort of next steps somebody might take after they have that initial visual exploration of the project?
163
00:14:38.600 --> 00:14:39.980
So I am agnostic
164
00:14:40.600 --> 00:14:41.740
about how
165
00:14:42.635 --> 00:14:43.935
developers use
166
00:14:44.394 --> 00:14:45.535
CodeDeploy specifically,
167
00:14:46.235 --> 00:14:49.615
but there are a couple of obvious
168
00:14:49.970 --> 00:14:54.709
use cases and next steps. Once they have this visual representation,
169
00:14:55.490 --> 00:14:57.110
they can then proceed onwards.
170
00:14:57.425 --> 00:15:02.805
The first is we've talked about is obvious, visualizing the internal structure,
171
00:15:03.264 --> 00:15:04.964
understanding that high level overview.
172
00:15:05.649 --> 00:15:09.910
The second, maybe a little less obvious, is finding orphan code.
173
00:15:10.529 --> 00:15:13.029
So this is something that is especially
174
00:15:14.165 --> 00:15:17.305
from the perspective of a language like JavaScript
175
00:15:18.005 --> 00:15:19.865
where you can often have
176
00:15:20.245 --> 00:15:25.420
code that is being bundled and being shipped to the end user,
177
00:15:25.880 --> 00:15:27.660
to the client side browser,
178
00:15:28.280 --> 00:15:30.940
but is not being used for anything.
179
00:15:31.455 --> 00:15:34.835
And when you have code like this, it literally slows down your website.
180
00:15:35.375 --> 00:15:36.915
But when you have
181
00:15:37.455 --> 00:15:41.130
code that's in any language that's not being used by anything,
182
00:15:41.510 --> 00:15:44.810
this is extra cognitive load for the developers,
183
00:15:45.725 --> 00:15:51.825
and it's best to eliminate this when possible. So Code 2flow is actually useful
184
00:15:52.445 --> 00:15:55.265
in finding this orphan code and
185
00:15:55.790 --> 00:16:00.290
determining that, okay, nothing actually connects to this thing, and we can delete this code.
186
00:16:00.910 --> 00:16:03.090
The other use case that I'll mention
187
00:16:03.715 --> 00:16:09.015
is being able to analyze the risk of changes that you might make.
188
00:16:09.395 --> 00:16:13.010
So if you have a pull request that touches
189
00:16:13.550 --> 00:16:19.810
1 function and you're a little bit concerned about it as you might be if it's a
190
00:16:20.455 --> 00:16:22.875
system that's in some way business critical.
191
00:16:23.495 --> 00:16:25.275
To be able to see
192
00:16:25.815 --> 00:16:30.750
the functions that are up stream and downstream of this function and try to understand
193
00:16:31.770 --> 00:16:35.710
what actually is the risk to the overall business
194
00:16:36.165 --> 00:16:38.105
based on this pull request.
195
00:16:38.485 --> 00:16:40.904
I think that that's also a useful
196
00:16:41.205 --> 00:16:42.345
use case of CodeDeploy.
197
00:16:42.964 --> 00:16:49.840
And so in terms of the implementation of CodeDeploy, I'm wondering if you can describe a bit about some of the architectural
198
00:16:50.220 --> 00:16:56.025
and design elements that you have built into it now that you've had the opportunity to learn from
199
00:16:56.565 --> 00:17:06.120
iterating on it for a few years with this regex focused approach and being able to revisit it with this paid engagement that has allowed you to rearchitect it from the ground up.
200
00:17:06.580 --> 00:17:07.080
The
201
00:17:07.380 --> 00:17:08.980
difference now is that
202
00:17:10.005 --> 00:17:12.245
well, for 1, the code is all better.
203
00:17:13.525 --> 00:17:21.039
And it's hard to describe what better is except for it's the results of 10 years of additional engineering experience.
204
00:17:21.659 --> 00:17:23.919
But to speak to the
205
00:17:24.220 --> 00:17:25.919
architectural element specifically,
206
00:17:26.655 --> 00:17:27.875
I have it broadly
207
00:17:28.175 --> 00:17:29.235
divided into
208
00:17:29.535 --> 00:17:30.755
3 files.
209
00:17:31.455 --> 00:17:37.740
So the first file is called engine. Py, and that's what starts everything off and is universal
210
00:17:38.440 --> 00:17:39.820
amongst all the languages.
211
00:17:40.519 --> 00:17:42.860
And that file will call
212
00:17:43.575 --> 00:17:48.235
various functions. I believe that there's 5 of them in each individual
213
00:17:48.775 --> 00:17:50.635
language file. So
214
00:17:51.340 --> 00:17:53.519
the tricky part has been managing
215
00:17:54.139 --> 00:17:54.879
the differences
216
00:17:55.179 --> 00:17:58.240
between each language because as you've noted,
217
00:17:58.860 --> 00:17:59.759
all of these
218
00:18:00.195 --> 00:18:02.135
4 languages are very different.
219
00:18:03.075 --> 00:18:05.095
So they all do their
220
00:18:05.475 --> 00:18:08.695
AST. Typically, they all have different rules for
221
00:18:09.235 --> 00:18:09.735
scope
222
00:18:10.630 --> 00:18:13.049
or doing the function calls, all of these things.
223
00:18:13.590 --> 00:18:14.090
So
224
00:18:14.549 --> 00:18:17.130
attempting to abstract away
225
00:18:17.855 --> 00:18:18.595
the interface
226
00:18:19.135 --> 00:18:20.595
so that you have this
227
00:18:21.135 --> 00:18:21.635
universal
228
00:18:22.015 --> 00:18:27.789
interface to
each language, and then each language handles the individual details has been a challenge.
229
00:18:28.490 --> 00:18:30.429
So to clarify, the second
230
00:18:30.890 --> 00:18:34.745
file that I was discussing in that 3 file hierarchy are the individual
231
00:18:35.045 --> 00:18:41.305
files for each language for there's a Python dot py, JavaScript dot py, Ruby dot py, PHP dot py.
232
00:18:41.899 --> 00:18:42.399
The
233
00:18:42.940 --> 00:18:46.320
3rd primary file in this hierarchy is just called model.py,
234
00:18:47.100 --> 00:18:49.919
and that's where I handle all of the
235
00:18:50.705 --> 00:18:52.645
internal things related to
236
00:18:53.665 --> 00:18:56.805
what is a function call, what is a variable,
237
00:18:57.425 --> 00:18:59.045
what do all these things mean?
238
00:18:59.570 --> 00:19:05.029
In terms of the actual target languages, I'm wondering what have been the driving forces
239
00:19:05.330 --> 00:19:05.830
to
240
00:19:06.289 --> 00:19:06.789
which
241
00:19:07.184 --> 00:19:14.325
language targets you decided to actually implement and focus on and sort of curious what types of projects have
242
00:19:14.710 --> 00:19:19.690
introduced the different requirements that led you to those different run times.
243
00:19:20.309 --> 00:19:21.690
Yeah. So the original
244
00:19:22.390 --> 00:19:25.975
languages were just JavaScript and Python.
245
00:19:26.595 --> 00:19:34.370
And the only reason for that was because I worked primarily in those 2 languages, and those were the ones that I was most interested in.
246
00:19:34.670 --> 00:19:40.130
Moving on from there, when CIDR had contacted me to update this project,
247
00:19:40.590 --> 00:19:46.155
they had requested the 2 additional dynamic languages that we now have, which are Ruby and PHP.
248
00:19:46.855 --> 00:19:49.515
Someday, I would like to add more languages.
249
00:19:50.295 --> 00:19:56.740
But for the moment, these are essentially the most used dynamic languages in the world.
250
00:19:57.360 --> 00:19:57.860
And
251
00:19:58.365 --> 00:20:03.505
if anything, I think that I would move into static languages yet and maybe try to pull in
252
00:20:03.805 --> 00:20:06.305
something like Java, Erlang, Rust.
253
00:20:06.850 --> 00:20:16.390
Given the fact that you are focused on being able to support generating these call graphs for these various dynamic languages, I'm wondering what are some of the
254
00:20:16.914 --> 00:20:22.215
edge cases that you've had to deal with across these language run times, how you've approached the actual
255
00:20:22.515 --> 00:20:27.160
implementation of being able to generate and parse the respective ASTs
256
00:20:27.540 --> 00:20:28.040
and
257
00:20:28.580 --> 00:20:31.160
how the respective ASTs
258
00:20:32.100 --> 00:20:37.894
structure themselves and sort of what that lowest common denominator ends up being across those run times?
259
00:20:38.434 --> 00:20:44.370
So I think that that's a couple of different questions.
The first 1 I'll address is the ASTs.
260
00:20:45.310 --> 00:20:47.250
So every language
261
00:20:47.550 --> 00:20:49.330
AST is different.
262
00:20:49.955 --> 00:20:50.455
And,
263
00:20:50.755 --> 00:20:57.415
essentially, how I get the ASTs from each individual language is to use their respective parsers.
264
00:20:58.035 --> 00:20:59.015
So I
265
00:20:59.639 --> 00:21:00.139
generate
266
00:21:01.159 --> 00:21:02.380
a system call
267
00:21:02.919 --> 00:21:05.340
to pull in the ASTs,
268
00:21:05.799 --> 00:21:10.535
and they're usually in some sort of a JSON or JSON like representation.
269
00:21:12.035 --> 00:21:13.575
And then from there,
270
00:21:14.035 --> 00:21:21.860
you can handle them in very similar ways, though each AST is different and each AST requires different approach.
271
00:21:22.400 --> 00:21:26.054
When I was working on this originally, my strategy,
272
00:21:26.595 --> 00:21:27.095
because
273
00:21:27.634 --> 00:21:32.134
there are a ton of edge cases when you're doing this sort of thing,
274
00:21:32.514 --> 00:21:33.495
was to
275
00:21:34.260 --> 00:21:35.000
pull in
276
00:21:35.540 --> 00:21:36.040
increasingly
277
00:21:37.220 --> 00:21:37.880
more complex
278
00:21:38.820 --> 00:21:44.485
features of a language or simply just increasingly more complex projects.
279
00:21:44.945 --> 00:21:47.684
And 1 by 1, handle the edge cases
280
00:21:47.985 --> 00:21:48.485
and
281
00:21:48.945 --> 00:21:49.445
check
282
00:21:49.905 --> 00:21:50.405
visually
283
00:21:50.785 --> 00:21:51.684
that the
284
00:21:52.370 --> 00:21:55.270
call graph was what was actually happening
285
00:21:55.649 --> 00:21:56.309
in the
286
00:21:56.610 --> 00:21:57.750
individual projects.
287
00:21:58.289 --> 00:22:03.505
And then as far as being able to turn the AST into a visual representation,
288
00:22:04.445 --> 00:22:08.385
what are some of the tweaks that you've had to make to be able to
289
00:22:09.100 --> 00:22:16.320
convert them into a common representation that you can then feed into that engine that will spit out the visualization
290
00:22:16.620 --> 00:22:21.165
from it? So that in itself, if I'm understanding your question correctly,
291
00:22:21.545 --> 00:22:23.805
has not been especially difficult.
292
00:22:24.425 --> 00:22:25.725
The common representation
293
00:22:26.425 --> 00:22:28.390
that I have is to
294
00:22:29.010 --> 00:22:30.070
have, I believe,
295
00:22:30.690 --> 00:22:32.390
3 different kinds of units.
296
00:22:32.690 --> 00:22:40.764
So you have the node, which is essentially a function. You have the edge, which is a function call, and you have the group,
297
00:22:41.304 --> 00:22:43.644
which is a namespace or a class.
298
00:22:44.250 --> 00:22:46.750
So everything gets converted into
299
00:22:47.050 --> 00:22:47.550
these
300
00:22:47.850 --> 00:22:48.830
3 elements.
301
00:22:49.610 --> 00:22:55.005
And from there, you can write that directly to this intermediate file format
302
00:22:55.385
--> 00:22:57.005
that graphviz uses.
303
00:22:57.385 --> 00:22:59.005
And graphviz is the
304
00:22:59.465 --> 00:23:00.445
tool to
305
00:23:01.100 --> 00:23:06.240
actually generate the visual graph. So once it's in this intermediate representation,
306
00:23:07.340 --> 00:23:11.505
you pull it into graphviz. Graphviz does its calculations. It determines
307
00:23:12.125 --> 00:23:12.625
where
308
00:23:13.165 --> 00:23:15.745
it would make the most sense to put
309
00:23:16.125 --> 00:23:17.185
every individual
310
00:23:17.645 --> 00:23:20.465
node, every individual group in order to
311
00:23:22.240 --> 00:23:23.860
make the graph as readable,
312
00:23:24.559 --> 00:23:25.940
and as clean as possible.
313
00:23:26.400 --> 00:23:32.455
Because of the fact that you are using graphviz for that visual element of it, I'm curious. What are some of
314
00:23:32.835 --> 00:23:36.055
the tweaks that you've had to make to be able to make that output useful
315
00:23:36.740 --> 00:23:43.880
where it's a static image and so you need to be able to make sure that it is uncluttered enough to be useful but
316
00:23:44.335 --> 00:23:46.435
informative enough to be
317
00:23:46.975 --> 00:23:51.315
representative of the program and some of the ways that you've thought about the
318
00:23:51.775 --> 00:23:54.409
constraints of that static representation
319
00:23:54.870 --> 00:23:56.409
of that sort of point in time
320
00:23:56.789 --> 00:23:58.250
representation of the program,
321
00:23:58.950 --> 00:24:02.250
how to structure that in a way that is
322
00:24:02.985 --> 00:24:04.365
relevant to the
323
00:24:04.825 --> 00:24:11.645
sort of tasks that somebody might be trying to achieve by virtue of using CodeDeploy to generate that representation?
324
00:24:12.639 --> 00:24:14.580
So there's 2 parts to that question.
325
00:24:14.960 --> 00:24:19.059
The first is that Graphviz does a lot of work for you.
326
00:24:19.360 --> 00:24:19.860
So
327
00:24:20.245 --> 00:24:22.825
Graphviz does have this internal algorithm
328
00:24:23.445 --> 00:24:29.145
where you can feed it any sort of directed graph, and it's going to be able to
329
00:24:29.620 --> 00:24:38.440
place things in as uncluttered of a way as possible. But the second part of that is that no matter what, when you have projects
330
00:24:38.985 --> 00:24:41.005
of greater than trivial complexity,
331
00:24:41.625 --> 00:24:45.085
the output can get very messy very quickly.
332
00:24:45.740 --> 00:24:48.080
So what I've done with that is
333
00:24:48.620 --> 00:24:51.200
implement a couple of command line arguments
334
00:24:51.740 --> 00:24:52.960
so that you can
335
00:24:53.384 --> 00:24:55.804
zoom in a little bit more into
336
00:24:56.264 --> 00:24:56.764
exactly
337
00:24:57.225 --> 00:25:00.080
the parts of the code that you're interested in.
338
00:25:00.640 --> 00:25:07.860
And I'll do 1 that's simple to explain and 1 that's a little more complex to explain. So 1 that's simple to explain is
339
00:25:08.164 --> 00:25:09.705
if you just want to
340
00:25:10.085 --> 00:25:11.304
exclude namespaces,
341
00:25:12.245 --> 00:25:24.850
if there are classes you don't care about, if there's functions you don't care about, there's command line arguments for that. So just to pull out the things that you are very sure you don't care about. But if you're sure that you actually care
342
00:25:25.390 --> 00:25:30.855
specifically about 1 thing, I have a command line argument that's called target function.
343
00:25:31.315 --> 00:25:32.135
So you can
344
00:25:32.435 --> 00:25:38.830
zoom in on this 1 very specific function. And then with that command line argument, there's 2 more arguments, which are
345
00:25:39.210 --> 00:25:41.305
downstream depth and upstream depth.
346
00:25:41.865 --> 00:25:43.005
So if you
347
00:25:43.385 --> 00:25:45.005
zoom in to, say,
348
00:25:45.305 --> 00:25:58.005
my fortune a and you have upstream depth of 2, you will be able to see 2 levels up everything that is calling this function. And if you do downstream depth of 3, you can,
349
00:25:58.465 --> 00:26:02.245
in the same way, see 3 levels down what's happening there.
350
00:26:02.625 --> 00:26:04.565
And after I implemented that,
351
00:26:04.990 --> 00:26:10.130
the graphs became a whole lot cleaner and a whole lot easier to read.
352
00:26:10.430 --> 00:26:12.690
And then as far as being able to actually
353
00:26:13.715 --> 00:26:18.775
apply code to flow to a given project, I'm curious if you can talk through some of the workflow
354
00:26:19.235 --> 00:26:20.054
of saying,
355
00:26:20.515 --> 00:26:28.679
I have this project. I want to apply code to flow to it to be able to visualize the representation. What are some of the limitations in terms of the
356
00:26:29.220 --> 00:26:40.995
scale and complexity of the project as far as the utility of CodeDeploy and its look ability to it, and how you might scope the specific execution of CodeDeploy to be able to
357
00:26:41.600 --> 00:26:42.100
componentize
358
00:26:42.400 --> 00:26:48.340
the different elements of that project if it is at a sufficient scale where running it across the entirety
359
00:26:48.640 --> 00:26:49.620
is infeasible?
360
00:26:50.495 --> 00:26:51.315
So CodeDeploy
361
00:26:51.695 --> 00:26:55.475
by itself is easy to apply to a project.
362
00:26:55.935 --> 00:26:57.795
It's not a complex
363
00:26:58.259 --> 00:27:10.335
installation process. So just do hit install code to flow. If you're using 1 of the 3 languages that isn't Python, you might have to install some dependencies for those as well, and that's well documented.
364
00:27:11.034 --> 00:27:14.095
And then all you do is do code to flow
365
00:27:14.490 --> 00:27:17.390
path to either my project directory
366
00:27:17.850 --> 00:27:19.390
or path to my
367
00:27:19.930 --> 00:27:20.990
individual file.
368
00:27:21.345 --> 00:27:24.805
And it will run. And then you will be able to see the
369
00:27:25.345
--> 00:27:26.245
visual representation
370
00:27:26.625 --> 00:27:27.525
of your code.
371
00:27:27.825 --> 00:27:32.690
You can dig in a little bit more with the command line arguments that we talked a little bit.
372
00:27:33.070 --> 00:27:36.929
But in terms of just applying it to individual
373
00:27:37.230 --> 00:27:40.865
parts of your code to try to get a more focused overview.
374
00:27:41.325 --> 00:27:44.225
I find that it is very helpful to
375
00:27:44.925 --> 00:27:52.880
apply it just to individual modules, just to individual files if you can, rather than project as a whole. As far as
376
00:27:53.260 --> 00:27:53.760
the
377
00:27:54.380 --> 00:27:58.965
dynamic nature of the languages that you're targeting. I'm curious. What are some of the
378
00:27:59.424 --> 00:27:59.924
limitations
379
00:28:00.544 --> 00:28:03.205
in terms of what Code2flow is able to
380
00:28:03.505 --> 00:28:05.684
interpret and visualize, particularly
381
00:28:05.985 --> 00:28:06.480
when
382
00:28:07.360 --> 00:28:10.900
the program author starts delving into things like metaprogramming
383
00:28:11.440 --> 00:28:13.700
or monkey patching or
384
00:28:14.000 --> 00:28:14.820
some of these
385
00:28:15.505 --> 00:28:24.005
ways that dynamic languages can be very powerful, but also can very easily be abused and start to become too clever for their own good?
386
00:28:24.370 --> 00:28:26.870
Sure. So that's a good question,
387
00:28:27.170 --> 00:28:30.870
and it really does touch at a key points
388
00:28:31.170 --> 00:28:32.955
of what CodeDeploy is.
389
00:28:33.355 --> 00:28:34.414
So the slogan
390
00:28:34.715 --> 00:28:39.135
of CodeDeploy that I have on GitHub is pretty good call graphs.
391
00:28:39.755 --> 00:28:45.120
And the important distinction there is that it is not perfect call graphs. And
392
00:28:45.500 --> 00:28:48.799
what CodeDeploy is doing is using a lot of clever
393
00:28:49.100 --> 00:28:49.600
heuristics
394
00:28:50.394 --> 00:28:53.534
to get to where it's going. There are a couple places
395
00:28:53.835 --> 00:28:54.335
where
396
00:28:54.955 --> 00:29:00.399
not just the lack of heuristics, but just the nature of dynamic languages like Python
397
00:29:00.779 --> 00:29:09.995
makes it very difficult or impossible to generate perfect call graph, and I'll go over 2 of those here. So the first is when you have factory functions.
398
00:29:10.615 --> 00:29:15.115
So the example would be is if you have a factory that takes a single
399
00:29:15.895 --> 00:29:16.395
parameter
400
00:29:17.100 --> 00:29:22.720
and depending on that single parameter returns 1 of 2 functions, either function a or function b.
401
00:29:23.340 --> 00:29:33.625
So let's just say for simplicity that parameter is a boolean. And if the boolean is true, it returns function a. If the boolean is false, it returns function b. You do not know until
402
00:29:34.005 --> 00:29:38.190
run time what's the value of that Boolean is, and therefore, you don't know
403
00:29:38.650 --> 00:29:46.565
in what way that's gonna connect. During the design phase of this, I had the option of doing 1 of 2 directions,
404
00:29:47.105 --> 00:29:48.165
which was either
405
00:29:48.545 --> 00:29:52.005
put both functions in and say that this particular
406
00:29:53.029 --> 00:29:55.690
node connected to both function a and function b,
407
00:29:56.230 --> 00:29:58.809
or just say that it connected to neither.
408
00:29:59.110 --> 00:30:05.035
And I decided that the safe thing to do was to say that it connected to neither in this case.
409
00:30:05.495 --> 00:30:06.235
The second
410
00:30:06.535 --> 00:30:11.330
example that I'd like to share is when you have a dictionary of functions. So
411
00:30:11.630 --> 00:30:15.410
in my view, this is a very good programming practice when
412
00:30:15.950 --> 00:30:17.890
depending on some string,
413
00:30:18.355 --> 00:30:24.455
you pull out a certain function to use for a certain case. I see it most often in very clean code.
414
00:30:24.995 --> 00:30:30.440
But in this case, again, you don't know what function is coming out when you're actually
415
00:30:31.140 --> 00:30:34.280
pulling it out until you're actually in runtime.
416
00:30:34.820 --> 00:30:36.440
So that is the limitation
417
00:30:36.820 --> 00:30:37.960
of this static
418
00:30:38.965 --> 00:30:41.225
approach. But, again, you get 90%
419
00:30:41.765 --> 00:30:42.905
of the structure
420
00:30:43.205 --> 00:30:44.105
of a project.
421
00:30:44.725 --> 00:30:45.225
And
422
00:30:45.605 --> 00:30:47.700
as far as generating this
423
00:30:48.100 --> 00:30:49.240
understanding and
424
00:30:49.700 --> 00:30:54.679
helping programmers along with their task, I believe that the 90% is very helpful.
425
00:30:55.424 --> 00:30:57.285
And I believe that
426
00:30:57.825 --> 00:31:07.460
this is actually the reason why we don't see more things like CodeDeploy because a lot of programmers get scared away by this sort of thing and say, well, if it's not a 100%, we shouldn't do it.
427
00:31:07.919 --> 00:31:11.120
But I do believe that having a little bit there
428
00:31:11.935 --> 00:31:16.035
and it's not a little bit, it's the majority of the project. Having the majority of the project there
429
00:31:16.415 --> 00:31:19.795
is incredibly helpful, especially if you're a new developer.
430
00:31:20.559 --> 00:31:33.225
Given the fact that you're focused currently on a static analysis model, I'm curious, what are some of the additional types of information that you might be able to pull in and some of the ways that it might enhance the visual representation of the code structure
431
00:31:33.684 --> 00:31:41.780
if you move to a profiling based model or something similar to Pylint where you're actually executing the code to be able to analyze these
432
00:31:42.240 --> 00:31:42.740
runtime
433
00:31:43.200 --> 00:31:53.745
requirements where you're saying before about not being able to determine at runtime whether it's linked to function a or function b, just some of the additional pieces that are
434
00:31:54.125 --> 00:31:54.625
useful
435
00:31:55.210 --> 00:31:59.310
based on that runtime information versus just this static analysis approach.
436
00:32:00.010 --> 00:32:00.750
Yeah. Absolutely.
437
00:32:01.050 --> 00:32:09.775
And I think that you touched on a good point that I'd also mentioned, which is that without being able to see what is happening at the run time,
438
00:32:10.235 --> 00:32:17.590
it's very hard to see these I don't know if I'd call them edge cases, but I'll just call them edge cases for the sake of arguments.
439
00:32:17.970 --> 00:32:22.789
And it's certainly something that I would like to do is to move to a runtime approach.
440
00:32:23.315 --> 00:32:28.054
I believe that there are still benefits to the static approach, specifically
441
00:32:28.995 --> 00:32:29.495
in
442
00:32:30.115 --> 00:32:30.615
showing
443
00:32:31.480 --> 00:32:33.100
the paths that are
444
00:32:33.640 --> 00:32:40.299
not traveled very often in addition to the so called happy paths. And when you are
445
00:32:41.515 --> 00:32:44.415
a developer and trying to figure out
446
00:32:44.955 --> 00:32:45.855
what is
447
00:32:46.475 --> 00:32:50.015
happening or trying to address a certain bug,
448
00:32:50.480 --> 00:32:51.700
That bug is
449
00:32:52.080 --> 00:32:53.140
probably there
450
00:32:53.600 --> 00:32:59.460
because somebody did not follow the happy path in a weird way. So being able to see
451
00:32:59.975 --> 00:33:03.515
the road west followed, just to channel Robert Frost there,
452
00:33:03.815 --> 00:33:09.035
I think is very important. I think it's not necessarily something that you get from the runtime analysis.
453
00:33:09.560 --> 00:33:13.820
That being said, there's no reason why you can't do both at the same time
454
00:33:14.280 --> 00:33:17.260
and get a more complete picture in general.
455
00:33:17.675 --> 00:33:24.815
If you were to move to that runtime based approach where you're actually executing that code, I'm wondering what are some of the
456
00:33:25.320 --> 00:33:26.539
additional complexities
457
00:33:26.840 --> 00:33:30.460
that that would bring to you as a maintainer and some of the
458
00:33:30.919 --> 00:33:33.179
challenges that that might introduce
459
00:33:33.605 --> 00:33:44.260
for some of the end users where they're expecting that the code is just going to be analyzed statically and maybe they need to worry about some of the sort of reentrant characteristics of the program.
460
00:33:44.960 --> 00:33:46.580
Yeah. Sure. So, clearly,
461
00:33:47.760 --> 00:33:50.179
the addition of a runtime component
462
00:33:50.684 --> 00:33:54.385
to CodeDeploy would make it more complex both from
463
00:33:55.005 --> 00:33:57.024
a maintenance perspective of CodeDeploy
464
00:33:57.644 --> 00:33:59.664
and from a developer perspective.
465
00:33:59.970 --> 00:34:02.790
That they would have to go in and
466
00:34:03.330 --> 00:34:03.830
actually
467
00:34:04.130 --> 00:34:06.550
run their program. And depending on whether
468
00:34:07.164 --> 00:34:12.065
what you're doing is actually some sort of a production system, it might be prohibitively
469
00:34:12.765 --> 00:34:13.905
difficult as well.
470
00:34:14.205 --> 00:34:16.224
But this is the sort of thing
471
00:34:16.599 --> 00:34:18.700
that you can add on incrementally
472
00:34:20.440 --> 00:34:21.660
and basically gets
473
00:34:21.960 --> 00:34:22.460
increasingly
474
00:34:22.920 --> 00:34:23.420
better
475
00:34:23.965 --> 00:34:27.105
views of what's actually happening with your program.
476
00:34:27.485 --> 00:34:30.740
The more times you run it, the more corner cases you run it on.
477
00:34:31.220 --> 00:34:33.560
And just to clarify, this is a fully
478
00:34:34.100 --> 00:34:38.680
hypothetical thing that we're talking about here. This is not the current behavior of CodeDeploy,
479
00:34:39.345 --> 00:34:44.885
but I do think that it would be an interesting path to go down, and it's certainly a path that I
480
00:34:45.265 --> 00:34:48.085
intend to explore as I continue to develop it.
481
00:34:48.500 --> 00:34:50.680
Another interesting element of Code 2flow,
482
00:34:51.060 --> 00:34:54.040
particularly in its current state of being a static analysis,
483
00:34:54.420 --> 00:34:55.160
is that
484
00:34:55.724 --> 00:34:56.704
it gives you this
485
00:34:57.005 --> 00:35:00.865
snapshot in time of what the logic of the program looks like. And I'm curious
486
00:35:01.484 --> 00:35:03.640
what you have seen as either some existing
487
00:35:04.020 --> 00:35:09.595
applications or some potential implementations of ways that somebody might incorporate it into a
488
00:35:10.075 --> 00:35:16.095
CICD workflow to be able to view the evolution of the program structure over time as
489
00:35:16.474 --> 00:35:19.850
people make the changes and introduce new capabilities and features?
490
00:35:20.630 --> 00:35:26.730
Sure. So I've not seen any specific examples of this actually being done, but I do believe
491
00:35:27.494 --> 00:35:29.434
that by being able to
492
00:35:29.815 --> 00:35:30.315
view
493
00:35:30.615 --> 00:35:31.515
the evolution
494
00:35:31.815 --> 00:35:35.355
of a program over time, you get a couple of benefits.
495
00:35:35.760 --> 00:35:38.900
The first is just kinda cool that you can see
496
00:35:39.440 --> 00:35:42.819
how what you're doing has gotten bigger and more complex.
497
00:35:43.355 --> 00:35:47.215
The second is you can see how things used to work and how things change.
498
00:35:47.675 --> 00:35:50.335
But I think that's the core of the question
499
00:35:50.635 --> 00:35:52.830
from the perspective of a
500
00:35:53.390 --> 00:35:53.890
CICD
501
00:35:54.430 --> 00:35:56.370
is how that could be useful.
502
00:35:56.910 --> 00:35:58.930
And I think that it is
503
00:35:59.710 --> 00:36:01.970
this potential ability to
504
00:36:02.475 --> 00:36:03.535
see the delta
505
00:36:04.075 --> 00:36:04.575
between
506
00:36:05.275 --> 00:36:12.110
what was there before the commit that you're doing and what the commit has changed in terms of the structure.
507
00:36:12.810 --> 00:36:17.710
And there's a couple ways that this can be used. 1 that I can think of is as
508
00:36:18.494 --> 00:36:20.595
essentially like an early warning alert
509
00:36:21.295 --> 00:36:22.994
that something big is
510
00:36:23.295 --> 00:36:23.795
happening
511
00:36:24.335 --> 00:36:28.950
in this pull request. It's changing the structure of the program in these
512
00:36:29.890 --> 00:36:31.910
very important, very key ways,
513
00:36:32.290 --> 00:36:41.315
and maybe you should take a closer look at this sort of thing. I assume that there's more use cases beyond that, but I've not thought about that in time.
514
00:36:41.615 --> 00:36:52.735
And so in your experience of building code to flow and then reimplementing it and rearchitecting it, I'm curious what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?
515
00:37:00.119 --> 00:37:02.540
To actually put out the PNG
516
00:37:03.080 --> 00:37:05.099
JPEG outputs of CodeDeploy.
517
00:37:05.880 --> 00:37:09.339
And 1 interesting thing that I've come across with graphviz
518
00:37:09.714 --> 00:37:10.454
is that
519
00:37:11.075 --> 00:37:17.095
depending on the size of the output that I'm giving it, it can run either in milliseconds
520
00:37:17.474 --> 00:37:19.540
or it can run until the end of
521
00:37:20.340 --> 00:37:24.040
time. So what I've had to do is depending on the
522
00:37:24.500 --> 00:37:25.000
specific
523
00:37:25.380 --> 00:37:25.880
complexity
524
00:37:26.340 --> 00:37:29.835
of what I'm getting it, pass it different options
525
00:37:30.375 --> 00:37:32.075
in order to have it
526
00:37:32.375 --> 00:37:37.435
run a little bit faster at the cost of a slightly prettier graph.
527
00:37:37.950 --> 00:37:40.050
And that's taken a lot of experimentation
528
00:37:40.430 --> 00:37:41.970
to get that balance correct.
529
00:37:42.510 --> 00:37:45.330
But I think that that's 1 thing that I was
530
00:37:45.825 --> 00:37:46.404
not anticipating
531
00:37:47.184 --> 00:37:47.684
that
532
00:37:48.065 --> 00:37:50.085
ended up becoming the case. Because
533
00:37:50.464 --> 00:37:53.845
with these sorts of things, you expect it to be all the hard problems
534
00:37:54.410 --> 00:37:58.270
that give you pause and, like, make the development process really difficult.
535
00:37:58.730 --> 00:38:02.990
But, ultimately, it ended up being this third party dependency that I have
536
00:38:03.485 --> 00:38:21.750
that's as I'm just passing it too much stuff. I've needed to play a little bit more nicely with it. Yeah. It's always interesting how you expect some big complexity and, you know, difficult challenge, and that's actually the easy part. And it's really just these little niggling details
that end up taking up all of your time.
537
00:38:22.775 --> 00:38:24.474
Absolutely. No niggling details.
538
00:38:25.815 --> 00:38:29.194
In terms of the sort of utility of CodeDeploy
539
00:38:29.575 --> 00:38:30.395
as this
540
00:38:31.070 --> 00:38:34.690
method of exploration and understanding of a program structure,
541
00:38:35.070 --> 00:38:40.455
what are the cases where CodeDeploy is the wrong choice and you might be better suited with something like profiling
542
00:38:40.995 --> 00:38:47.415
or some other visualization element or something like source graph that has more of this sort of functional indexing capabilities?
543
00:38:48.390 --> 00:38:51.610
So I don't think that Code 2flow is ever
544
00:38:52.070 --> 00:38:57.130
the wrong choice. I think that it's 1 tool in your toolbox,
545
00:38:57.535 --> 00:39:00.515
and it is the first step in what you're doing.
546
00:39:00.895 --> 00:39:03.555
Maybe not even the first step. Maybe in the top
547
00:39:03.855 --> 00:39:05.369
10 first steps.
548
00:39:05.829 --> 00:39:13.049
But it does what it does, which gives you the high level overview of what you're doing. And I think that its limitations
549
00:39:13.349 --> 00:39:14.329
as far as
550
00:39:14.855 --> 00:39:15.355
understanding
551
00:39:15.975 --> 00:39:19.915
what's actually happening in the program is that it doesn't actually capture
552
00:39:20.535 --> 00:39:21.435
business logic
553
00:39:21.895 --> 00:39:23.275
or to your point,
554
00:39:23.700 --> 00:39:24.200
runtime.
555
00:39:24.740 --> 00:39:28.200
But for business logic, this is often
556
00:39:28.820 --> 00:39:31.640
a very important part of what a program
557
00:39:32.275 --> 00:39:33.015
actually does.
558
00:39:33.555 --> 00:39:40.214
And I think that in order to dig into something like that, there's not really a way around actually
559
00:39:41.030 --> 00:39:45.530
looking at the source code, spending a lot of time with it, and understanding it.
560
00:39:45.990 --> 00:39:47.450
And I think that CodeDeploy
561
00:39:47.910 --> 00:39:55.175
is not going to be able to do that. And I can't think off the top of my head of any tool that makes that particular step any easier.
562
00:39:55.635 --> 00:40:01.050
In terms of the near to medium term future of the project, I'm curious about some of the
563
00:40:01.510 --> 00:40:09.865
next steps that you have in mind or any project that you're particularly excited to dig into to help improve it or add new functionality or
564
00:40:10.245 --> 00:40:14.745
or any ways that you are looking for help and feedback on the project?
565
00:40:15.530 --> 00:40:17.390
Talk about the future first.
566
00:40:17.930 --> 00:40:21.310
So what I would like to do with it in the
567
00:40:21.770 --> 00:40:23.150
immediate term is
568
00:40:23.450 --> 00:40:24.714
make CodeDeploy interactive.
569
00:40:25.494 --> 00:40:27.515
So we've talked a lot about the
570
00:40:27.895 --> 00:40:29.435
static nature of the outputs,
571
00:40:29.815 --> 00:40:31.275
the limitations of that,
572
00:40:31.734 --> 00:40:33.194
and also the
573
00:40:33.650 --> 00:40:34.710
overall complexity
574
00:40:35.170 --> 00:40:36.630
of potentially having
575
00:40:37.650 --> 00:40:38.309
a large
576
00:40:38.770 --> 00:40:45.105
graph. And while there are ways to limit this large graph and to make it a little more manageable,
577
00:40:45.724 --> 00:40:47.265
I don't think that it's necessarily
578
00:40:47.645 --> 00:40:51.184
the right approach. So what I would like to do in the near term
579
00:40:51.720 --> 00:40:57.819
is build a GUI for CodeDeploy so that instead of the static output, you have this
580
00:40:58.200 --> 00:40:59.420
interactive tool
581
00:40:59.744 --> 00:41:02.405
where you can click and scroll through
582
00:41:02.865 --> 00:41:05.605
your program and zoom in and out and
583
00:41:06.145 --> 00:41:08.085
just have it as more of an exploratory conversation rather than this generated artifact. As far as
584
00:41:09.810 --> 00:41:10.790
rather than this
585
00:41:11.170 --> 00:41:12.230
generated artifact.
586
00:41:12.610 --> 00:41:13.670
As far as
587
00:41:14.050 --> 00:41:14.550
where
588
00:41:15.010 --> 00:41:24.105
help might be needed, I certainly think that languages are the obvious answers. I think that I've cut across a large part of the
589
00:41:24.405 --> 00:41:29.099
programming world with the 4 languages it does support, but certainly
590
00:41:29.720 --> 00:41:33.825
not all of it. And I'm not even sure whether that would be half of it.
591
00:41:34.305 --> 00:41:34.805
So
592
00:41:35.345 --> 00:41:36.724
adding more languages,
593
00:41:37.025 --> 00:41:41.525
I talked a little bit about Rust that's becoming popular in many circles, but
594
00:41:41.825 --> 00:41:44.390
there's no reason that CodeDeploy
595
00:41:44.770 --> 00:41:46.470
can't also manage
596
00:41:46.850 --> 00:41:47.750
static languages.
597
00:41:48.370 --> 00:41:50.630
Rust, c, c plus plus Java,
598
00:41:51.090 --> 00:41:52.950
things that people use every day.
599
00:41:53.325 --> 00:41:54.065
I'm curious
600
00:41:54.365 --> 00:41:56.465
what you see as some of the
601
00:41:57.005 --> 00:42:04.670
steps or modifications that would be needed to be able to add support for some of these static languages and just some of the ways that you have
602
00:42:05.130 --> 00:42:06.350
architected the project
603
00:42:06.730 --> 00:42:14.615
to allow for this extension into other language runtimes rather than being tied to the existing ones that you've already targeted?
604
00:42:15.315 --> 00:42:18.695
I don't think that it would be especially difficult
605
00:42:19.395 --> 00:42:21.799
to add a static language to CodeDeploy.
606
00:42:22.339 --> 00:42:31.145
I think that going the other direction would be far far more difficult to have a tool that was designed for static languages and move to dynamic languages
607
00:42:31.765 --> 00:42:36.505
because what you're looking at when
you're looking at the structure of a static language
608
00:42:37.000 --> 00:42:40.380
is a dynamic language, but with a lot less gotchas,
609
00:42:41.000 --> 00:42:41.820
in my opinion.
610
00:42:42.360 --> 00:42:44.220
So I do believe that
611
00:42:44.520 --> 00:42:45.500
the same
612
00:42:45.965 --> 00:42:50.545
method that I've used for the 4 dynamic languages could be applied to static languages.
613
00:42:51.165 --> 00:42:51.665
Absolutely.
614
00:42:52.525 --> 00:43:01.160
Alright. Well, for anybody who wants to follow along with you and get in touch, I'll have you add your preferred contact information to the show notes. And so with that, I'll take us into the picks.
615
00:43:01.460 --> 00:43:06.765
This week, I'm going to pick taking a vacation because it's the first time in
616
00:43:07.065 --> 00:43:15.809
many years that I have actually taken a real vacation, and that's where I'm recording this episode from. So definitely recommend that for folks who have not taken the opportunity.
617
00:43:16.349 --> 00:43:17.809
Personally, I am currently
618
00:43:18.195 --> 00:43:37.190
taking the week at Universal Studios Florida, which has been quite a lot of fun. So if you have the time and the means to be able to do so, I recommend checking that out. And so with that, I'll pass it to you, Scott. Do you have any picks this week? Sure. I think that my pick and I thought about this for a little while after listening to a couple of your earlier versions.
619
00:43:37.555 --> 00:43:39.575
But what I'd like to add is that
620
00:43:40.115 --> 00:43:45.095
service work has been valuable in a way that I didn't initially anticipate.
621
00:43:45.710 --> 00:43:56.965
I'm really happy to be here, and I think that I'm growing a lot as a person by working with people. And I am excited to see how I'm going to be able to apply this back into the
622
00:43:57.345 --> 00:44:04.190
software world. And I'm not sure how that's going to work yet, whether it's gonna be in terms of a different way of thinking
623
00:44:04.809 --> 00:44:08.509
or a different way of communicating with people because, fundamentally,
624
00:44:09.105 --> 00:44:10.965
software is a collaborative
625
00:44:11.505 --> 00:44:12.005
business.
626
00:44:12.545 --> 00:44:16.885
We do things like this all the time. So I think that it is undervalued
627
00:44:17.830 --> 00:44:20.250
as a thing to do to go on service projects.
628
00:44:20.870 --> 00:44:26.810
Absolutely. Well, thank you very much for taking the time today to join me and share the work that you're doing on CodeDeploy
629
00:44:27.494 --> 00:44:29.275
and for the time that you're spending
630
00:44:29.575 --> 00:44:48.555
doing that service work where you are right now. So definitely excited to be able to speak to you and understand more about the project that you've built and some of the ways that it can be applied to some of the projects that I've been working on and other folks have been working on. So definitely glad that you have taken the time to share that with the world, and I hope you enjoy the rest of your day.
631
00:44:48.855 --> 00:44:50.235
Thank you. You as well.
632
00:44:52.695 --> 00:44:58.760
Thank you for listening. Don't forget to check out our other show, the data engineering podcast at data engineering podcast.com
633
00:44:59.300 --> 00:45:01.160
for the latest on modern data management.
634
00:45:01.955 --> 00:45:03.335
And visit the site of pythonpodcast.com
635
00:45:04.435 --> 00:45:07.895
to subscribe to the show, sign up for the mailing list, and read the show notes.
636
00:45:08.275 --> 00:45:13.309
And if you've learned something or tried out a project from the show, then tell us about it. Email host at podcastinit.com
637
00:45:14.089 --> 00:45:14.910
with your story.
638
00:45:15.609 --> 00:45:19.385
To help other people find the show, please leave a review on Itunes and tell your friends
639
00:45:22.645 --> 00:45:23.705
and coworkers.