Kavya Bhat 0:00
So the initial plan was to set up an IPv6 only test bed, and
that's what we worked on for the first couple of months. When we
got on the project, we set this up in our Department of Computer
Science. So one thing we had to ensure was that this IPv6 traffic
wouldn't be affecting any other devices on the same network. And
one reason why we had to ensure that was because we planned to
look at a DHCPv6 server, a DNS server, running on our test bed,
and if we allowed that traffic to go to the other devices, it
would probably cause a lot of chaos in the network.
George Michaelson 0:48
you're listening to PING, a podcast by APNIC discussing all things
related to measuring the Internet. I'm your host, George
Michaelson, this time I'm talking to Vanessa Fernandez and Kavya
Bhat from the National Institute of Technology Karnataka, which is
in Mangalore in India. NITK has almost 50,000 end points in a
campus wide network, which ultimately will have to encompass a 40
kilometer campus expansion. They've got over 350 switches, 1200
WiFi access points and a campus data center with servers. They've
also been running a multi year student led project to deploy
campus wide IPv6. Two faculty members are overseeing the project,
but they also have alumni from past years and an advisory group,
which include Nalini Elkins and Dhruv Dhody from IETF and ISOC
respectively, Vanessa and Kavya have now graduated and entered the
workplace. They'll become alumni in due course for a new crop of
NITK students, who will in turn continue the deployment onward.
This episode of PING is a reminder that large deployments often
happen largely with volunteer and sometimes previously untrained
hands. Vanessa and Kavya presented at two IETF meetings on
deployment experience and measurements of their network with
assistance from the APNIC foundation under a n ISIF project. Hello
and welcome to ping. Vanessa Kavya, could you introduce yourselves
please,
Vanessa Fernandez 2:28
Sure. I'm Vanessa Fernandez. I'm a recent undergrad from the
National Institute of Technology Karnataka, and I was part of the
IPv6 migration team there.
George Michaelson 2:39
And Kavya,
Kavya Bhat 2:40
Hi, I'm Kavya Bhat. I'm recent undergrad from the National
Institute of Technology Karnataka, and along with Vanessa, I was
part of the IPv6 deployment team that worked on migrating the
campus network IPv6
George Michaelson 2:54
So you were both students in 2023 in a university in Mangalore,
weren't you at the NITK Vanessa, could you describe the university
a bit? How large is it?
Vanessa Fernandez 3:07
We're a fairly large university campus in terms of the network. We
have more than 45,000 terminals connected to the Internet. And we
also can keep in mind that we have an upcoming campus expansion,
probably within 40 kilometers of the existing campus. In terms of
our infrastructure, we have more than 350 switches, 1200 indoor,
outdoor WiFi access points. And we also have a dedicated data
center within our campus. It hosts all of our critical infra, like
the firewalls, the core switches, any of our web servers,
application servers, servers for DHCPv6, DNS and so on. And in
terms of our population, we have more than 7000 students within
our campus, residing in our campus, and more than 600 staff,
teaching and non teaching staff. Yeah, that's about our campus.
George Michaelson 4:04
So Kavya, is this like a single building campus, or is it a multi
building location?
Kavya Bhat 4:10
It's multiple buildings. So we have across like about 200 to 300
acres. We have like, different buildings, such as the
administrative building, different departmental buildings for
computer science, IT electronics and such. And then we also have
the hostel buildings. So it's multiple buildings spread across a
pretty large area.
George Michaelson 4:29
And this work that you were doing, I came to learn about this as
part of IETF118, there's been this program where we've been
looking for people to bring practical deployment experience in
IPv6 into the standards process so that we can understand things
better. This was the project that you were working on, right? You
were actually doing a V6 deployment at your university,
Vanessa Fernandez 4:54
yeah, this was a project that we were working on, so we presented
pretty much what we had done so far at IETF118, in terms of
deployment, and also in terms of this testbed that we had set up
and the applications that we were able to migrate at that point.
George Michaelson 5:10
And Kavya, you were working with a professor on campus, Professor
Tahiliani. I mangled that. How do i pronounce his name?
Kavya Bhat 5:19
Yeah, his name is Professor Mohit Tahiliani, and he is sort of the
sort of our guide in terms of a lot of projects that we do at
NITK. And it was through him that I got to know about a lot of
opportunities such as this one. And when he asked about whether
I'd be interested in joining the project team, I definitely said,
Yes,
George Michaelson 5:42
how big was the team?
Kavya Bhat 5:43
It started off as a small team. So we actually had Professor Mohit
and another professor, Saumya Hegde, and two alumni, students,
Sushanth Rao and Akshay Revankar, working on the IPv6 deployment
thing. And after this, when our alumni graduated, we needed
students specifically working hands on in the campus, and that's
where Vanessa and I joined in. We had worked on previous projects
with Professor Mohit, and through that, we kind of knew of this
opportunity.
George Michaelson 6:16
And Vanessa This was also working with Nalini Elkins, I believe,
yes.
Vanessa Fernandez 6:21
So Nalini and Mike were part of our advisory team, along with
Dhruv from ISOC, yeah. So this whole team of professor, Mohit,
our alumni, Nalini, Mike, Dhruv, they were really instrumental in
mentoring us throughout follow the format once every two weeks or
once every week to kind of catch up and see what work had been
done, work could be done. What were the problems we were facing,
how we could tackle these problems? And, yeah, they just gave us
mentorship throughout, and, you know, helped us work on the
project present at the IETF, things like that that, you know, we
really didn't have much experience in
George Michaelson 6:57
so you said that this is about 45,000 terminal devices that to be
connected, and quite a large number of switches, 350 switches, and
1200 WiFi points. So I'm guessing you already had quite a large
body of work here, finding the devices, getting consoles, getting
images worked out. There was a lot of pre planning involved in
this. Wasn't there.
Kavya Bhat 7:23
This was sort of formed as a project proposal to get a grant to
work on it. There was a very detailed plan of action that was laid
out, and that involved figuring out our address and plan, figuring
out how we would migrate individual VLANs and then ensure that
connectivity was up, while also deploying or enabling IPv6 routing
on particular VLANs.
George Michaelson 7:49
So what size IPv six block were you working under? What was your
amount of V6 address that you had to work with?
Kavya Bhat 7:56
We directly got a slash 36 block that was leased from IRINN, and
we asked BSNL to advertise that particular IPv6 block that we
leased.
George Michaelson 8:06
So BSNL is BSNL is the university's upstream?
Kavya Bhat 8:10
Yes, the BSNL router acts as our upstream. Yeah.
George Michaelson 8:14
And your plan for an address map over this you were doing subnets
for each switch. Did you have a structure diagram in mind for how
you were going to allocate the address plan across this? What was
your population that you were figuring on putting per switch?
Maybe that's not how you did it. Actually, I'd be quite interested
in knowing how you decided to subnet this, because it's not
actually immediately clear that you need subnet structure here. So
did you discuss running a layer two flat network across all of
these switches, or was it a given that you were going to have a
hierarchy?
Kavya Bhat 8:51
So I think we decided to sort of stay a bit with our IPv4
addressing scheme, where we had different VLANs for different
administrative buildings. So we kind of went with that and gave
particular subnets to particular VLANs.
George Michaelson 9:08
Did you do a population count so you would give more addresses to
a network if you knew that it had a higher density of students? Or
were you able to give everyone pretty much exactly the same size
subnet structure.
Vanessa Fernandez 9:21
I don't think we did any such like pre calculation for assigning
addresses. Maybe we should have, in hindsight,
George Michaelson 9:28
no, I don't. It's not a given at all. The thing is that in V6 it's
so different to v4 right? In IPv4, you would be counting every
individual host to make sure you didn't have too many but in V6
Well, it is relative abundance, right?
Vanessa Fernandez 9:45
Yeah, that's true, and that's probably why we kind of stuck to the
v4 addressing scheme. And pretty much whatever we did with v4 we
thought would be a fine template for us to use for V6 as well. So
George Michaelson 9:57
So you obviously didn't start, I know. That's tomorrow. Turn on V6
on the entire campus. Boom, done. Let's go and have a cup of
coffee. What did you actually start with? You started with a test
bed of some kind.
Kavya Bhat 10:09
Yeah. So the initial plan was to set up an IPv6 only test bed, and
that's what we worked on for the first couple of months. When we
got on the project, we set this up in our Department of Computer
Science. One thing we had to ensure was that this IPv6 traffic
wouldn't be affecting any other devices on the same network. And
one reason why we had to ensure that was because we planned to
look at a DHCPv6 server, a DNS server, running on our test bed,
and if we allowed that traffic to go to the other devices, it
would probably cause a lot of chaos in the network.
George Michaelson 10:47
You would have had random devices picking up V6 addresses
everywhere. That wouldn't have been very good. So you ran a small,
isolated network, and you gained experience with running the
fundamental building blocks of a network, a DHCPv6 server and a
DNS server, and you didn't do Stateless auto config, you decided
not to do SLAAC.
Vanessa Fernandez 11:14
So we decided not to do slack in our production environment. For
that reason, we based most of our experiments on DHCPv6 but we did
try out SLAAC on our test bed as well. Like we did try out
configurations, router advertisements and things like that, just
to see how it would work in the grand scheme of things. But for a
production environment, the idea was pretty clear to only use
DHCPv6 which is why we based most of our experiments around that.
George Michaelson 11:41
So you were looking at a fully managed network where address
assignment was explicit to device. You weren't just going to allow
random hookups here. People have to provision. Do you already have
a campus network management system that you were looking at
targeting
Kavya Bhat 11:57
on the staff side of things? They do have a network management
service, openNMS, that they use to monitor devices on the network.
One thing we haven't explored yet is whether openNMS specifically
supports IPv6. We didn't look at the network management systems
yet, but with that in mind, we chose to not go with SLAAC and
prefer DHCPv6 addressing
George Michaelson 12:22
so what was that experience like in general? Did everything work
smoothly? Was it just instant, okay? Or did you find any little
corner cases you had to look at?
Vanessa Fernandez 12:31
This was the first experience for like, both of us, pretty much
working hands on with IPv six and setting the test bit up and
getting it to work end to end was not really as straight forward
as we thought it would be. So there were quite a few issues. Some
that I can think of on top of my head were like setting up the
VLANs, sometimes the DNS server would not work as expected and
things like that. So there were many issues that we face along the
way that we would spend quite some time debugging. So yeah,
George Michaelson 13:00
your presentation at IETF, 118, I think it noted that you had some
unexpected outcomes with things like router signaling in DHCPv6
the systems just didn't behave quite the way you expected by
default.
Kavya Bhat 13:15
Yeah, I think the issues we highlighted there were with respect to
the DSPv6 delay, one of these things was that we see the delay for
a message going out from the client reaching the end server, but
we never saw the deployed reply reaching back to the client. We
were initially very confused about what was going on, because all
the messages seem to be going find from the client, reaching the
opensense router that we had set up, and reaching the DHCPv6
server. And then we kind of concluded, after a bunch of packet
traces and on the different hosts, that it was probably an issue
with the L2 managed switch that we were using, and that it was
somehow dropping the delayed reply messages when we had a DHCP
filter turned on. So we had a lot of issues, like this is where we
spent hours, sometimes days, debugging it, discussing it with our
professors, with our mentors, and it seems to be a hardware issue,
but this is again something that we plan to test with another
switch and conclude properly on our results.
George Michaelson 14:27
There are some concerns that people have talked about in scaling
aspects of the V6 protocol to suit large deployments like Neighbor
Discovery and the risks of flooding. Did you do anything that was
kind of in the defensive space, constructing limits and checks on
the way people were using this service? How did you actually
monitor how things were being deployed?
Kavya Bhat 14:52
Again, with respect to Neighbor Discovery, we mostly worked on the
test bed. We haven't explored too much in the production network.
But there's this one feature called Neighbor Discovery snooping,
where the switch will filter Neighbor Discovery packets based on
IPv6 address to MAC address table that it has constructed and on
our test bed. One thing we noticed was that if the systems are
connected and then we enable this feature, we never see those
entries being generated and put in the table, and we were always
confused as to why, because while it should filter those messages,
and that's what you would expect to see. And after again, a couple
of hours of debugging, we found out that if the system newly
joined the network, say it was unplugged or turned off, and then
it newly joined and started broadcasting Neighbor Discovery
messages, we would see those entries being generated
automatically. And we feel this could be a useful feature to have
on our production network, but it would require some sort of
static IPv6 address to MAC address binding, because, again, if
some new host automatically comes on the network, they could
advertise any particular address they wanted. So there are
definitely some security aspects that we still need to explore,
and that's part of our next steps for the project.
Vanessa Fernandez 16:16
So Vanessa, I think you were also looking at the DHCP V6 server
using real database, persistent store, rather than just memory,
back state. Were you actually doing code work on that?
Not much code work. So the aim was to just look into the
production environment best practices for DHCPv6, like load
balancing, whether we would need a redundant server or not, what
type of redundancy you want to introduce and things like that, and
then also migrating our back end from the basic mem file storage
to a more persistent database, like a SQL database in the back
end. So I hadn't looked into scripting or something like that, but
just the migration of the database from mem file to a persistent
database.
George Michaelson 17:02
So the capability was built into the tool. You just had to select
the right options for it to go into a deployment in the field,
rather than just as an experiment,
Vanessa Fernandez 17:11
kind of Yeah. So we would have to set up a SQL database in the
back end, and then give the DHCP V6 server access to that
database, and then it would work fine. Yeah.
George Michaelson 17:22
So what state of deployment Have you actually reached at the end
of this activity today? Do you now have a fully enabled campus, or
are you still in deployment phases? VLAN? By VLAN,
Vanessa Fernandez 17:33
we are currently in deployment phases. So we've managed to migrate
our data center VLAN to IPv6, which is great. And we've also set
up a DHCPv6 server at the data center, VLAN, that serves our TRIPC
VLAN, that's our central Computing Center VLAN. So that's done,
but we now are looking into setting up an VLAN, which is a trusted
VLAN, and we would add trusted users to this VLAN enable IPv6
routing on it and see if there are any potential issues that need
to be addressed. If everything goes fine, we hope to migrate VLANs
one by one, maybe start at the hostel buildings, the academic
buildings and so on, and just totally expand our migration.
George Michaelson 18:17
What did you present at IETF? What did you find most interesting
about this activity that you presented to IETF?
Kavya Bhat 18:25
I think the first IETF we attended, IETF118 was at the point where
we had almost completed setting up our test bed. And that was one
of our first activities with respect to IPv six hands on work. And
there were a lot of challenges we faced in terms of initial
understanding of the particular protocols or how they were
expected to work. Even when deploying, we would see issues where
we weren't able to ping the external Internet. In some cases, we'd
have DNS resolution issues. And these findings were something we
were particularly excited to share, because as students, we were
looking forward to interacting with the whole IETF community and
getting input on how things worked, or whether our observations
were as expected or sort of out of the blue.
George Michaelson 19:17
And for the second presentation that you did at IETF119,
Vanessa Fernandez 19:22
yeah. So for the second presentation, we had a couple more updates
in terms of issues that we had faced on a test bed, like as Kavya
mentioned at the first IETF, we had just finished setting up our
test bed. So there we received a lot of feedback on things that we
could try out. But at our second IETF, we had already tried out a
bunch of things. For example, the DHCPv6 filtering, which was a
problem that we had faced, the Neighbor Discovery issues, even DNS
scope, sometimes the end client did not receive information from
the DNS server, and also some route advertisement configuration
issues that we had faced. So there were a couple issues like this
that we wanted to bring up, and as Kavya mentioned, just see if
this was something that only we were facing, or if other
enterprises were facing similar problems during deployment as
well.
George Michaelson 20:10
So you wouldn't have gone into this activity completely cold. You
did V6 training as part of your entry into this I think you've
told me previously that you did some APNIC Academy work, is that
right?
Vanessa Fernandez 20:24
Yes. So before we joined the project, our professor suggested to
us that we do the APNIC Academy IPv6 tutorials. So that was my
first introduction into IPv6. And along with that, we also did the
labs, which we found really, really helpful for the testbed
deployment, because there was some useful labs, especially the
APNIC lab on Kia, DHCPv6 server. That was one that I can think of
the top of my head, that we went back to again and again, just to
make sure we were doing things right.
George Michaelson 20:56
So it's a virtual network setup. You're actually doing
configuration on Routing and Switching infrastructure in a virtual
network.
Vanessa Fernandez 21:03
Yes, those are all VM setups, and it's playground, pretty much for
us to configure our own servers and end clients.
George Michaelson 21:10
Happy eyeballs. Everybody wants to believe that the minute you
deploy IPv six, that's it. Boom. Game over. People stop using v4
but it isn't really that simple, is it? So, Kavya, did you do any
work measuring happy eyeballs and the relative amounts of usage?
Kavya Bhat 21:27
We didn't specifically work on Happy eyeballs itself, but we did
observe some statistics from IRIS, which is our Student Portal.
It's our learning management system, and the statistics obtained
from that showed that in some cases, with certain OS, such as
Windows, we were seeing that there were more IPv4 hits from users
rather than IPv6. So when we discussed these results with Nalini
Elkins at that time, I think happy eyeballs had been discussed a
bit, and we thought that maybe happy eyeballs could be a factor
for that, but we didn't specifically go into measuring or testing
if that was the definite reason.
Vanessa Fernandez 22:10
Just want to add that we did bring up happy eyeballs at IETF118, I
mean, we thought it was a potential issue,d mentioned, but we
realized later on that at that point, most of our on campus
population, like the staff and students, were using Windows
laptops or Windows devices, and that's probably why we saw more
IPv4 hits versus IPv6 hits, like there was no IPv6 routing enabled
on Campus at that point, right? And all of our staff and students
would be accessing IRIS through v4,
George Michaelson 22:45
Whereas the VPN access and the remote access would have included
more people on mobile devices, and because mobile handsets a dual
stack, and because you already had from the alumni, a dual stack
deployment in the VPN, you would have seen higher numbers there,
right?
Kavya Bhat 23:03
Our statistics kind of consolidated that theory also, because we
did see that IPv6 hits from Android users were higher compared to
the IPv4 hits. And if you consider that mobile users would be
using their cellular data, and a lot of people use ISP such as Jio
or airtel, which do have IPv6 support, and that kind of
consolidates our theories. But again, we could also look into
whether happy eyeballs does play a role.
George Michaelson 23:33
You mentioned this campus course management system IRIS is it? Did
you actually have to do liaison work to help people do a migration
to make that dual stack or was there very little effort required
to turn that into a dual stack service? How did that go?
Vanessa Fernandez 23:49
Actually, IRIS is a student developed information management
system so entirely developed by our alumni and maintained by the
current students at NITK. So yeah, it was a student led effort,
even migrating the IRIS applications and servers to IPv6 entirely
done by students. So there was a team, dedicated team, working on
migrating the applications,
George Michaelson 24:13
and they got access to your testbed network. They were able to
validate that application and make sure it was working. So the
future deployments are going to go ahead because they know it
works okay in the dual stack environment, [Vanessa yep] Do you see
yourselves carrying on in the network space? Has this experience
been sufficiently positive? It kind of makes you think, oh, yeah,
I could be in networking. It's fun.
Vanessa Fernandez 24:36
Yes, at least for me. I think after having this experience working
on IPv6 migration. I'm lucky that I got such a hands on experience
at college as an undergrad student, and I am even more excited now
to continue working in this space, continue being part of the
migration effort at NITK, and yeah, just being involved in any
capacity that I can.
George Michaelson 24:58
And Kavya, you're at CMU next year or this year?
Kavya Bhat 25:02
Um, yeah, I'll be joining the Masters in Information networking at
CMU this fall. So definitely, my undergrad experience in the on
networking projects definitely made my whole undergrad life
positive experience, I'd say, and consolidated my decision to go
for higher studies in the same field, and I'm hoping that I'll get
to continue working on networking projects, and I'll definitely be
helping the NITK community with the projects they take up as well.
George Michaelson 25:33
So you're both going to become an alumni and work on this activity
the same way the alumni you worked with did carry the story
forward, [Kavya yeah] it's really great. You got some assistance
from APNIC foundation towards attendance at the IETF meetings.
Vanessa Fernandez 25:49
Yep. So for IETF118, we were able to attend because of the IPv6
deployment grant that we got from APNIC foundation in ISIF Asia.
So we were able to attend through that. And for IETF119, APNIC
foundation selected Kavya and I as one of the Internet standards
ambassadors to attend in Brisbane. So yeah, both the IETFs, we
were able to attend through the APNIC foundation. Really grateful
for that.
George Michaelson 26:16
So is a write up available of what went on here? Is there anything
we can point people at to look at your experiences doing this
deployment.
Vanessa Fernandez 26:24
I don't think anything has been published yet, like we have
documents ready, but I don't think they've been published.
George Michaelson 26:30
But I could point people at the IETF 118 presentation for sure.
But the side meeting at 119 was not recorded.
Vanessa Fernandez 26:40
No, the side meeting was not recorded, but we do have a recording
for it.
George Michaelson 26:45
That's really super. Thank you very much for coming on the
program. Thank you Vanessa, thank you Kavya,
Vanessa Fernandez 26:50
thank you George. It's been wonderful.
Kavya Bhat 26:52
Thank you for this opportunity. And yeah, we had a wonderful time.
George Michaelson 26:57
If you've got a story or research to share here on ping. Why not
get in contact by email to ping@apnic.net or via the APNIC social
media channels. Also remember that measurement@apnic.net mailing
list on orbit is there to discuss and share relevant collaborative
opportunities, grants and funding opportunities, jobs and graduate
placings, or to seek feedback from the community on your own
measurement projects, be sure to check out the APNIC website for
All your resource and community needs until next time.
So the initial plan was to set up an IPv6 only test bed, and
that's what we worked on for the first couple of months. When we
got on the project, we set this up in our Department of Computer
Science. So one thing we had to ensure was that this IPv6 traffic
wouldn't be affecting any other devices on the same network. And
one reason why we had to ensure that was because we planned to
look at a DHCPv6 server, a DNS server, running on our test bed,
and if we allowed that traffic to go to the other devices, it
would probably cause a lot of chaos in the network.
George Michaelson 0:48
you're listening to PING, a podcast by APNIC discussing all things
related to measuring the Internet. I'm your host, George
Michaelson, this time I'm talking to Vanessa Fernandez and Kavya
Bhat from the National Institute of Technology Karnataka, which is
in Mangalore in India. NITK has almost 50,000 end points in a
campus wide network, which ultimately will have to encompass a 40
kilometer campus expansion. They've got over 350 switches, 1200
WiFi access points and a campus data center with servers. They've
also been running a multi year student led project to deploy
campus wide IPv6. Two faculty members are overseeing the project,
but they also have alumni from past years and an advisory group,
which include Nalini Elkins and Dhruv Dhody from IETF and ISOC
respectively, Vanessa and Kavya have now graduated and entered the
workplace. They'll become alumni in due course for a new crop of
NITK students, who will in turn continue the deployment onward.
This episode of PING is a reminder that large deployments often
happen largely with volunteer and sometimes previously untrained
hands. Vanessa and Kavya presented at two IETF meetings on
deployment experience and measurements of their network with
assistance from the APNIC foundation under a n ISIF project. Hello
and welcome to ping. Vanessa Kavya, could you introduce yourselves
please,
Vanessa Fernandez 2:28
Sure. I'm Vanessa Fernandez. I'm a recent undergrad from the
National Institute of Technology Karnataka, and I was part of the
IPv6 migration team there.
George Michaelson 2:39
And Kavya,
Kavya Bhat 2:40
Hi, I'm Kavya Bhat. I'm recent undergrad from the National
Institute of Technology Karnataka, and along with Vanessa, I was
part of the IPv6 deployment team that worked on migrating the
campus network IPv6
George Michaelson 2:54
So you were both students in 2023 in a university in Mangalore,
weren't you at the NITK Vanessa, could you describe the university
a bit? How large is it?
Vanessa Fernandez 3:07
We're a fairly large university campus in terms of the network. We
have more than 45,000 terminals connected to the Internet. And we
also can keep in mind that we have an upcoming campus expansion,
probably within 40 kilometers of the existing campus. In terms of
our infrastructure, we have more than 350 switches, 1200 indoor,
outdoor WiFi access points. And we also have a dedicated data
center within our campus. It hosts all of our critical infra, like
the firewalls, the core switches, any of our web servers,
application servers, servers for DHCPv6, DNS and so on. And in
terms of our population, we have more than 7000 students within
our campus, residing in our campus, and more than 600 staff,
teaching and non teaching staff. Yeah, that's about our campus.
George Michaelson 4:04
So Kavya, is this like a single building campus, or is it a multi
building location?
Kavya Bhat 4:10
It's multiple buildings. So we have across like about 200 to 300
acres. We have like, different buildings, such as the
administrative building, different departmental buildings for
computer science, IT electronics and such. And then we also have
the hostel buildings. So it's multiple buildings spread across a
pretty large area.
George Michaelson 4:29
And this work that you were doing, I came to learn about this as
part of IETF118, there's been this program where we've been
looking for people to bring practical deployment experience in
IPv6 into the standards process so that we can understand things
better. This was the project that you were working on, right? You
were actually doing a V6 deployment at your university,
Vanessa Fernandez 4:54
yeah, this was a project that we were working on, so we presented
pretty much what we had done so far at IETF118, in terms of
deployment, and also in terms of this testbed that we had set up
and the applications that we were able to migrate at that point.
George Michaelson 5:10
And Kavya, you were working with a professor on campus, Professor
Tahiliani. I mangled that. How do i pronounce his name?
Kavya Bhat 5:19
Yeah, his name is Professor Mohit Tahiliani, and he is sort of the
sort of our guide in terms of a lot of projects that we do at
NITK. And it was through him that I got to know about a lot of
opportunities such as this one. And when he asked about whether
I'd be interested in joining the project team, I definitely said,
Yes,
George Michaelson 5:42
how big was the team?
Kavya Bhat 5:43
It started off as a small team. So we actually had Professor Mohit
and another professor, Saumya Hegde, and two alumni, students,
Sushanth Rao and Akshay Revankar, working on the IPv6 deployment
thing. And after this, when our alumni graduated, we needed
students specifically working hands on in the campus, and that's
where Vanessa and I joined in. We had worked on previous projects
with Professor Mohit, and through that, we kind of knew of this
opportunity.
George Michaelson 6:16
And Vanessa This was also working with Nalini Elkins, I believe,
yes.
Vanessa Fernandez 6:21
So Nalini and Mike were part of our advisory team, along with
Dhruv from ISOC, yeah. So this whole team of professor, Mohit,
our alumni, Nalini, Mike, Dhruv, they were really instrumental in
mentoring us throughout follow the format once every two weeks or
once every week to kind of catch up and see what work had been
done, work could be done. What were the problems we were facing,
how we could tackle these problems? And, yeah, they just gave us
mentorship throughout, and, you know, helped us work on the
project present at the IETF, things like that that, you know, we
really didn't have much experience in
George Michaelson 6:57
so you said that this is about 45,000 terminal devices that to be
connected, and quite a large number of switches, 350 switches, and
1200 WiFi points. So I'm guessing you already had quite a large
body of work here, finding the devices, getting consoles, getting
images worked out. There was a lot of pre planning involved in
this. Wasn't there.
Kavya Bhat 7:23
This was sort of formed as a project proposal to get a grant to
work on it. There was a very detailed plan of action that was laid
out, and that involved figuring out our address and plan, figuring
out how we would migrate individual VLANs and then ensure that
connectivity was up, while also deploying or enabling IPv6 routing
on particular VLANs.
George Michaelson 7:49
So what size IPv six block were you working under? What was your
amount of V6 address that you had to work with?
Kavya Bhat 7:56
We directly got a slash 36 block that was leased from IRINN, and
we asked BSNL to advertise that particular IPv6 block that we
leased.
George Michaelson 8:06
So BSNL is BSNL is the university's upstream?
Kavya Bhat 8:10
Yes, the BSNL router acts as our upstream. Yeah.
George Michaelson 8:14
And your plan for an address map over this you were doing subnets
for each switch. Did you have a structure diagram in mind for how
you were going to allocate the address plan across this? What was
your population that you were figuring on putting per switch?
Maybe that's not how you did it. Actually, I'd be quite interested
in knowing how you decided to subnet this, because it's not
actually immediately clear that you need subnet structure here. So
did you discuss running a layer two flat network across all of
these switches, or was it a given that you were going to have a
hierarchy?
Kavya Bhat 8:51
So I think we decided to sort of stay a bit with our IPv4
addressing scheme, where we had different VLANs for different
administrative buildings. So we kind of went with that and gave
particular subnets to particular VLANs.
George Michaelson 9:08
Did you do a population count so you would give more addresses to
a network if you knew that it had a higher density of students? Or
were you able to give everyone pretty much exactly the same size
subnet structure.
Vanessa Fernandez 9:21
I don't think we did any such like pre calculation for assigning
addresses. Maybe we should have, in hindsight,
George Michaelson 9:28
no, I don't. It's not a given at all. The thing is that in V6 it's
so different to v4 right? In IPv4, you would be counting every
individual host to make sure you didn't have too many but in V6
Well, it is relative abundance, right?
Vanessa Fernandez 9:45
Yeah, that's true, and that's probably why we kind of stuck to the
v4 addressing scheme. And pretty much whatever we did with v4 we
thought would be a fine template for us to use for V6 as well. So
George Michaelson 9:57
So you obviously didn't start, I know. That's tomorrow. Turn on V6
on the entire campus. Boom, done. Let's go and have a cup of
coffee. What did you actually start with? You started with a test
bed of some kind.
Kavya Bhat 10:09
Yeah. So the initial plan was to set up an IPv6 only test bed, and
that's what we worked on for the first couple of months. When we
got on the project, we set this up in our Department of Computer
Science. One thing we had to ensure was that this IPv6 traffic
wouldn't be affecting any other devices on the same network. And
one reason why we had to ensure that was because we planned to
look at a DHCPv6 server, a DNS server, running on our test bed,
and if we allowed that traffic to go to the other devices, it
would probably cause a lot of chaos in the network.
George Michaelson 10:47
You would have had random devices picking up V6 addresses
everywhere. That wouldn't have been very good. So you ran a small,
isolated network, and you gained experience with running the
fundamental building blocks of a network, a DHCPv6 server and a
DNS server, and you didn't do Stateless auto config, you decided
not to do SLAAC.
Vanessa Fernandez 11:14
So we decided not to do slack in our production environment. For
that reason, we based most of our experiments on DHCPv6 but we did
try out SLAAC on our test bed as well. Like we did try out
configurations, router advertisements and things like that, just
to see how it would work in the grand scheme of things. But for a
production environment, the idea was pretty clear to only use
DHCPv6 which is why we based most of our experiments around that.
George Michaelson 11:41
So you were looking at a fully managed network where address
assignment was explicit to device. You weren't just going to allow
random hookups here. People have to provision. Do you already have
a campus network management system that you were looking at
targeting
Kavya Bhat 11:57
on the staff side of things? They do have a network management
service, openNMS, that they use to monitor devices on the network.
One thing we haven't explored yet is whether openNMS specifically
supports IPv6. We didn't look at the network management systems
yet, but with that in mind, we chose to not go with SLAAC and
prefer DHCPv6 addressing
George Michaelson 12:22
so what was that experience like in general? Did everything work
smoothly? Was it just instant, okay? Or did you find any little
corner cases you had to look at?
Vanessa Fernandez 12:31
This was the first experience for like, both of us, pretty much
working hands on with IPv six and setting the test bit up and
getting it to work end to end was not really as straight forward
as we thought it would be. So there were quite a few issues. Some
that I can think of on top of my head were like setting up the
VLANs, sometimes the DNS server would not work as expected and
things like that. So there were many issues that we face along the
way that we would spend quite some time debugging. So yeah,
George Michaelson 13:00
your presentation at IETF, 118, I think it noted that you had some
unexpected outcomes with things like router signaling in DHCPv6
the systems just didn't behave quite the way you expected by
default.
Kavya Bhat 13:15
Yeah, I think the issues we highlighted there were with respect to
the DSPv6 delay, one of these things was that we see the delay for
a message going out from the client reaching the end server, but
we never saw the deployed reply reaching back to the client. We
were initially very confused about what was going on, because all
the messages seem to be going find from the client, reaching the
opensense router that we had set up, and reaching the DHCPv6
server. And then we kind of concluded, after a bunch of packet
traces and on the different hosts, that it was probably an issue
with the L2 managed switch that we were using, and that it was
somehow dropping the delayed reply messages when we had a DHCP
filter turned on. So we had a lot of issues, like this is where we
spent hours, sometimes days, debugging it, discussing it with our
professors, with our mentors, and it seems to be a hardware issue,
but this is again something that we plan to test with another
switch and conclude properly on our results.
George Michaelson 14:27
There are some concerns that people have talked about in scaling
aspects of the V6 protocol to suit large deployments like Neighbor
Discovery and the risks of flooding. Did you do anything that was
kind of in the defensive space, constructing limits and checks on
the way people were using this service? How did you actually
monitor how things were being deployed?
Kavya Bhat 14:52
Again, with respect to Neighbor Discovery, we mostly worked on the
test bed. We haven't explored too much in the production network.
But there's this one feature called Neighbor Discovery snooping,
where the switch will filter Neighbor Discovery packets based on
IPv6 address to MAC address table that it has constructed and on
our test bed. One thing we noticed was that if the systems are
connected and then we enable this feature, we never see those
entries being generated and put in the table, and we were always
confused as to why, because while it should filter those messages,
and that's what you would expect to see. And after again, a couple
of hours of debugging, we found out that if the system newly
joined the network, say it was unplugged or turned off, and then
it newly joined and started broadcasting Neighbor Discovery
messages, we would see those entries being generated
automatically. And we feel this could be a useful feature to have
on our production network, but it would require some sort of
static IPv6 address to MAC address binding, because, again, if
some new host automatically comes on the network, they could
advertise any particular address they wanted. So there are
definitely some security aspects that we still need to explore,
and that's part of our next steps for the project.
Vanessa Fernandez 16:16
So Vanessa, I think you were also looking at the DHCP V6 server
using real database, persistent store, rather than just memory,
back state. Were you actually doing code work on that?
Not much code work. So the aim was to just look into the
production environment best practices for DHCPv6, like load
balancing, whether we would need a redundant server or not, what
type of redundancy you want to introduce and things like that, and
then also migrating our back end from the basic mem file storage
to a more persistent database, like a SQL database in the back
end. So I hadn't looked into scripting or something like that, but
just the migration of the database from mem file to a persistent
database.
George Michaelson 17:02
So the capability was built into the tool. You just had to select
the right options for it to go into a deployment in the field,
rather than just as an experiment,
Vanessa Fernandez 17:11
kind of Yeah. So we would have to set up a SQL database in the
back end, and then give the DHCP V6 server access to that
database, and then it would work fine. Yeah.
George Michaelson 17:22
So what state of deployment Have you actually reached at the end
of this activity today? Do you now have a fully enabled campus, or
are you still in deployment phases? VLAN? By VLAN,
Vanessa Fernandez 17:33
we are currently in deployment phases. So we've managed to migrate
our data center VLAN to IPv6, which is great. And we've also set
up a DHCPv6 server at the data center, VLAN, that serves our TRIPC
VLAN, that's our central Computing Center VLAN. So that's done,
but we now are looking into setting up an VLAN, which is a trusted
VLAN, and we would add trusted users to this VLAN enable IPv6
routing on it and see if there are any potential issues that need
to be addressed. If everything goes fine, we hope to migrate VLANs
one by one, maybe start at the hostel buildings, the academic
buildings and so on, and just totally expand our migration.
George Michaelson 18:17
What did you present at IETF? What did you find most interesting
about this activity that you presented to IETF?
Kavya Bhat 18:25
I think the first IETF we attended, IETF118 was at the point where
we had almost completed setting up our test bed. And that was one
of our first activities with respect to IPv six hands on work. And
there were a lot of challenges we faced in terms of initial
understanding of the particular protocols or how they were
expected to work. Even when deploying, we would see issues where
we weren't able to ping the external Internet. In some cases, we'd
have DNS resolution issues. And these findings were something we
were particularly excited to share, because as students, we were
looking forward to interacting with the whole IETF community and
getting input on how things worked, or whether our observations
were as expected or sort of out of the blue.
George Michaelson 19:17
And for the second presentation that you did at IETF119,
Vanessa Fernandez 19:22
yeah. So for the second presentation, we had a couple more updates
in terms of issues that we had faced on a test bed, like as Kavya
mentioned at the first IETF, we had just finished setting up our
test bed. So there we received a lot of feedback on things that we
could try out. But at our second IETF, we had already tried out a
bunch of things. For example, the DHCPv6 filtering, which was a
problem that we had faced, the Neighbor Discovery issues, even DNS
scope, sometimes the end client did not receive information from
the DNS server, and also some route advertisement configuration
issues that we had faced. So there were a couple issues like this
that we wanted to bring up, and as Kavya mentioned, just see if
this was something that only we were facing, or if other
enterprises were facing similar problems during deployment as
well.
George Michaelson 20:10
So you wouldn't have gone into this activity completely cold. You
did V6 training as part of your entry into this I think you've
told me previously that you did some APNIC Academy work, is that
right?
Vanessa Fernandez 20:24
Yes. So before we joined the project, our professor suggested to
us that we do the APNIC Academy IPv6 tutorials. So that was my
first introduction into IPv6. And along with that, we also did the
labs, which we found really, really helpful for the testbed
deployment, because there was some useful labs, especially the
APNIC lab on Kia, DHCPv6 server. That was one that I can think of
the top of my head, that we went back to again and again, just to
make sure we were doing things right.
George Michaelson 20:56
So it's a virtual network setup. You're actually doing
configuration on Routing and Switching infrastructure in a virtual
network.
Vanessa Fernandez 21:03
Yes, those are all VM setups, and it's playground, pretty much for
us to configure our own servers and end clients.
George Michaelson 21:10
Happy eyeballs. Everybody wants to believe that the minute you
deploy IPv six, that's it. Boom. Game over. People stop using v4
but it isn't really that simple, is it? So, Kavya, did you do any
work measuring happy eyeballs and the relative amounts of usage?
Kavya Bhat 21:27
We didn't specifically work on Happy eyeballs itself, but we did
observe some statistics from IRIS, which is our Student Portal.
It's our learning management system, and the statistics obtained
from that showed that in some cases, with certain OS, such as
Windows, we were seeing that there were more IPv4 hits from users
rather than IPv6. So when we discussed these results with Nalini
Elkins at that time, I think happy eyeballs had been discussed a
bit, and we thought that maybe happy eyeballs could be a factor
for that, but we didn't specifically go into measuring or testing
if that was the definite reason.
Vanessa Fernandez 22:10
Just want to add that we did bring up happy eyeballs at IETF118, I
mean, we thought it was a potential issue,d mentioned, but we
realized later on that at that point, most of our on campus
population, like the staff and students, were using Windows
laptops or Windows devices, and that's probably why we saw more
IPv4 hits versus IPv6 hits, like there was no IPv6 routing enabled
on Campus at that point, right? And all of our staff and students
would be accessing IRIS through v4,
George Michaelson 22:45
Whereas the VPN access and the remote access would have included
more people on mobile devices, and because mobile handsets a dual
stack, and because you already had from the alumni, a dual stack
deployment in the VPN, you would have seen higher numbers there,
right?
Kavya Bhat 23:03
Our statistics kind of consolidated that theory also, because we
did see that IPv6 hits from Android users were higher compared to
the IPv4 hits. And if you consider that mobile users would be
using their cellular data, and a lot of people use ISP such as Jio
or airtel, which do have IPv6 support, and that kind of
consolidates our theories. But again, we could also look into
whether happy eyeballs does play a role.
George Michaelson 23:33
You mentioned this campus course management system IRIS is it? Did
you actually have to do liaison work to help people do a migration
to make that dual stack or was there very little effort required
to turn that into a dual stack service? How did that go?
Vanessa Fernandez 23:49
Actually, IRIS is a student developed information management
system so entirely developed by our alumni and maintained by the
current students at NITK. So yeah, it was a student led effort,
even migrating the IRIS applications and servers to IPv6 entirely
done by students. So there was a team, dedicated team, working on
migrating the applications,
George Michaelson 24:13
and they got access to your testbed network. They were able to
validate that application and make sure it was working. So the
future deployments are going to go ahead because they know it
works okay in the dual stack environment, [Vanessa yep] Do you see
yourselves carrying on in the network space? Has this experience
been sufficiently positive? It kind of makes you think, oh, yeah,
I could be in networking. It's fun.
Vanessa Fernandez 24:36
Yes, at least for me. I think after having this experience working
on IPv6 migration. I'm lucky that I got such a hands on experience
at college as an undergrad student, and I am even more excited now
to continue working in this space, continue being part of the
migration effort at NITK, and yeah, just being involved in any
capacity that I can.
George Michaelson 24:58
And Kavya, you're at CMU next year or this year?
Kavya Bhat 25:02
Um, yeah, I'll be joining the Masters in Information networking at
CMU this fall. So definitely, my undergrad experience in the on
networking projects definitely made my whole undergrad life
positive experience, I'd say, and consolidated my decision to go
for higher studies in the same field, and I'm hoping that I'll get
to continue working on networking projects, and I'll definitely be
helping the NITK community with the projects they take up as well.
George Michaelson 25:33
So you're both going to become an alumni and work on this activity
the same way the alumni you worked with did carry the story
forward, [Kavya yeah] it's really great. You got some assistance
from APNIC foundation towards attendance at the IETF meetings.
Vanessa Fernandez 25:49
Yep. So for IETF118, we were able to attend because of the IPv6
deployment grant that we got from APNIC foundation in ISIF Asia.
So we were able to attend through that. And for IETF119, APNIC
foundation selected Kavya and I as one of the Internet standards
ambassadors to attend in Brisbane. So yeah, both the IETFs, we
were able to attend through the APNIC foundation. Really grateful
for that.
George Michaelson 26:16
So is a write up available of what went on here? Is there anything
we can point people at to look at your experiences doing this
deployment.
Vanessa Fernandez 26:24
I don't think anything has been published yet, like we have
documents ready, but I don't think they've been published.
George Michaelson 26:30
But I could point people at the IETF 118 presentation for sure.
But the side meeting at 119 was not recorded.
Vanessa Fernandez 26:40
No, the side meeting was not recorded, but we do have a recording
for it.
George Michaelson 26:45
That's really super. Thank you very much for coming on the
program. Thank you Vanessa, thank you Kavya,
Vanessa Fernandez 26:50
thank you George. It's been wonderful.
Kavya Bhat 26:52
Thank you for this opportunity. And yeah, we had a wonderful time.
George Michaelson 26:57
If you've got a story or research to share here on ping. Why not
get in contact by email to ping@apnic.net or via the APNIC social
media channels. Also remember that measurement@apnic.net mailing
list on orbit is there to discuss and share relevant collaborative
opportunities, grants and funding opportunities, jobs and graduate
placings, or to seek feedback from the community on your own
measurement projects, be sure to check out the APNIC website for
All your resource and community needs until next time.
