Elizabeth (Plabayo): Welcome to netstack.fm, episode 33, recorded on the 31st March, 2026. Our protocol shorts. So if I understand well, middle boxes can see the server name indication.
Glen (Plabayo): Today
Elizabeth (Plabayo): And does this affect how networks behave? ⁓ Today, we are talking with Glen about TLS encrypted Hello, which I understand about hiding which websites you are visiting, ⁓ not just the data itself. Can you, Glen, ⁓ walk us through into why matters?
Glen (Plabayo): So if I understand well, middle boxes can see the server name indication. ⁓ this affect how networks ⁓ ⁓ In a perfect world, wouldn't matter at all because in a perfect world, these middle boxes, let's say the middle boxes operated by telecom providers or whoever is helping us to operate the internet would not really check at all what is being trafficked over TCP. All they would need to know is what is the destination IP and they would do the routing and it will end up in this destination. However, we know that they do check on things they sometimes shouldn't check. For example, they might are talking with Glen about DLS encrypted client hello ⁓ Which understand it's about hiding which websites you are visiting ⁓ not the data itself ⁓ Can Glen ⁓ walk through ⁓ into that matters? ⁓ Yeah, sure, Elizabeth. So TLS is all about encrypting the traffic and most people know it with the padlock in their browser. it adds encryption to your web traffic.
Elizabeth (Plabayo): Yeah, sure, Elizabeth. So TLS is all about encrypting the traffic and most people know it with the backlog in their browser. ⁓ It adds encryption to your traffic and allows you visit the website without anybody snooping on what you're doing. This is especially information when you're dealing with testing information such as payment flows. ⁓
Glen (Plabayo): and allowing you to visit a website without anybody snooping on what you're doing. This is especially information when you are dealing with sensitive information such as payment flows or personal information or anything else. Or it also ensures that you can trust the data you are receiving back is actually coming from the server you're expecting to. So far, check sometimes TCP port they have to use then they might filter out or drop traffic that they don't want to. ⁓ Or they might accept TCP but they don't accept UDP. Similarly, and I don't know why, it could be all kinds of they might wish to drop your traffic in case they see it's traffic destined for a website they don't support. ⁓ When we were talking about TLS or when we were dealing with TLS, we were only dealing with encrypting the web traffic itself or the application data, such as the HTML or the CSS pages or JavaScript or whatever you are sending back and forth between client and server. What we were not yet encrypting, and for the most part still are not, are the parameters that we are negotiating in the handshake phase of the protocol. For example, maybe a government decides that a certain chat platform is not allowed and so they might drop all traffic destined for that ⁓ server. Or maybe they want to see, let's say they are a telecom provider, maybe they want to see kind of websites are my clients using because then they could try to bundle this in all kinds of products. mostly not much is it a network issue, it's mostly a matter of privacy. If that makes sense to you, Elizabeth. of this information includes sensitive information such as the server name indication which is basically the domain of the website that you are visiting as well as the application layer protocol such as HTTP 2 or HTTP 1 that you are using to visit the website.
Elizabeth (Plabayo): Or maybe they want to see, let's say they are a telecom provider, maybe they want to see kind of websites ⁓ my clients using because then they could try to bundle this in all kinds of products. mostly not so much is it a network issue, it's mostly a matter of privacy. If that makes sense to you Elizabeth.
Glen (Plabayo): This information was so far in the clear and could be used by any party that can see your traffic, including all the middleboxes and routers your traffic is going through.
Elizabeth (Plabayo): This information was so far in the clear and could be used by any party that can see your traffic, including all the middleboxes and routers your traffic is going through. Yeah, this seems like a privacy gap. Have people been trying to solve this for a while?
Glen (Plabayo): For a while, yes, definitely several years and I imagine that behind closed doors people have been thinking about it for much longer. What I know is that since around 2018... Companies like Cloudflare and the IETF, started to experiment with something that is called encrypting the server name indication, meaning they introduced a new extension that could be added to the client hello, where they would send the server name indication but in an encrypted format. And the way it would work is that they could check the text DNS records ⁓ of same website, where the public key was advertised that
Elizabeth (Plabayo): Before we go deeper, can you give us a quick refresher on how a TLS handshake actually works? And at what point does encryption start that process? Yeah, of course. Sounds like a idea. Now, an entire discussion or a complete explanation on how TLS works is a bit out of scope for this podcast episode because it would take too long. But I will try to highlight the relevant parts.
Glen (Plabayo): Yeah, of course, sounds like a good idea. Now, an entire discussion or a complete explanation on how TLS works is a bit out of scope for this podcast episode because it would take too long. But I will try to highlight the relevant parts. So like any protocol, we first need to negotiate. This is because protocols can have different versions, but also most protocols allow configurations and these configurations need to be negotiated because if the client and server are using Inco
Elizabeth (Plabayo): Netstack.fm is brought to you by Plabayo building secure, and resilient infrastructure with Rust protocols, and purpose. This show is also made possible by Rama, the open source networking framework. Plabayo offers service contracts and welcome sponsorships to keep building and supporting its ecosystem.
Glen (Plabayo): would use to encrypt the server name indication. So they would use that public key found in the text DNS records, encrypt the server name indication and give it together with the client hello as a new extension for a name indication with an encrypted format. And this kind of worked. ⁓ configurations that wouldn't work. but they had some issues and on top of that, it was only covering the server name indication. But that's more or less the first public research and attempt tried by companies such as Cloudflare.
Elizabeth (Plabayo): The theme music of this podcast was composed by DJ Mailbox. but they had some issues and on top of that it was only covering the server name indication. But that's more or less the first public research and attempt by companies such as Cloudflare. If you enjoyed this episode, don't forget to subscribe on your favorite platform and leave a five-star review. It really helps others discover the show. So how does ECH actually fix this Thanks for tuning in. We'll see you next time for the next handshake. privacy gap? Well, similar to the first where they were encrypting the server name indication, the goal is to encrypt most of the client hello. That would leave your actual client hello rather small. You would still include some information that doesn't really matter.
Glen (Plabayo): So as with most client server protocols, it is the client that will initiate the handshake, just like it was also the client initiating the connection. It will send over to the server what is called the client hello. This includes all the parameters that are configured by the client for this TLS connection. It includes things like its version, things like which encryption algorithms it speaks, things like... privacy gap? Well, similar to the first attempt where they were encrypting the server name indication, the goal is to encrypt most of the client's hello. That would leave your actual client hello rather small. You would still include some information that doesn't really matter, but most of the other parameters you want to put in a similar kind of records. and have it also encrypted and send that to the server. And similarly to the initial attempt where they were getting a public key from the domain in the DNS records, they will now use what is called an HTTPS DNS records instead of the text records. The cool thing about an HTTPS record is that this is using DNS over TLS, usually DNS over HTTPS, meaning that your actual check Did we have a connection before and can we speed up the process by sharing already the secret that we negotiated before? But especially it also includes things like which application layer protocols do I speak, such as HTTP2, and also the server name indication, which is basically the domain of the server it wants to reach. This information is important because quite often you have a gateway which is doing the TLS encryption and decryption and it is only like some kind of backend server behind the gateway which is then dealing with the actual HTTP protocol itself. Meaning the gateway will serve web traffic for many different web servers. ⁓ is also done encrypted. can see that the client is doing a lookup but they wouldn't be able to see what is the DNS lookup being done because ⁓ also the traffic between the client and the DNS server also be encrypted. That is what DNS over HTTPS is doing which I believe is sometimes abbreviated as DOH. ⁓ So the server name indication is very important for that. So we are at the point that the client sends all the information to the server. The server will see this and see if the client hello config is compatible with how it operates. If it is, including if it knows the destination that is targeted using the server name indication. And if it knows that that web server can handle that protocol version, such as HTTP2. And so they are getting this public key from the HTTPS records and using that public key, they can encrypt. most of the client hello content and save it as an extension called the encrypted client hello extension. And that is then being sent as part of the regular client hello to the TLS server. The cool thing about this is, is that the server is the only one, the TLS server is the only one who can see the actual content of most of the client hello content. It will then send back the server which basically just the information agreed upon on the client hello and the server, like which parameters do they agree on. It will send this back to the client together with its certificate, which is the public key of the TLS server. And then it will end this with a server hello done. The client will receive this and at this point, And then that would mean in an ideal world that the public information available in the non-encrypted part of the client hello is no longer containing any sensitive information, such as the actual destination server name indication. It will use that public key of the server, the certificate, to check, do I trust that? It can go up to the chain and whatever thing that signs that key or whatever thing that sign as a certificate, it needs to trust it, which is usually in a trust chain on its browser or its operating system. Nowadays, a lot of web traffic is using certificates signed by Let's Encrypt, which made...
Elizabeth (Plabayo): And then that would mean in an ideal world that the public information available in the non-encrypted part of the client's hello is no longer containing any sensitive information, such as the actual destination server name indication. indication... disappears completely with ECH.
Glen (Plabayo): You could imagine a world where that's true, but most likely no. I don't see a future right now where that's happening. Because something that is terminating at the LS traffic will still want to know if it's destined for it or not. So most likely it still wants some of serving of indication. certificate signing free. Before that, companies had to pay money every year, just like companies pay money for domains. But thanks to initiatives like Let's Encrypt, now we can all get free TLS certificates. But anyway, it doesn't really matter for this conversation. What matters is the client is validating the certificate and checking trusts it, it will use that to negotiate with the server. on the private key used for the actual encryption of the web traffic because public and private key encryption is very expensive and so we want to get to a point where we just use regular encryption meaning the encryption and decryption is done using the same key and so this is negotiated using public information between the client and the server. so let's give an example just to it clear. As I said, most web servers are not terminating their own TLS traffic. Now that we are in the cloud era, maybe even past the cloud era, I'm not sure in history, most companies operate within the cloud and those clouds are big companies such as Cloudflare, Google, Microsoft, etc. What is quite often the case nowadays is that those companies are the ones actually terminating your TLS traffic and sending the decrypted HTTP traffic to the actual destination backend service. And so once this final negotiation is finished, we have established between the client and the server, not only the configuration used, but also the shared secret that we'll use to encrypt the actual traffic. And at that point, everything else ⁓ is encrypted. And if you would like inspect this web traffic using a tool like Wireshark. operated by some company in that cloud. So let's say example.com. that would mean that the actual plain text server name indication will be instead of the real destination address, will be something like tls.cloudcompany.com instead of example.com. anybody. ⁓ you will just see segments as application data, but the content you will no longer be able to understand because you don't own the private key or either the server or the client. that sees the traffic will only be able to see at that point this is traffic destined for that cloud. That's however not information they would also not be able to decipher or deduce from the IP address because if you would do a lookup on the IP address which is anyway plainly visible you would also find that it's an IP address owned by that same cloud operator. So at that point you removed the privacy gap because you are not leaking anything in your client hello that you also couldn't deduce from lower layers on the network.
Elizabeth (Plabayo): because you are not leaking anything in your client hello that you also couldn't deduce from lower layers on the network.
Glen (Plabayo): Well, many TLS is a black box, especially for but even for most... developers, TLS is a bit of a black book. So they might use it and they might not realize it, that's true. So if we look at the timeline a bit, we talked first about the encryption server name indication. And that was an idea that was started by the Cloudflare and ITF and maybe some other companies. And that was in 2018. Now, That didn't go anywhere, but we learned a lot from it, or rather they learned a lot from it. And then around 2020, they started playing with this idea of what we now call Encrypted Client Hello. And so the Encrypted Client Hello started around 2020, where Cloudflare was supporting it on the server side, because Cloudflare is one of those clouds which is terminating the TLS traffic for many web servers. And a bit later, I believe it was Firefox, which is a web client, a browser, was also shipping its version where they support that. Because in order for this to work, both the client and the server need to be able to support it. Now, given that most web traffic is initiated from a browser, And also knowing the fact that there only so many browsers, the majority of browsers being either like Firefox or some kind of Chromium derived fork with Google Chrome being the most popular one. you know that most clients already support ECH. So you were asking if I visit the website, will I be able to use it? Yes, most likely. But also the server needs to be able to support it. But also given the fact that most web servers are not doing their own TLS, but rather are delegating that work to the cloud that they are using, or some kind of middleman such as Cloudflare. and given that they also can easily support it and most already do by this point, then the answer is yes. Most likely you're already using at least for some of your web traffic ECH and enjoying the privacy benefits that go with it. You simply don't realize. But this is not only something that users might not realize, it's also something that developers might not realize because either the developer is not doing TLS themselves because they are delegating it to the cloud where their web server is hosted. Or even if they do TLS, it's not that they are hand-trolling their own TLS implementations. They will be using a TLS library which is doing the TLS for it. When they are writing the web server which is serving HTTPS traffic, so HTTP over TLS, they are using a library for TLS and there are only a couple of them. Examples include OpenSSL, Rust TLS, they are only like a handful commonly used TLS libraries. And one by one, if not all of them already, start or already support ECH. Meaning if that developer that is still doing TLS themselves in their web server is updating its TLS library, it will now also support ECH. And so for example recently there was even like a blog post by a curl contributor where they advertised we are now shipping a first version of curl with ECH enabled. And that's not so much because they had to do a lot of work for it. is more that they had to update their TLS dependencies, their libraries that they use for TLS to start benefiting from ECH, but all a bit behind the library they're using. They're not doing it themselves.
Elizabeth (Plabayo): is more that they have to update their TLS dependencies, their libraries that they use for TLS, to start benefiting from ECH. But all a bit behind the library they're using. They're not doing it themselves. Very, very interesting. Is ECH already something people are using today or is it still experimental? For example, if I'm just browsing the web, there any chance that I'm already using ECH ⁓ without realizing it? what are the consequences of using ECH for existing infrastructures? For systems that rely on inspecting traffic, what happens to them?
Glen (Plabayo): are the consequences of using ECH for existing infrastructures? For systems that rely on inspecting traffic, ⁓ what to them? Yeah, excellent question. ⁓ So now we are a bit talking about an edge case because for most of the traffic flows, they shouldn't matter at all because this is really something that the client and server agree upon and parties in the middle should in theory, according to the letter of the protocols, not really inspect or let alone... intercept this kind of traffic for any purpose. However, there are religious reasons why you want to do this. For example, if you're in an enterprise or any other kind of environment where security is of the highest importance, it can be important and acceptable that the company wants to terminate and inspect this traffic. This is for example because they wanna protect the data that is going out of the company or ensuring that no bad malware is coming inside the organization. To reduce the workload on these kind of layer 4 proxies, and we talked about this in a previous protocol short episode, we will also link it in the show notes. They rely on things such as what protocol is being used, TCP, UDP, but then also on the TLS layer, they will inspect the client hello to check on the server name indication and... Most traffic they will not care about, but some domains they might want to inspect further because these are suspicious domains or these are domains for which there is a high security matter. In case the server name indication which is readily available to anybody reading the traffic no longer contains the real server name indication, it means that those proxies can no longer rely on this field to know whether or not they want to inspect and decrypt the TLS traffic. Meaning they have only two choices. Either they do not encrypt the traffic or they decrypt all traffic, which... increases the workload they have to do and also increases the exposure of possible mistakes they might make because man-in-the-middle traffic means you're also changing the traffic a bit even if not on purpose and so you might make a mistake as a proxy implementer and so that's why also not only to reduce your workload you also want to prevent the amount of mistakes you can make. Now in a way, like the bits. There's more than two choices. It's not that they have to choose between not decrypting anything or decrypting everything. They could also try instead They could also choose to also man in the middle of DNS traffic. Because as we explained in the previous question, the way this works is that the client hello is encrypted using the public key found in the HTTPS records. That means that that client needs to talk to a DNS server, the DNS server configured in its system ⁓ in its router. Just like the layer 4 proxy can terminate the web traffic, it can also terminate the DNS traffic or inspect it. And it could choose to give it for that record, for that lookup, its own public key, the public key that is linked to the private key that the proxy owns. Instead of relaying that DNS query to the actual DNS resolver servers. That would mean that the public key received by the TLS client supporting encryptClientHello would now use this public key of the proxy to encrypt the encrypted client Hello. that would allow the proxy to once again ⁓ the client Hello. This would work ⁓ ⁓ If one, the proxy does that, so also is intercepting the DNS traffic. And two, similar like how the client would need to trust the certificate used or issued by the proxy for the web traffic, it will also need to trust the public for encrypted client hello. But of course, because it's given via DNS record, the client will trust it. So this would be one way. for the layer four proxy in this kind of security infrastructure to still use a relatively cheap lookup on whether or not we want to man in the middle of this traffic. But it is extra work for the infrastructure. so existing infrastructure as is, which is only focused on inspecting web traffic, will no longer work. It will need to add this new capability to also intercept the DNS traffic.
Elizabeth (Plabayo): traffic will no longer work. It will need to add this new capability to also intercept the DNS traffic. I understand. And does this mean tools like Rama need to higher up the stack to stay effective? So Rama is the free and open source framework that we develop and maintain at Labio.
Glen (Plabayo): So Rama is the free and open source framework that we develop and maintain at Plabaio. And so it is a modular framework for building network service in Rust. And it handles the different protocols involved in network services, going from TCP, UDP, DNS, HTTP, TLS, et cetera. Some companies, such as security companies, also use Rama to... inspect traffic and man-in-the-middle traffic that is of relevance. And so those companies that are maybe for now only man-in-the-middling the web traffic or the application layer TCP traffic would also start to inspect and intercept the DNS traffic. That's correct because if they do not do that yet, then they would indeed not be able to rely on the plain text server name indication.
Elizabeth (Plabayo): it well. Where do you see this is going in the next few years? Is this a clear win for privacy or are we trading off something important on the network side? What do you think?
Glen (Plabayo): where do you see this is going in the next few years? Is a clear win for privacy or are we trading off something important on the network side? What do you think? It's a bit of a difficult question to answer. Intuitively you would think this is a clear win for privacy and I do believe it is. However, in the end you do still put your trust somewhere. So we talked a bit about middleboxes and we talked about, so far this information was available in the clear text client hello, which means middlebox from your telecom providers could read it or your government or whoever else might be interested in your traffic and they could see what web traffic you are visiting. But we also explained that most... Web servers are not terminating their own TLS traffic and instead relying on their cloud. they are using a cloud such as Microsoft or Cloudflare. Well, that kind of means you have to put your trust in those companies instead of putting your trust in your telecom provider. And I'm not entirely convinced that those companies are less likely to leak at some point your data, be it by accident or selling it on purpose. But you reduce the blast radius because in the end before... anybody seeing your traffic could see that data. So anybody who could see your network traffic before was able to see those domains, those application protocols. Now it is only a handful of actors which do have some kind of image to upheld. Like people do pay them and do have trust in them that their data is not leaked by them, at least not on purpose. So I think overall it's a privacy win, but as long as not every web server is doing its own TLS, it does mean that you still at the very least put your trust in those very few amount of cloud companies that are available in the world right now.
Elizabeth (Plabayo): not leaked by them, at least on purpose. So I think overall it's a privacy win, but as long as not every web server is doing its own TLS, it does mean that you still at the very least put your trust in those very few amount of cloud companies that are available in the world right now. thank you for giving us a ⁓ check. ⁓ Okay. Thank you very much, Glen. I appreciate you give us the time to give a clarification on this subject, TLS ECH. And thank you to all our audience have a good week. Bye. Bye, ⁓ I didn't record.
Glen (Plabayo): Thank you for giving us a reality check. Okay, thank you very much, and I appreciate you give us a time to give a clarification on this subject TLS ECH. And thank you to our audience and have a good week. Bye. bye. ⁓
Glen (Plabayo): Today
Elizabeth (Plabayo): And does this affect how networks behave? ⁓ Today, we are talking with Glen about TLS encrypted Hello, which I understand about hiding which websites you are visiting, ⁓ not just the data itself. Can you, Glen, ⁓ walk us through into why matters?
Glen (Plabayo): So if I understand well, middle boxes can see the server name indication. ⁓ this affect how networks ⁓ ⁓ In a perfect world, wouldn't matter at all because in a perfect world, these middle boxes, let's say the middle boxes operated by telecom providers or whoever is helping us to operate the internet would not really check at all what is being trafficked over TCP. All they would need to know is what is the destination IP and they would do the routing and it will end up in this destination. However, we know that they do check on things they sometimes shouldn't check. For example, they might are talking with Glen about DLS encrypted client hello ⁓ Which understand it's about hiding which websites you are visiting ⁓ not the data itself ⁓ Can Glen ⁓ walk through ⁓ into that matters? ⁓ Yeah, sure, Elizabeth. So TLS is all about encrypting the traffic and most people know it with the padlock in their browser. it adds encryption to your web traffic.
Elizabeth (Plabayo): Yeah, sure, Elizabeth. So TLS is all about encrypting the traffic and most people know it with the backlog in their browser. ⁓ It adds encryption to your traffic and allows you visit the website without anybody snooping on what you're doing. This is especially information when you're dealing with testing information such as payment flows. ⁓
Glen (Plabayo): and allowing you to visit a website without anybody snooping on what you're doing. This is especially information when you are dealing with sensitive information such as payment flows or personal information or anything else. Or it also ensures that you can trust the data you are receiving back is actually coming from the server you're expecting to. So far, check sometimes TCP port they have to use then they might filter out or drop traffic that they don't want to. ⁓ Or they might accept TCP but they don't accept UDP. Similarly, and I don't know why, it could be all kinds of they might wish to drop your traffic in case they see it's traffic destined for a website they don't support. ⁓ When we were talking about TLS or when we were dealing with TLS, we were only dealing with encrypting the web traffic itself or the application data, such as the HTML or the CSS pages or JavaScript or whatever you are sending back and forth between client and server. What we were not yet encrypting, and for the most part still are not, are the parameters that we are negotiating in the handshake phase of the protocol. For example, maybe a government decides that a certain chat platform is not allowed and so they might drop all traffic destined for that ⁓ server. Or maybe they want to see, let's say they are a telecom provider, maybe they want to see kind of websites are my clients using because then they could try to bundle this in all kinds of products. mostly not much is it a network issue, it's mostly a matter of privacy. If that makes sense to you, Elizabeth. of this information includes sensitive information such as the server name indication which is basically the domain of the website that you are visiting as well as the application layer protocol such as HTTP 2 or HTTP 1 that you are using to visit the website.
Elizabeth (Plabayo): Or maybe they want to see, let's say they are a telecom provider, maybe they want to see kind of websites ⁓ my clients using because then they could try to bundle this in all kinds of products. mostly not so much is it a network issue, it's mostly a matter of privacy. If that makes sense to you Elizabeth.
Glen (Plabayo): This information was so far in the clear and could be used by any party that can see your traffic, including all the middleboxes and routers your traffic is going through.
Elizabeth (Plabayo): This information was so far in the clear and could be used by any party that can see your traffic, including all the middleboxes and routers your traffic is going through. Yeah, this seems like a privacy gap. Have people been trying to solve this for a while?
Glen (Plabayo): For a while, yes, definitely several years and I imagine that behind closed doors people have been thinking about it for much longer. What I know is that since around 2018... Companies like Cloudflare and the IETF, started to experiment with something that is called encrypting the server name indication, meaning they introduced a new extension that could be added to the client hello, where they would send the server name indication but in an encrypted format. And the way it would work is that they could check the text DNS records ⁓ of same website, where the public key was advertised that
Elizabeth (Plabayo): Before we go deeper, can you give us a quick refresher on how a TLS handshake actually works? And at what point does encryption start that process? Yeah, of course. Sounds like a idea. Now, an entire discussion or a complete explanation on how TLS works is a bit out of scope for this podcast episode because it would take too long. But I will try to highlight the relevant parts.
Glen (Plabayo): Yeah, of course, sounds like a good idea. Now, an entire discussion or a complete explanation on how TLS works is a bit out of scope for this podcast episode because it would take too long. But I will try to highlight the relevant parts. So like any protocol, we first need to negotiate. This is because protocols can have different versions, but also most protocols allow configurations and these configurations need to be negotiated because if the client and server are using Inco
Elizabeth (Plabayo): Netstack.fm is brought to you by Plabayo building secure, and resilient infrastructure with Rust protocols, and purpose. This show is also made possible by Rama, the open source networking framework. Plabayo offers service contracts and welcome sponsorships to keep building and supporting its ecosystem.
Glen (Plabayo): would use to encrypt the server name indication. So they would use that public key found in the text DNS records, encrypt the server name indication and give it together with the client hello as a new extension for a name indication with an encrypted format. And this kind of worked. ⁓ configurations that wouldn't work. but they had some issues and on top of that, it was only covering the server name indication. But that's more or less the first public research and attempt tried by companies such as Cloudflare.
Elizabeth (Plabayo): The theme music of this podcast was composed by DJ Mailbox. but they had some issues and on top of that it was only covering the server name indication. But that's more or less the first public research and attempt by companies such as Cloudflare. If you enjoyed this episode, don't forget to subscribe on your favorite platform and leave a five-star review. It really helps others discover the show. So how does ECH actually fix this Thanks for tuning in. We'll see you next time for the next handshake. privacy gap? Well, similar to the first where they were encrypting the server name indication, the goal is to encrypt most of the client hello. That would leave your actual client hello rather small. You would still include some information that doesn't really matter.
Glen (Plabayo): So as with most client server protocols, it is the client that will initiate the handshake, just like it was also the client initiating the connection. It will send over to the server what is called the client hello. This includes all the parameters that are configured by the client for this TLS connection. It includes things like its version, things like which encryption algorithms it speaks, things like... privacy gap? Well, similar to the first attempt where they were encrypting the server name indication, the goal is to encrypt most of the client's hello. That would leave your actual client hello rather small. You would still include some information that doesn't really matter, but most of the other parameters you want to put in a similar kind of records. and have it also encrypted and send that to the server. And similarly to the initial attempt where they were getting a public key from the domain in the DNS records, they will now use what is called an HTTPS DNS records instead of the text records. The cool thing about an HTTPS record is that this is using DNS over TLS, usually DNS over HTTPS, meaning that your actual check Did we have a connection before and can we speed up the process by sharing already the secret that we negotiated before? But especially it also includes things like which application layer protocols do I speak, such as HTTP2, and also the server name indication, which is basically the domain of the server it wants to reach. This information is important because quite often you have a gateway which is doing the TLS encryption and decryption and it is only like some kind of backend server behind the gateway which is then dealing with the actual HTTP protocol itself. Meaning the gateway will serve web traffic for many different web servers. ⁓ is also done encrypted. can see that the client is doing a lookup but they wouldn't be able to see what is the DNS lookup being done because ⁓ also the traffic between the client and the DNS server also be encrypted. That is what DNS over HTTPS is doing which I believe is sometimes abbreviated as DOH. ⁓ So the server name indication is very important for that. So we are at the point that the client sends all the information to the server. The server will see this and see if the client hello config is compatible with how it operates. If it is, including if it knows the destination that is targeted using the server name indication. And if it knows that that web server can handle that protocol version, such as HTTP2. And so they are getting this public key from the HTTPS records and using that public key, they can encrypt. most of the client hello content and save it as an extension called the encrypted client hello extension. And that is then being sent as part of the regular client hello to the TLS server. The cool thing about this is, is that the server is the only one, the TLS server is the only one who can see the actual content of most of the client hello content. It will then send back the server which basically just the information agreed upon on the client hello and the server, like which parameters do they agree on. It will send this back to the client together with its certificate, which is the public key of the TLS server. And then it will end this with a server hello done. The client will receive this and at this point, And then that would mean in an ideal world that the public information available in the non-encrypted part of the client hello is no longer containing any sensitive information, such as the actual destination server name indication. It will use that public key of the server, the certificate, to check, do I trust that? It can go up to the chain and whatever thing that signs that key or whatever thing that sign as a certificate, it needs to trust it, which is usually in a trust chain on its browser or its operating system. Nowadays, a lot of web traffic is using certificates signed by Let's Encrypt, which made...
Elizabeth (Plabayo): And then that would mean in an ideal world that the public information available in the non-encrypted part of the client's hello is no longer containing any sensitive information, such as the actual destination server name indication. indication... disappears completely with ECH.
Glen (Plabayo): You could imagine a world where that's true, but most likely no. I don't see a future right now where that's happening. Because something that is terminating at the LS traffic will still want to know if it's destined for it or not. So most likely it still wants some of serving of indication. certificate signing free. Before that, companies had to pay money every year, just like companies pay money for domains. But thanks to initiatives like Let's Encrypt, now we can all get free TLS certificates. But anyway, it doesn't really matter for this conversation. What matters is the client is validating the certificate and checking trusts it, it will use that to negotiate with the server. on the private key used for the actual encryption of the web traffic because public and private key encryption is very expensive and so we want to get to a point where we just use regular encryption meaning the encryption and decryption is done using the same key and so this is negotiated using public information between the client and the server. so let's give an example just to it clear. As I said, most web servers are not terminating their own TLS traffic. Now that we are in the cloud era, maybe even past the cloud era, I'm not sure in history, most companies operate within the cloud and those clouds are big companies such as Cloudflare, Google, Microsoft, etc. What is quite often the case nowadays is that those companies are the ones actually terminating your TLS traffic and sending the decrypted HTTP traffic to the actual destination backend service. And so once this final negotiation is finished, we have established between the client and the server, not only the configuration used, but also the shared secret that we'll use to encrypt the actual traffic. And at that point, everything else ⁓ is encrypted. And if you would like inspect this web traffic using a tool like Wireshark. operated by some company in that cloud. So let's say example.com. that would mean that the actual plain text server name indication will be instead of the real destination address, will be something like tls.cloudcompany.com instead of example.com. anybody. ⁓ you will just see segments as application data, but the content you will no longer be able to understand because you don't own the private key or either the server or the client. that sees the traffic will only be able to see at that point this is traffic destined for that cloud. That's however not information they would also not be able to decipher or deduce from the IP address because if you would do a lookup on the IP address which is anyway plainly visible you would also find that it's an IP address owned by that same cloud operator. So at that point you removed the privacy gap because you are not leaking anything in your client hello that you also couldn't deduce from lower layers on the network.
Elizabeth (Plabayo): because you are not leaking anything in your client hello that you also couldn't deduce from lower layers on the network.
Glen (Plabayo): Well, many TLS is a black box, especially for but even for most... developers, TLS is a bit of a black book. So they might use it and they might not realize it, that's true. So if we look at the timeline a bit, we talked first about the encryption server name indication. And that was an idea that was started by the Cloudflare and ITF and maybe some other companies. And that was in 2018. Now, That didn't go anywhere, but we learned a lot from it, or rather they learned a lot from it. And then around 2020, they started playing with this idea of what we now call Encrypted Client Hello. And so the Encrypted Client Hello started around 2020, where Cloudflare was supporting it on the server side, because Cloudflare is one of those clouds which is terminating the TLS traffic for many web servers. And a bit later, I believe it was Firefox, which is a web client, a browser, was also shipping its version where they support that. Because in order for this to work, both the client and the server need to be able to support it. Now, given that most web traffic is initiated from a browser, And also knowing the fact that there only so many browsers, the majority of browsers being either like Firefox or some kind of Chromium derived fork with Google Chrome being the most popular one. you know that most clients already support ECH. So you were asking if I visit the website, will I be able to use it? Yes, most likely. But also the server needs to be able to support it. But also given the fact that most web servers are not doing their own TLS, but rather are delegating that work to the cloud that they are using, or some kind of middleman such as Cloudflare. and given that they also can easily support it and most already do by this point, then the answer is yes. Most likely you're already using at least for some of your web traffic ECH and enjoying the privacy benefits that go with it. You simply don't realize. But this is not only something that users might not realize, it's also something that developers might not realize because either the developer is not doing TLS themselves because they are delegating it to the cloud where their web server is hosted. Or even if they do TLS, it's not that they are hand-trolling their own TLS implementations. They will be using a TLS library which is doing the TLS for it. When they are writing the web server which is serving HTTPS traffic, so HTTP over TLS, they are using a library for TLS and there are only a couple of them. Examples include OpenSSL, Rust TLS, they are only like a handful commonly used TLS libraries. And one by one, if not all of them already, start or already support ECH. Meaning if that developer that is still doing TLS themselves in their web server is updating its TLS library, it will now also support ECH. And so for example recently there was even like a blog post by a curl contributor where they advertised we are now shipping a first version of curl with ECH enabled. And that's not so much because they had to do a lot of work for it. is more that they had to update their TLS dependencies, their libraries that they use for TLS to start benefiting from ECH, but all a bit behind the library they're using. They're not doing it themselves.
Elizabeth (Plabayo): is more that they have to update their TLS dependencies, their libraries that they use for TLS, to start benefiting from ECH. But all a bit behind the library they're using. They're not doing it themselves. Very, very interesting. Is ECH already something people are using today or is it still experimental? For example, if I'm just browsing the web, there any chance that I'm already using ECH ⁓ without realizing it? what are the consequences of using ECH for existing infrastructures? For systems that rely on inspecting traffic, what happens to them?
Glen (Plabayo): are the consequences of using ECH for existing infrastructures? For systems that rely on inspecting traffic, ⁓ what to them? Yeah, excellent question. ⁓ So now we are a bit talking about an edge case because for most of the traffic flows, they shouldn't matter at all because this is really something that the client and server agree upon and parties in the middle should in theory, according to the letter of the protocols, not really inspect or let alone... intercept this kind of traffic for any purpose. However, there are religious reasons why you want to do this. For example, if you're in an enterprise or any other kind of environment where security is of the highest importance, it can be important and acceptable that the company wants to terminate and inspect this traffic. This is for example because they wanna protect the data that is going out of the company or ensuring that no bad malware is coming inside the organization. To reduce the workload on these kind of layer 4 proxies, and we talked about this in a previous protocol short episode, we will also link it in the show notes. They rely on things such as what protocol is being used, TCP, UDP, but then also on the TLS layer, they will inspect the client hello to check on the server name indication and... Most traffic they will not care about, but some domains they might want to inspect further because these are suspicious domains or these are domains for which there is a high security matter. In case the server name indication which is readily available to anybody reading the traffic no longer contains the real server name indication, it means that those proxies can no longer rely on this field to know whether or not they want to inspect and decrypt the TLS traffic. Meaning they have only two choices. Either they do not encrypt the traffic or they decrypt all traffic, which... increases the workload they have to do and also increases the exposure of possible mistakes they might make because man-in-the-middle traffic means you're also changing the traffic a bit even if not on purpose and so you might make a mistake as a proxy implementer and so that's why also not only to reduce your workload you also want to prevent the amount of mistakes you can make. Now in a way, like the bits. There's more than two choices. It's not that they have to choose between not decrypting anything or decrypting everything. They could also try instead They could also choose to also man in the middle of DNS traffic. Because as we explained in the previous question, the way this works is that the client hello is encrypted using the public key found in the HTTPS records. That means that that client needs to talk to a DNS server, the DNS server configured in its system ⁓ in its router. Just like the layer 4 proxy can terminate the web traffic, it can also terminate the DNS traffic or inspect it. And it could choose to give it for that record, for that lookup, its own public key, the public key that is linked to the private key that the proxy owns. Instead of relaying that DNS query to the actual DNS resolver servers. That would mean that the public key received by the TLS client supporting encryptClientHello would now use this public key of the proxy to encrypt the encrypted client Hello. that would allow the proxy to once again ⁓ the client Hello. This would work ⁓ ⁓ If one, the proxy does that, so also is intercepting the DNS traffic. And two, similar like how the client would need to trust the certificate used or issued by the proxy for the web traffic, it will also need to trust the public for encrypted client hello. But of course, because it's given via DNS record, the client will trust it. So this would be one way. for the layer four proxy in this kind of security infrastructure to still use a relatively cheap lookup on whether or not we want to man in the middle of this traffic. But it is extra work for the infrastructure. so existing infrastructure as is, which is only focused on inspecting web traffic, will no longer work. It will need to add this new capability to also intercept the DNS traffic.
Elizabeth (Plabayo): traffic will no longer work. It will need to add this new capability to also intercept the DNS traffic. I understand. And does this mean tools like Rama need to higher up the stack to stay effective? So Rama is the free and open source framework that we develop and maintain at Labio.
Glen (Plabayo): So Rama is the free and open source framework that we develop and maintain at Plabaio. And so it is a modular framework for building network service in Rust. And it handles the different protocols involved in network services, going from TCP, UDP, DNS, HTTP, TLS, et cetera. Some companies, such as security companies, also use Rama to... inspect traffic and man-in-the-middle traffic that is of relevance. And so those companies that are maybe for now only man-in-the-middling the web traffic or the application layer TCP traffic would also start to inspect and intercept the DNS traffic. That's correct because if they do not do that yet, then they would indeed not be able to rely on the plain text server name indication.
Elizabeth (Plabayo): it well. Where do you see this is going in the next few years? Is this a clear win for privacy or are we trading off something important on the network side? What do you think?
Glen (Plabayo): where do you see this is going in the next few years? Is a clear win for privacy or are we trading off something important on the network side? What do you think? It's a bit of a difficult question to answer. Intuitively you would think this is a clear win for privacy and I do believe it is. However, in the end you do still put your trust somewhere. So we talked a bit about middleboxes and we talked about, so far this information was available in the clear text client hello, which means middlebox from your telecom providers could read it or your government or whoever else might be interested in your traffic and they could see what web traffic you are visiting. But we also explained that most... Web servers are not terminating their own TLS traffic and instead relying on their cloud. they are using a cloud such as Microsoft or Cloudflare. Well, that kind of means you have to put your trust in those companies instead of putting your trust in your telecom provider. And I'm not entirely convinced that those companies are less likely to leak at some point your data, be it by accident or selling it on purpose. But you reduce the blast radius because in the end before... anybody seeing your traffic could see that data. So anybody who could see your network traffic before was able to see those domains, those application protocols. Now it is only a handful of actors which do have some kind of image to upheld. Like people do pay them and do have trust in them that their data is not leaked by them, at least not on purpose. So I think overall it's a privacy win, but as long as not every web server is doing its own TLS, it does mean that you still at the very least put your trust in those very few amount of cloud companies that are available in the world right now.
Elizabeth (Plabayo): not leaked by them, at least on purpose. So I think overall it's a privacy win, but as long as not every web server is doing its own TLS, it does mean that you still at the very least put your trust in those very few amount of cloud companies that are available in the world right now. thank you for giving us a ⁓ check. ⁓ Okay. Thank you very much, Glen. I appreciate you give us the time to give a clarification on this subject, TLS ECH. And thank you to all our audience have a good week. Bye. Bye, ⁓ I didn't record.
Glen (Plabayo): Thank you for giving us a reality check. Okay, thank you very much, and I appreciate you give us a time to give a clarification on this subject TLS ECH. And thank you to our audience and have a good week. Bye. bye. ⁓
