David:
00:00:02
Hello, and welcome to Android bites powered by Asper I'm David Ruddick. And each week I'm joined by my cohost Michelle ramen diving deep into the world of Android this week. We're talking about apps and really app stores actually, which is a topic on Android that obviously has a history as long as the platform itself and allow Michelle to introduce our guests.
Mishaal:
00:00:23
This. Thanks David. So on today's show, we have invited Logan McGee. He's an independent developer. He's working on a project called Crescent. So thanks for joining us, Logan. Thank you for having me. Okay. And as David mentioned, we wanna talk about apps, more specifically, app stores and app distribution in a O S P. You don't have an app store by default. You have no way to actually get apps. And if you install a pure, fresh AOSP build, you gotta find and procure apps on your. I was actually reading an interesting article earlier today on L w N I don't know if you subscribed to that website or not, but they're basically like a newsletter that covers Linux. And they talked about way Android, which is a project we've actually talked about before on the show way back when we were actually doing the Twitter spaces. So way droid for those you don't know is a software project that enables running the entire Android user space and a container. It's a really interesting and neat project, especially if you have a Linux smartphone like the pine phone, and you wanna run Android apps because there's not. Native Linux optimized apps for smartphones, but it like any other non GMs, a O S P project has the same issue. Where do users get their apps from? Of course, if you're a privacy conscious user, you've probably heard of various app repositories, like fro you know how to procure APKs off of various websites. But as an ecosystem, this is a huge challenge because you can't assume that every user has the same technical capabilities as your average Linux developer, does. You gotta assume that most users are gonna want to, or they're gonna expect that there's an app store on their device and they're gonna click that and they're gonna download everything and everything's gonna be handled for them all easy peasy. That's why most OEMs major companies, they go through the process of licensing GMs, which if you, um, listen to our podcast, our previous podcast with the execs from awesome, you know, that's a pretty arduous task, but most OEMs go through it anyways, because it's absolutely necessary for their users, which consists of millions of average people to have access to the play store and Google play services, which ensures not only do they have access to millions of Android apps, but also the APIs that many of them depend. And because so many apps have GMs dependencies, it's pretty much impossible to talk about alternative app distribution models on Android without factoring in where GMs comes in. So if a device has GMs, then it's mostly just a matter of where do you source the apps from? Is it from Google play store or is it from somewhere else? You're not gonna have that many problems unless you're dealing with like paid apps. If a device doesn't have GMs, though, then you need to take extra steps to make sure that the app that you're trying to. We'll actually run on your device because a lot of apps, like I said, have dependencies on GMs, including things like safety net at station fireplace, cloud messaging for push notifications, the Google maps, API for mapping, et cetera. But thankfully at least since a S P is open and supports side loading, other apps stores that aren't bundled with the OS image, you know, it is possible to actually have alternative app repositories that don't really care about what's going on with GMs. Fro is one example that I mentioned the graph OS project. They recently released their own quote unquote app store, which is basically just like a website that has some of their apps that you need to set up their sandbox, Google play implementation. If you haven't listened to our graph OS episode, I'd recommend go doing that. So that's basically what an app store really boils down to. It's just a browsable catalog of apps that users can download install. So if you're thinking about designing your own app store and distributing apps on Android, you need to answer these questions. How do you connect users to that catalog of apps? How do you get legitimate apps onto the catalog while weeding out illegitimate ones? How do you securely and efficiently store and distribute apps to users? And how do you handle distributing app update? None of these are straightforward things to solve, but fortunately we have over a decade of history of looking at what Google play has done. So we can basically take a look and see what does it take to go about designing your own app store to distribute apps on Android. But before we dive into the meat of this discussion, I wanted to ask both of you for your overall thoughts and experiences of using other app. Such as you know, Wawa's app gallery, Samsung's galaxy store F Android, et cetera. Have you ever used an Android phone without GMs and exclusively had to get apps from outside of Google play? And what was it like? Have you used an Android phone with GMs, but still chose to get apps from outside of Google play? Or have you done a mix of both play and non Google play?
Logan:
00:04:37
Personally I used Graphos as recently, actually, so that doesn't have GMs installed by default. And so for a while, I was trying to get apps from other places like GitHub releases or fro, and honestly it was kind of painful to deal with. And so eventually I actually started using GMs through their sandbox, Google play compatibility layer, just because it is kind of a pain to get apps on Android. If you don't have GM.
David:
00:05:04
My history with various app stores. I mean the most prominent one and the one that I probably use the most would be actually the Amazon app store, which was such an interesting experiment. I mean, it's still, obviously very active, used by tens of millions of people who own Kindle. Fire tablets. But I remember the challenges Amazon dealt with in terms of simply like keeping apps up to date with their Google play counterparts fact that the Kindle fire devices ran an older OS version. I believe the Kindle fires initially launched based on ice cream sandwich initially back in the day, correct me if I'm wrong there, but it was a different time and it was still a time when legitimately you could. Somebody may be trying to take a step at Google play. And Amazon threw money at developers, especially game developers with all this in-app currency promotion stuff. And they threw money at advertising to try to get people excited about it. And it underscored to me that the usefulness of an app store. Is really dependent on what kind of walls are put up around that device. And in Amazon's case, it was, you could not easily install, play, or a GMs applications on those devices. And they've continuously, you know, I'm not saying they made it harder, but they fixed loopholes. And obviously Google doesn't want them to work very well either. And of course I've used Chinese phones from so many OEMs that don't ship a GMs. I even back in the day, received a review unit from a smartphone OEM. And I'm sure Michelle you've experienced this as well. It came with an application on it, specifically designed to display a WebApp view that linked to the files you needed to download the core GMs bits and then install them as though it was like they called it the Google installer, I think, on this device. So, yeah, it's just hard to imagine a mass market. App store that could de throw Google play at this point. And Amazon did try. So that then begs the question. The need for specialty app stores is probably there. And what kind of specialties will they have? And that's always been interesting to me, especially since I worked at Android police until Valette bought Android. Police is the sister side of AP came. Dot com. So I'm aware of things involving Android app distribution, to some
Mishaal:
00:07:24
extent. Yeah. So if you're going to go about designing an app store or app repository of like APK mirror, well, APK mirror is a special exception because it's not really designed to actually get people to use it as like an app store. It's just basically a APK backup posting site for the most part. But if you're going to design an app store that you actually want users to use, you need to answer the question. How are you gonna get it into the hands of users? Obviously on many devices out in the market, you don't have access to the OS image. So you don't have the ability to bundle your app store with the device. If you are a business and you want to have a small selection of apps that you want your employees to access, and you know, you're going to buy a whole bunch of GMs, Android devices. Then you could pretty much skip this entirety of this discussion and just use something like managed Google play, which is like a small island of apps that are only available for these devices. And those are the only apps they can use. But if you're not going to be doing that, if you want to ship a device without GMs, then you're going to find another way to get your app store onto devices. So you could either direct users to a website, which is the lowest hanging fruit, or you could create an app that connects to your own customer repository backend, or you could piggyback on existing solution, like say creating a repo and then using an fro client customized or not to connect to it rather than the main FOID repository. So if you set up a device to be enterprise managed from the get go, like through our SPER device management platform, then it's easy to push something. And install apps and update them. If you build the firmware yourself, which is something we can also do with foundation. I know, I know I gotta stop plugging here. We'll do that later with David, then the app store app could also be bundled with the OS image, and then you could also automate app installs and updates and have them be unintended updates. So otherwise, if you can't do any of those things, you need to have the user manually seek out your repository, which requires them knowing it exists in the first place, and then requires them to manually accept, installing and updating the apps in most. So, I guess this is the perfect time to segue into talking about the app store that you're creating. Logan. Can you give us a brief introduction to Crescent and why you're building it? How will users access it? And what are your plans to spread the word about it? Once it launch. Yes.
Logan:
00:09:40
So Crescent is, uh, an in progress app store. I'm currently building, that's mainly focused on security, privacy, and usability. It works kind of similar to the play store in that it allows developers to. Upload apps that they have compiled themselves, but it's different in that it doesn't require them to give up their signing keys. They don't have to upload an app bundle with their signing keys or have Google generate signing keys for them. So they generate the split AKs themselves locally and then upload them to Crescent. So Crescent still has all of the benefits of split APKs without the downsides of trust and security. it also does something kind of neat regarding app signatures. And so it has this signed piece of repository metadata. The store is really just a static web server. And so it uses that metadata to verify that apps downloaded are signed by who they're supposed to be. So even if the server was taken over by an attacker or something, uh, users who download apps will know for sure that the apps they download. Are, um, developed by who they say they are. Um, but it also focuses on privacy in some ways. Uh, it doesn't send any device identifiers. It doesn't need to, right now it just sends. Like the device architecture, uh, screen density and primary language, and it needs that to fetch split APKs. And another reason building a Crescent was for usability, right? I'm sure a lot of people have had problems with say, uh, on Android, you have to manually accept and install every single time you update an app. Whereas AC Crescent uses some modern APIs to allow you to update them. In the background without user interaction, but without compromising on security. So those are a couple of the things that Crescent does to ensure better user security and privacy or existing solutions.
Mishaal:
00:11:35
Thanks for the rundown Logan. So you brought up a lot of things that I kind of wanna touch on in a bit, especially like split it, BKS Android, app bundles, et cetera. Some of our listeners may not be familiar with these concepts, but before we actually dive into considerations that you need to make to get apps onto a store, I wanted to talk a bit about Android security model when it comes to applications. So every app that you install on an Android device comes in a file format called an Android package or APK file. An APK is a self-contained archive, which is essentially a derivative of Java archives or jar. But with a few Android specific additions to it. So the jar file format is itself based on the popular zip format, which, you know, if you ever use the desktop computer, zip it's that handy archive file format, where you store a bunch of files and compress it and it saves you space. And since jar is based on zip. APK by extension is as well, which is why you can open up APK files and see what's inside them using something like seven zip on windows or other platforms. So within the APK file, there's several key Android specific components. Namely the Android manifest, which contains the apps metadata, including what components it supports, what components it has, what permissions it requests and other various. Things that it needs like libraries that might require there's also the DAIC executable bike code files, which has the apps code for the JVM to interpret. Then there's the resource catalog, which is like, uh, resources.ar SC, and then the actual resources that they map to. Then there's unmanaged resources, which are also called assets and then optionally some native libraries. There's also a digital signature, which is inherited in part from the jar format. And that digital signature is generated whenever the developer signs, the application package using a signing key. So every app is installed onto a device, gets a unique, or has a unique package name. Developers can technically give whatever package name they want to their application, but Android will reject the insulation of. That has a package name matching an existing app that's installed on the device provided that app's digital signature doesn't match the existing apps this way. If you have one app installed a developer, can't just say, I want my app to have the same package name, and then try to install on top of that existing one. That's one of the key parts of Android security model when it comes to applications. And since the signing key used to generate an app's digital signature is generally assumed to be kept private and secure, you know, somewhere where only the developer who signs the app has access to or in, uh, Google play's case, Google themselves. It's generally assumed that only the app's original developers or Google will be able to push updates to existing apps that Android will. This is how Android securely updates apps. So I've done a bit of oversimplification here. APKs, don't actually use like the exact same digital signature implementation that you see in jar files. They've evolved over time to have like a proprietary implementation. There's various implementations called the APK signature schemes. And I wanted to ask you Logan, if you can explain to our listeners a bit about these, a signature schemes, like what does the signature scheme V3 bring to the table? For example? APK signature
Logan:
00:14:44
scheme three is a slight improvement over V2, which is what replaced the legacy jar signing you were just talking about and V3 allows for key rotation. So that means if a developer has a certain signing key and they might suspect it's compromised. For example, they can sign a new key with their old key and then make that new key, the new trusted key for that app. So they. Migrate away from old keys to avoid compromise without having to make the user uninstall and reinstall their app, which would've been the case before. There's a couple other newer ones too, like before, which is kind of different. It's not embedded in the APK itself. It's in a separate file and that can be used for this feature called Fs Verity. I don't wanna get too technical about it. Essentially what Fs Verity does. And what V four signatures allow is for the OS to cryptographically verify an, a PK every single time you launch it, or every single time it's read by the operating system, which is very helpful for reducing persistence. I believe the Google play store uses. Before signatures
Mishaal:
00:15:48
for this reason, he talked about the technical implementation, but one of the key features that it enables is the app incremental feature. So I think, uh, I forgot what the marketing name is for it. I think plays you download. Yeah. That's what it's called. So Android 12 Google introduced plays, you download, which basically lets you start playing a game while a lot of it's assets are still being. So because the signature is separate from the actual APK, it's able to be verified as the bulk of it's still being downloaded. So that's one of the features that V4 Napes and which is why it's not really mandatory to support yet. Whereas V3 is a much more significant improvement in terms of security. So I wanted to ask you next, are there any flaws in this Android security model when it comes to app signing and app distribution, and how does a Christian plan to deal with. Yes, sort of, I wouldn't necessarily
Logan:
00:16:39
call it a flaw. It's more of a design decision, but the way Android works with app signatures is the first time you install it, it pins that signing key. So say you install an app called secure chat or something. I install that. And then the Android operating system parses it and finds the signature and says, okay, this is the key that sign this app. And I will not accept any updates if they're not signed by this. And so that secures updates later on, right. But that doesn't help with the first install. If you install a malicious version of that app, then it'll updates that are legitimate. And presumably you'll be getting updates that are malicious and Android doesn't really do anything itself to solve this problem. It just leaves that up to the application layer. What a Crescent does to solve. This is when you upload an app to the store. Or whoever maintains their repository will sign their key to validate basically to the app. This is the signing key. We will accept for this app. The Crescent will only accept this signing key for this app. And so that means the first time you install it, you can know for certain that you are installing the legitimate version of that app, and it can't be replaced with a malicious version or something, even if the server was taken over, if that makes
Mishaal:
00:17:56
sense. Gotcha. Yeah. I mean, that is a big problem. Like figuring out how to install the legitimate version of an app, especially when there's so many clones and copycats out there. And so many different ways to side load apps or trick users into side loading apps that seem legitimate, but they're actually not. And once they have it, how do they know it's actually the real deal or not? And many times they don't. That is one, as you mentioned, it's not really a, a flaw, more like a intentional design decision, and there are solutions to mitigate that. But one of the, I'd say key failures of most app stores is actually getting developers on board. And I think this is one of the areas where why so many people don't even bother trying to make an app store. So kudos to you attempting to do that. For those of you who, uh, aren't familiar with this major chicken egg problem, you have an app store, right? What you need to do is you need to get apps on board. So you need to get developers to upload their. But the problem is developers. Aren't going to do that. If there aren't users who are using your app store, because after all, most developers want to make money off their applications. You know, they need to earn a living or sell a product or do something with it. So if there aren't users, then they aren't going to be making any money. And if there aren't any apps, there's not going to be any developers. So it's just a vicious cycle that kind of ensures that most burgeoning app stores don't ever really take off. And it's why even app stores created by the biggest tech companies on the planet, just pale in comparison to Google's play store. There are a couple of things that Google play does particularly really well and what others have attempted to copy in order to get developers, to actually upload their apps and share them there. So Google play, for example, they offer pretty robust tools around handling rollouts and managing your storefront. And then offering various ways to promote your apps within Google play outside of Google play. Other app stores try to offer lucrative revenue, sharing fees, much lower than Google, but still, if you're trying to launch an app store and you want to compete with Google play, that's a really hard sell. But if you are trying to just launch an app store that serves a very specific purpose, just a handful of users, you know what you're targeting, then you're setting much more realistic goals for yourself. But if you're trying to launch very open app store, that steps. App submissions by any developer around the world, and you're going to deal with the problem. And it's a problem that pretty much most app repositories deal with. And it's how do you moderate to ensure that bad actors aren't abusing your platform to spread their malicious status, sucking applications or straight up malware? and if you're dealing a Google play scale or like Samsung scale, et cetera, then the answer is, in my opinion, you can't, I think it's literally impossible for a company like Google to be able to thoroughly audit each and every single app that comes out their door. There's just too much to deal with. There are a few things they do, and they do well and some not so well to raise the barrier to entry, to ensure. There aren't as many malicious apps or privacy afflicting apps that are submitted to the app store. Like the most basic thing they do is they require a fee and, you know, personnel contact information. When you sign up for developer account. I don't think it's very effective, but I'm pretty sure that does stop a lot of bot submissions, Google play, and other app stores. They also perform static analysis of manifest files to look for potential violations of privacy. So for example, Google play is a lot of policies around what permissions and app can request and how they want to use them. So if it detects that your app is trying to request the SMS permission, then Google play will reject your. If you don't submit a forum and your app doesn't meet one of the pre-approved categories of apps that can request access to SMS. Google play also does automatic dynamic analysis of apps. So it analyzes risky code or behavior such as attempts to gain root access or disabled se Linux play protect. Does this on GMs devices? Google play also has human moderators to review app submissions and updates to look for violations. And finally, Google play also forces developers to declare their use of data permissions. Although more recently, if you heard the news, Google play is actually starting to hide the list of permissions in favor of the developers submitted data safety form. You've all heard the stories. I don't know if you've ever personally come across this or dealt with this before, but you've heard horror stories about developers dealing with Google play support or that their moderation practices are ineffective, inconsistent, or straight up incompetent. And from what I've ever heard, the other major app stores aren't really any better. Do you really think there's anything Google and others could do to improve the quality of apps and limit the number of fake or malicious apps? Or do you think they're pretty much just trying to prevent the dam from. I
Logan:
00:22:27
would say similar to what you said, it's very hard for Google to make the system. They have work at their massive scale. It's hard to get frankly, enough competent reviewers and enough analysis to really prove for sure whether an app is malicious or not. It's just not really possible at that scale. They are doing quite a few things to help though, like reviewing certain permissions or making sure they're only used for legitimate purposes. For example, there's a legacy file management permission that some apps request, which basically gives them access to manage all of your files. But sometimes they request that when they don't actually need access to all of 'em, they just need one at a time when they want to send something. And so that's part of what they can do. I don't really know how to solve the problem. On Google side, because honestly I don't upload apps to the play store. So I'm not as familiar with the problems there. David might have a little more insight into that.
David:
00:23:23
Over the years we received so many meals. I'm sure you did Michelle as well from developers who were having problems with basically the content review team or content moderation team on the Google play store. And there are so many reasons that they can reject to. and often of course, they're really terrible at actually explaining the rejection reason because usually the person rejecting you is probably somebody not making very much money. And they're working through as many of these approvals as they can in a given Workday. Their goal is volume. Not quality. No, you can't just make better people for this. Like you said, that's not really a scalable solution. And to me, Google is pulling the levers a can, which are restricting permission access, making it more stringent, or at least making the declarations very obvious when it's something really sensitive, which we're seeing now in Android 13. And we've seen in previous Android versions. And I think that we'll continue to see that trend and we'll continue to be upsetting to a particular audience of hin enthusiasts who don't like that. Google is clamping down this way. At the same time, it's hard to see this being a problem that can be solved at scale without immensely powerful machine learning and really rigorous controls. And that just seems to be the direction Google is heading. That's how most of their products work. So I feel like, yeah, the horror stories from the play store are really frustrating. You hear about devs who get delisted for totally bogus reasons because the reviewer either didn't understand something about a trademark or copyright complaint, or they thought there was something in the app that had changed, but they misunderstood and seeing that kind of stuff. And like seeing people's livelihoods be affected by that. That sucks at the same time. Michelle, what you said, I think is what I've heard as well, that there is no app store. That's good at this. You hear complaints about apple on the app store all the
Logan:
00:25:21
time. This is part of why I believe it's a good idea to leverage the Android security model to enforce this level of quality control. For example, the play store requires that all apps uploaded must target a certain SDK. And basically what that means is it's tested on a certain Android version and the OS enforces certain restrictions around that app that typically get stricter as the Android version goes up. And so that's one way they can enforce quality control. The other one. Permissions, which they do quite heavily. Part of it is informing users as well. So they understand, you know, when I'm giving an app access to this thing, it has apps don't have access to a whole lot of sensitive information without user consent. And so having to rely on the app store to communicate, that can be helpful. But part of it is also up to the user.
Mishaal:
00:26:09
I wanted to ask your thoughts on the growing number of permissions that seem to be purposely designed so that their actual enforcement mechanism isn't the Android OS itself, but rather Google play. And I know like there's a lot of permission that says, oh, app stores may enforce. The use of this permission, but in actuality, you know, it was designed around Google play. So I already mentioned SMS, but there's also call log. Those two permissions existed a while ago, but I'm seeing more and more permissions being added with each new OS version and their permission level might be normal, which means that they're granted an install time, but then. There's an asterisk in the documentation that says app stores may restrict this permission. So I wanted to ask you, do you think that's a good model for Google to take and do you actually see other app stores following through and actually enforcing these permissions? Like Google? I think it can
Logan:
00:26:58
be a good thing, but I think it's also a hard balancing act. You don't wanna ask the user for every little low level permission that they might understand. For example, there's the quarry, all packages permission, which basically allows an app to list and read all the apps that you have installed in your device, which you would think is pretty sensitive stuff. But the OS doesn't have a mechanism yet to enforce that permission. Even if you don't have the qual packages permission, there are other ways to get around it and list all the apps you have installed. And so the play store restricts this, but the OS doesn't that specific example I think could be enforced by the OS later, but due to how Android works currently, it's hard to do that without breaking certain app functionality. Another example is the internet permission a long time ago, it used to be a dangerous permission, which means the user had to opt in and allow an app network access. But Google later decided that would just cause permission, fatigue. And so if the user is being requested, all these permissions that they have to give almost every single app, then they'll just start blindly accepting without really looking into what they are giving an app access to. And so it's a, it's a hard balancing act. It's definitely better for the OS to enforce a permission, but sometimes it's hard to, for very low level things that the user might not understand what they're needed for or what
Mishaal:
00:28:16
they. Yeah, we've talked about this balancing act that Google has to do before. And you know, when you're developing an operating system that serves billions of users, tens of thousands of developers around the world and hundreds of millions of regular users. It's complicated. There's so many considerations that they have to deal with that you don't really have to care about. If you're only designing a platform that's for a specific use case, if you are able to own the OS image, like if you're developing your own AOS P build for your own devices, and therefore you're able to bundle your own app store. Then you can bypass a lot of Google's or Android security restrictions on third party app installers. And this is one of the things that Google does for the benefit of the overall ecosystem. But if you can develop your own operating system based on a O S P, then you don't have to really consider the security implications of that because you can whitelist your own app store. You can grant it all the remissions that Google play store might have access. Without this permission, though, if you're a regular third party app store running on a Android device that you can buy off the shelf. Then you can't install apps on your own without holding the request, install, packages, permission, and that's the permission that's shown to the user it's install, unknown apps. What it does is it allows an app to send an intent to the system package installer, or to use the session based package installer, API, to request users, to grant approval for app installs, which is why the permission name has request in its name. So when you use. API, the user has to actually grant their approval for each and every installation and subsequent update. And that can be a real pain when they wanna do batch installations or batch updates of apps. So privileged system apps like Google play store on all GMs devices or whatever apps store you pre-B bundle on the device can request the install, packages and delete packages. Permiss. To be able to fully unattended, installs and uninstalls on first installs and all subsequent app updates, but that's not attainable by user apps unless, you know, the user app is able to gain route access, which is not possible unless the boot load is unlocked on most devices. There is. However, one way that third party app stores can do unattended updates. And you did mention this before Logan, when you were talking about a Crescent and it's through a new API that was introduced in Android. So I wanted to ask you, can you explain what this API, this new feature in Android 12 does and like, how does it exactly. Yes. So
Logan:
00:30:44
Crescent does use that API, Android 12 introduced this API that allows app stores to update apps without requesting user permission all the time. And the way it works is with this concept called the installer of record. So say AC Crescent, installs, secure chat again. Crescent is then marked as the installer of record for secure chat. When it first installs it, it brings up the normal package prompt. Do you want to install this app by the OS, but once you grant consent, then in this case, Crescent is allowed to update secure chat automatically without asking the user. And this works in tandem with other Android security features like downgrade protection and signature pining, making sure that it's signed by the same developer and that you can't downgrade it. So that's what a Crescent does. The privileged permissions you were talking about, like you said, they don't ask for user permission, but what's nice about the Android 12 unattended update permission is that's not privileged. You don't have to whitelist an app in your operating system. You can use it on any Android 12 and up device. That's basically how that API works.
Mishaal:
00:31:47
And, uh, fun fact, this whole API, this feature directly came about as a result of epic lawsuit against Google, as well as all the antitrust scrutiny that Google has been facing. If it weren't for all that scrutiny Google's been having lately, then we wouldn't have this feature or the reduction in the revenue sharing fee. So. Developers. It is really important that you keep an eye out and you watch what's happening with these lawsuits and these regulations around the world, because they're having a direct impact on the operating system and the ecosystem as a whole. They're really important, even though there's a lot of legalese and a lot of politicing back and forth, but yeah, I definitely keep an eye out on all that. If you're developing an app store, I'm sure you probably don't really consider the politics behind it. If you're just actually a developer working on the backend. But what you do care about is how do you deal with the bandwidth and storage needs of serving apps to millions of users and actually storing those apps on a server when you're at Google scale, this is a very, very important discussion to have because everybody optimization. Both on the service side to reduce storage and bandwidth and on the delivery end to improve, download speeds for users and thus user retention. There was a story I remember, I think it was in 2017. When in Google intern introduced the more efficient compression method called Broley. It was used for both OS system updates and also APK files themselves. And I was surprised by the number that Google shared. It saves them about 1.5 petabytes of data per day, just by switching to a more efficient compression. Just being able to switch one little one correction method, apply that to all the apps that they apps that they serve to save so much data and thus reduce bandwidth costs on both. That's something that, you know, it's kind of like a backend thing that most users will never most users and developers will never have to consider. But one thing that I'm sure most developers are familiar with, and at least a lot of users are, have heard of is what Logan brought up before the whole split APKs and or app bundles feature and how it's delivered to users through the Google play feature delivery feature, which used to be called dynamic delivery. This is both a really neat feature and also pretty contr. Which lo alluded to earlier and I'll get into in a second. So for those of you who don't know the idea behind the split APK and, and or at bundle is that most users don't need to have installed a monolithic APK file that contains every asset code chunk and language resource. Instead, you only need the bare minimum to get the app and its core features starting all that's contained within the base PK. And then Google play can push the remaining split APK files that contain the assets, features and language, resources that are actually needed by the user. So for example, if your assistant language set to English, then you'll get the split APK file that contains English language strings. You won't get the French language strings cuz you don't need that. So this concept is actually quite old. It's older than thera. Split a BKS were introduced with Android 5.0, but a lot of developers didn't really make use of them until the Android app bundle requirement came into effect in August, 2021. So starting August, 2021, every new app that's uploaded to Google play has to use the Android app bundle format. The bundle format was introduced with Android. And to explain a bit about what it is. It's also a valid zip file, just like APKs. So within this AAB file, you have all the resources, assets, and libraries that are needed to generate multiple configurations of the app. And it's structured in a way so that you have all of these segments kind of a segment apart from each other, you have a base module, you have dynamic feature packs and you have asset packs that this tool called bundle tool uses to generate specific split APK configurations. So what developers do is they generate the Android app bundle file in Android studio. They upload that to Google play and then Google play generates split APK files on demand when the user downloads the app. So this has a potential to save users, a lot of storage space on APKs and also thus reduce how much bandwidth is needed to download. So while the idea behind this is sound, you've mentioned Logan, that there are some issues with how it's implemented. Can you explain what some of those concerns are and what Google's done to address them? If anything. There are a few
Logan:
00:36:01
valid concerns with not necessarily app bundles themselves, but the implications of how they're used. So Google play uses this feature called play app signing. And so you upload your app bundle and then Google will sign and generate the APKs themselves. As we talked about earlier, the Android operating system. Will pin the signing keys of those AP case. But in this case, it's Google, who's signing them. And so theoretically, Google could modify the APK and deliver a malicious update either intentionally or after being compromised. And so that's one big concern with, uh, app bundles. They've mitigated this somewhat with this feature called code transparency. How that works is a developer can generate this separate signature file and upload it to the play store. And users can download that and make sure that the APK that they get was generated from the app bundle that the developer made, but it's not a perfect solution for one, the developer has to choose to generate this and upload it. Then the user has to download it themselves and verify it themselves, which already cuts out a lot of people. Most people aren't gonna try to do that for every single app. They. Another problem is it doesn't verify the entire file code transparency doesn't verify the entire APK. And so things like resources could still be modified, which could be used to change the UI or something and trick the user into doing something that they don't intend to. Those are a couple of the concerns. The way Crescent works around this is that developers upload the split APKs themselves. The tool that Google uses to generate split APKs is freely available and open source, and you can download it and use it yourself. And so AC Crescent allows developers to do that and just upload the split APKs. And so they can still be used without the trust issues with app bundles. To be fair to Google. They do a very good job of this and they're handling these keys. They have a whole white paper on their infrastructure, and there are legitimate reasons for it. For example, I didn't learn this until recently, but developers often lose their signing keys. Once you lose your signing. Key, the OS won't accept updates with a different signing key. If you didn't rotate it right. And so if you lose your signing key, all of your users are left in the dark without an update. Whereas with Google play, if you lose your upload key, which you have to use to authenticate yourself, Google still has the signing key. So they can just generate a new one for your account and you can still deliver updates as normal. So there are benefits to it, but there are definite security and trust drawback.
Mishaal:
00:38:26
What about the challenges this whole play app signing introduces when it comes to distributing apps across multiple app stores, like since Google is signing the apps that get delivered to your device, right? If you were to take that app, And you were to try to say you had Samsung galaxy app store installed as well. And you tried to install an update that was already available through Samsung's galaxy app store for the same exact app you wouldn't be able to because the app that's uploaded to Samsung store is signed with a different key than the one that's uploaded to Google play. Can you talk a bit about that? That's
Logan:
00:38:59
partially a store concern and partially a developer concern. Generally speaking, if you have an app that's being signed with different keys, say for Google play and say, you're really seeking it on GI Huber, your website or something. If you're not using the same key,
Mishaal:
00:39:13
you should
Logan:
00:39:13
change the app ID, which is basically just an internal identifier to make an app uniquely identifiable to the operating system. I know I keep bringing it up, but for example, Graphos does this with their camera. There's a version. They sign themselves and the version that Google signs. And so they have different app IDs, so they can both be installed at the same time without conflicting. So that's generally what developers should do. If they're uploading to multiple stores that are signing with different keys, they should change the app ID. So they don't conflict. But many aren't aware of that. And so that does cause a challenge, especially for apps that are say on both Google play and F Android since F Android signs with their own keys, but doesn't change the app ID. They conflict with Google play versions, and you can't have both installed at the same time or use one to update to the other.
Mishaal:
00:39:57
I've talked about this before, but the Android app bundle, there's a lot of clear and obvious benefits. It brings to both Google and to users when it comes to storage and bandwidth. But it's hard not to have like a cynical view. Oh, how does this lock users more into the Google play ecosystem? Like, is it a coincidence that it has these benefits? You tell me because I'm sure many people will think that there's no coincidence that, you know, it has this benefit of doing that. But to be fair, there are some rather unique things that can be done with Android app bundles that you can't do easily with the monolithic KPK. So, for example, at the beginning of this year, Google announced the new feature that Google play will be getting called app archiving. So basically what this does is that Google will be using the bundle that delivers upload to generate a new split APK file called an archived APK. So this archived APK is basically a. That holds nothing but code to launch the app store to redownload the full app. So this stub basically has nothing in it. It's just an empty app. That's signed by the same key because Google signed it. It has the app icon. Because it's actually still installed on your device, but it has like no resources, no nothing. It's basically like a multi kilobyte file instead of like the tens, potentially hundreds of megabytes file that the original app was using up on your device. So by introducing this app, archiving feature users will be able to archive an app and reduce its storage requirements without actually uninstalling the. And if you're interested in learning more about that, by the way, I did a deep dive on the Android buys column. I, I think that's quite an interesting feature. And I'm really curious to see if other app stores actually replicate this. Like Google did design it in such a way that other app stores could replicate it. But again, like the whole permissions data, the permission safety form stuff, I don't really know if any alarm stores actually follow through and do what Google's doing and all, everything they're doing. But before we close off this discussion, cuz we're getting close to time. I wanted to bring up a prime example. Of a market where non Google play app distribution is successful. And that's China, China, for those of you who don't know, does not allow access to any Google services, the fact that Chinese Android users are using non Google play app repository is more out of necessity than anything else. So from what I've heard, speaking to Chinese users and reading online is that Chinese users source APKs from a multitude of app stores, a multitude of websites, many third party app posting websites, and more like, it's a huge mess of where they get apps from. It's it's really decentralized and that's actually, you know, Kind of the model that desktop users have been dealing with for a long time, the decentralization, but without that centralized, Google play and GMs providing those APIs, the Chinese market kind of struggles to deal with things like push notifications, whether like 13 different push notification APIs, which requires like, if you have 13 different apps, all using different push notification implement. Then that means you have 13 different apps having these foreground services, constantly waiting for push notifications to be pushed to your device and that results in a whole bunch of apps waking up at various random times and then destroying your battery life. So if you're wondering why a lot of Chinese phone brands seem to have these really, really, really aggressive battery saving measure. It's a result of this because there's no unified centralized, Google play services. Like there is outside of China, knowing what you know about how the situation is in China. Do you prefer the centralized model we have now with Google? Or do you think a fully decentralized approach would be better or maybe something in between. At
Logan:
00:43:31
a high level, I would say the decentralized approach could be better, but it won't be. And the reason I say that is because when you have multiple parties providing updates to their own mechanisms, a lot of them aren't doing it well or doing it right. Like we've seen how a Google play store, although this largely due to GMs the place where actually does security evaluations, they do app distribution. Very, very well. It's hard to make sure that those same restrictions, uh, security and privacy are being enforced across all these different mechanisms. And most app developers won't make sure updates are delivered securely and timely and automatically. And so that's why I would say the decentralized approach could be better, but it won't be say, if you have an app that updates itself, that's like the ideal situation you can have, but most developers won't want to do that. It's extra work and it's hard to do. Decentralization in the sense of say one app updating from multiple different places actually violates Android security model there's security model mandates that one app store is just one app source. It downloads apps from one place, and there are problems if you try to change that. And then there's also a problem. If you're downloading from multiple servers with the same app, say like, does you can't use certain security features like TLS certificate pinning that would. How I would evaluate the overall situation. Centralization is unfortunate in some ways, but I think if the centralized party knows how to do things well, I would say Google does, then the situation would be much better. In that case, you don't have to trust as many parties to do the right thing. Right.
Mishaal:
00:45:08
And fortunately, you know, Android isn't as locked down as iOS, where you have no choice between centralization and decentralized. It's all centralized all the time. At least with Android. 99% is through Google play, but that 1% outside of China, at least you have the freedom to choose what apps and what app stores you're using. And of course, since a S P itself is open. You also have the freedom to customize a O S P. And then build in your own app distribution ecosystem to that distribution and to the devices that you are deploying this build on, which of course is really important because a S P as I said before, does not come with an app store by default. So you have to do something custom. If you're going to build a O S P without GMs.
David:
00:45:53
App distribution is actually something that is very near and dear toper, you might even say it's kind of the core of our business in a lot of ways, because what Esper does every day is deliver applications and updates to applications, to dedicated devices, which are things like kiosks business tablets, things that typically don't have Google mobile services to begin with. And wouldn't have a reason to other than having a way to. Apps, but obviously that's not a good solution for a lot of reasons, for a lot of different kinds of devices, some of which are cost, some of which are time, some of which are engineering effort. I mean, all of those things generally are true. Being a GMs partner versus using AOS. P. So if you're using a O S P to build a kiosk or to build a point of sale or to build display, signage, whatever, it might be a medical device, perhaps home exercise equipment, we're literally on a climbing machine. You need a way to deploy updated APKs to that Android device. And if you don't have the play store, that's a little easier said than done at scale, easy to do with a few dozen. And once you get to a few hundred, few thousand tens of thousands of devices, That gets pretty hard. And so what Asper provides along with our robust device management tools is a really, really sophisticated way to deploy your apps, stage updates, group them choose when and where they go out, set conditions for success. You. And deploy your app in a way that is highly observable and where you know exactly where it's going. You know, every device that's received it, every device that hasn't, you're getting exception reports when something goes wrong. So if you've been wondering how you can deploy applications to Android devices in a really specific, targeted, highly reliable way with an enterprise grade QoS, come talk to us at SPER we're at SPER dot. I.
Mishaal:
00:47:42
Thanks David, for the obligatory per plug in the Android bites podcast. And of course, here's the obligatory closing segment of the podcast where I give our guests the opportunity to explain where can you follow him online? So Logan, where can people find you and your work? You can find
Logan:
00:48:00
me on Twitter and on GitHub as Berry ma that's my handle on most other places. And then Crescent is on github.com/crescent, ACC, R E S C E
Mishaal:
00:48:13
NT. And you can find David and I on ESER that's blog.sper.io. And thank you all again for listening to another episode of Android bites.
00:00:02
Hello, and welcome to Android bites powered by Asper I'm David Ruddick. And each week I'm joined by my cohost Michelle ramen diving deep into the world of Android this week. We're talking about apps and really app stores actually, which is a topic on Android that obviously has a history as long as the platform itself and allow Michelle to introduce our guests.
Mishaal:
00:00:23
This. Thanks David. So on today's show, we have invited Logan McGee. He's an independent developer. He's working on a project called Crescent. So thanks for joining us, Logan. Thank you for having me. Okay. And as David mentioned, we wanna talk about apps, more specifically, app stores and app distribution in a O S P. You don't have an app store by default. You have no way to actually get apps. And if you install a pure, fresh AOSP build, you gotta find and procure apps on your. I was actually reading an interesting article earlier today on L w N I don't know if you subscribed to that website or not, but they're basically like a newsletter that covers Linux. And they talked about way Android, which is a project we've actually talked about before on the show way back when we were actually doing the Twitter spaces. So way droid for those you don't know is a software project that enables running the entire Android user space and a container. It's a really interesting and neat project, especially if you have a Linux smartphone like the pine phone, and you wanna run Android apps because there's not. Native Linux optimized apps for smartphones, but it like any other non GMs, a O S P project has the same issue. Where do users get their apps from? Of course, if you're a privacy conscious user, you've probably heard of various app repositories, like fro you know how to procure APKs off of various websites. But as an ecosystem, this is a huge challenge because you can't assume that every user has the same technical capabilities as your average Linux developer, does. You gotta assume that most users are gonna want to, or they're gonna expect that there's an app store on their device and they're gonna click that and they're gonna download everything and everything's gonna be handled for them all easy peasy. That's why most OEMs major companies, they go through the process of licensing GMs, which if you, um, listen to our podcast, our previous podcast with the execs from awesome, you know, that's a pretty arduous task, but most OEMs go through it anyways, because it's absolutely necessary for their users, which consists of millions of average people to have access to the play store and Google play services, which ensures not only do they have access to millions of Android apps, but also the APIs that many of them depend. And because so many apps have GMs dependencies, it's pretty much impossible to talk about alternative app distribution models on Android without factoring in where GMs comes in. So if a device has GMs, then it's mostly just a matter of where do you source the apps from? Is it from Google play store or is it from somewhere else? You're not gonna have that many problems unless you're dealing with like paid apps. If a device doesn't have GMs, though, then you need to take extra steps to make sure that the app that you're trying to. We'll actually run on your device because a lot of apps, like I said, have dependencies on GMs, including things like safety net at station fireplace, cloud messaging for push notifications, the Google maps, API for mapping, et cetera. But thankfully at least since a S P is open and supports side loading, other apps stores that aren't bundled with the OS image, you know, it is possible to actually have alternative app repositories that don't really care about what's going on with GMs. Fro is one example that I mentioned the graph OS project. They recently released their own quote unquote app store, which is basically just like a website that has some of their apps that you need to set up their sandbox, Google play implementation. If you haven't listened to our graph OS episode, I'd recommend go doing that. So that's basically what an app store really boils down to. It's just a browsable catalog of apps that users can download install. So if you're thinking about designing your own app store and distributing apps on Android, you need to answer these questions. How do you connect users to that catalog of apps? How do you get legitimate apps onto the catalog while weeding out illegitimate ones? How do you securely and efficiently store and distribute apps to users? And how do you handle distributing app update? None of these are straightforward things to solve, but fortunately we have over a decade of history of looking at what Google play has done. So we can basically take a look and see what does it take to go about designing your own app store to distribute apps on Android. But before we dive into the meat of this discussion, I wanted to ask both of you for your overall thoughts and experiences of using other app. Such as you know, Wawa's app gallery, Samsung's galaxy store F Android, et cetera. Have you ever used an Android phone without GMs and exclusively had to get apps from outside of Google play? And what was it like? Have you used an Android phone with GMs, but still chose to get apps from outside of Google play? Or have you done a mix of both play and non Google play?
Logan:
00:04:37
Personally I used Graphos as recently, actually, so that doesn't have GMs installed by default. And so for a while, I was trying to get apps from other places like GitHub releases or fro, and honestly it was kind of painful to deal with. And so eventually I actually started using GMs through their sandbox, Google play compatibility layer, just because it is kind of a pain to get apps on Android. If you don't have GM.
David:
00:05:04
My history with various app stores. I mean the most prominent one and the one that I probably use the most would be actually the Amazon app store, which was such an interesting experiment. I mean, it's still, obviously very active, used by tens of millions of people who own Kindle. Fire tablets. But I remember the challenges Amazon dealt with in terms of simply like keeping apps up to date with their Google play counterparts fact that the Kindle fire devices ran an older OS version. I believe the Kindle fires initially launched based on ice cream sandwich initially back in the day, correct me if I'm wrong there, but it was a different time and it was still a time when legitimately you could. Somebody may be trying to take a step at Google play. And Amazon threw money at developers, especially game developers with all this in-app currency promotion stuff. And they threw money at advertising to try to get people excited about it. And it underscored to me that the usefulness of an app store. Is really dependent on what kind of walls are put up around that device. And in Amazon's case, it was, you could not easily install, play, or a GMs applications on those devices. And they've continuously, you know, I'm not saying they made it harder, but they fixed loopholes. And obviously Google doesn't want them to work very well either. And of course I've used Chinese phones from so many OEMs that don't ship a GMs. I even back in the day, received a review unit from a smartphone OEM. And I'm sure Michelle you've experienced this as well. It came with an application on it, specifically designed to display a WebApp view that linked to the files you needed to download the core GMs bits and then install them as though it was like they called it the Google installer, I think, on this device. So, yeah, it's just hard to imagine a mass market. App store that could de throw Google play at this point. And Amazon did try. So that then begs the question. The need for specialty app stores is probably there. And what kind of specialties will they have? And that's always been interesting to me, especially since I worked at Android police until Valette bought Android. Police is the sister side of AP came. Dot com. So I'm aware of things involving Android app distribution, to some
Mishaal:
00:07:24
extent. Yeah. So if you're going to go about designing an app store or app repository of like APK mirror, well, APK mirror is a special exception because it's not really designed to actually get people to use it as like an app store. It's just basically a APK backup posting site for the most part. But if you're going to design an app store that you actually want users to use, you need to answer the question. How are you gonna get it into the hands of users? Obviously on many devices out in the market, you don't have access to the OS image. So you don't have the ability to bundle your app store with the device. If you are a business and you want to have a small selection of apps that you want your employees to access, and you know, you're going to buy a whole bunch of GMs, Android devices. Then you could pretty much skip this entirety of this discussion and just use something like managed Google play, which is like a small island of apps that are only available for these devices. And those are the only apps they can use. But if you're not going to be doing that, if you want to ship a device without GMs, then you're going to find another way to get your app store onto devices. So you could either direct users to a website, which is the lowest hanging fruit, or you could create an app that connects to your own customer repository backend, or you could piggyback on existing solution, like say creating a repo and then using an fro client customized or not to connect to it rather than the main FOID repository. So if you set up a device to be enterprise managed from the get go, like through our SPER device management platform, then it's easy to push something. And install apps and update them. If you build the firmware yourself, which is something we can also do with foundation. I know, I know I gotta stop plugging here. We'll do that later with David, then the app store app could also be bundled with the OS image, and then you could also automate app installs and updates and have them be unintended updates. So otherwise, if you can't do any of those things, you need to have the user manually seek out your repository, which requires them knowing it exists in the first place, and then requires them to manually accept, installing and updating the apps in most. So, I guess this is the perfect time to segue into talking about the app store that you're creating. Logan. Can you give us a brief introduction to Crescent and why you're building it? How will users access it? And what are your plans to spread the word about it? Once it launch. Yes.
Logan:
00:09:40
So Crescent is, uh, an in progress app store. I'm currently building, that's mainly focused on security, privacy, and usability. It works kind of similar to the play store in that it allows developers to. Upload apps that they have compiled themselves, but it's different in that it doesn't require them to give up their signing keys. They don't have to upload an app bundle with their signing keys or have Google generate signing keys for them. So they generate the split AKs themselves locally and then upload them to Crescent. So Crescent still has all of the benefits of split APKs without the downsides of trust and security. it also does something kind of neat regarding app signatures. And so it has this signed piece of repository metadata. The store is really just a static web server. And so it uses that metadata to verify that apps downloaded are signed by who they're supposed to be. So even if the server was taken over by an attacker or something, uh, users who download apps will know for sure that the apps they download. Are, um, developed by who they say they are. Um, but it also focuses on privacy in some ways. Uh, it doesn't send any device identifiers. It doesn't need to, right now it just sends. Like the device architecture, uh, screen density and primary language, and it needs that to fetch split APKs. And another reason building a Crescent was for usability, right? I'm sure a lot of people have had problems with say, uh, on Android, you have to manually accept and install every single time you update an app. Whereas AC Crescent uses some modern APIs to allow you to update them. In the background without user interaction, but without compromising on security. So those are a couple of the things that Crescent does to ensure better user security and privacy or existing solutions.
Mishaal:
00:11:35
Thanks for the rundown Logan. So you brought up a lot of things that I kind of wanna touch on in a bit, especially like split it, BKS Android, app bundles, et cetera. Some of our listeners may not be familiar with these concepts, but before we actually dive into considerations that you need to make to get apps onto a store, I wanted to talk a bit about Android security model when it comes to applications. So every app that you install on an Android device comes in a file format called an Android package or APK file. An APK is a self-contained archive, which is essentially a derivative of Java archives or jar. But with a few Android specific additions to it. So the jar file format is itself based on the popular zip format, which, you know, if you ever use the desktop computer, zip it's that handy archive file format, where you store a bunch of files and compress it and it saves you space. And since jar is based on zip. APK by extension is as well, which is why you can open up APK files and see what's inside them using something like seven zip on windows or other platforms. So within the APK file, there's several key Android specific components. Namely the Android manifest, which contains the apps metadata, including what components it supports, what components it has, what permissions it requests and other various. Things that it needs like libraries that might require there's also the DAIC executable bike code files, which has the apps code for the JVM to interpret. Then there's the resource catalog, which is like, uh, resources.ar SC, and then the actual resources that they map to. Then there's unmanaged resources, which are also called assets and then optionally some native libraries. There's also a digital signature, which is inherited in part from the jar format. And that digital signature is generated whenever the developer signs, the application package using a signing key. So every app is installed onto a device, gets a unique, or has a unique package name. Developers can technically give whatever package name they want to their application, but Android will reject the insulation of. That has a package name matching an existing app that's installed on the device provided that app's digital signature doesn't match the existing apps this way. If you have one app installed a developer, can't just say, I want my app to have the same package name, and then try to install on top of that existing one. That's one of the key parts of Android security model when it comes to applications. And since the signing key used to generate an app's digital signature is generally assumed to be kept private and secure, you know, somewhere where only the developer who signs the app has access to or in, uh, Google play's case, Google themselves. It's generally assumed that only the app's original developers or Google will be able to push updates to existing apps that Android will. This is how Android securely updates apps. So I've done a bit of oversimplification here. APKs, don't actually use like the exact same digital signature implementation that you see in jar files. They've evolved over time to have like a proprietary implementation. There's various implementations called the APK signature schemes. And I wanted to ask you Logan, if you can explain to our listeners a bit about these, a signature schemes, like what does the signature scheme V3 bring to the table? For example? APK signature
Logan:
00:14:44
scheme three is a slight improvement over V2, which is what replaced the legacy jar signing you were just talking about and V3 allows for key rotation. So that means if a developer has a certain signing key and they might suspect it's compromised. For example, they can sign a new key with their old key and then make that new key, the new trusted key for that app. So they. Migrate away from old keys to avoid compromise without having to make the user uninstall and reinstall their app, which would've been the case before. There's a couple other newer ones too, like before, which is kind of different. It's not embedded in the APK itself. It's in a separate file and that can be used for this feature called Fs Verity. I don't wanna get too technical about it. Essentially what Fs Verity does. And what V four signatures allow is for the OS to cryptographically verify an, a PK every single time you launch it, or every single time it's read by the operating system, which is very helpful for reducing persistence. I believe the Google play store uses. Before signatures
Mishaal:
00:15:48
for this reason, he talked about the technical implementation, but one of the key features that it enables is the app incremental feature. So I think, uh, I forgot what the marketing name is for it. I think plays you download. Yeah. That's what it's called. So Android 12 Google introduced plays, you download, which basically lets you start playing a game while a lot of it's assets are still being. So because the signature is separate from the actual APK, it's able to be verified as the bulk of it's still being downloaded. So that's one of the features that V4 Napes and which is why it's not really mandatory to support yet. Whereas V3 is a much more significant improvement in terms of security. So I wanted to ask you next, are there any flaws in this Android security model when it comes to app signing and app distribution, and how does a Christian plan to deal with. Yes, sort of, I wouldn't necessarily
Logan:
00:16:39
call it a flaw. It's more of a design decision, but the way Android works with app signatures is the first time you install it, it pins that signing key. So say you install an app called secure chat or something. I install that. And then the Android operating system parses it and finds the signature and says, okay, this is the key that sign this app. And I will not accept any updates if they're not signed by this. And so that secures updates later on, right. But that doesn't help with the first install. If you install a malicious version of that app, then it'll updates that are legitimate. And presumably you'll be getting updates that are malicious and Android doesn't really do anything itself to solve this problem. It just leaves that up to the application layer. What a Crescent does to solve. This is when you upload an app to the store. Or whoever maintains their repository will sign their key to validate basically to the app. This is the signing key. We will accept for this app. The Crescent will only accept this signing key for this app. And so that means the first time you install it, you can know for certain that you are installing the legitimate version of that app, and it can't be replaced with a malicious version or something, even if the server was taken over, if that makes
Mishaal:
00:17:56
sense. Gotcha. Yeah. I mean, that is a big problem. Like figuring out how to install the legitimate version of an app, especially when there's so many clones and copycats out there. And so many different ways to side load apps or trick users into side loading apps that seem legitimate, but they're actually not. And once they have it, how do they know it's actually the real deal or not? And many times they don't. That is one, as you mentioned, it's not really a, a flaw, more like a intentional design decision, and there are solutions to mitigate that. But one of the, I'd say key failures of most app stores is actually getting developers on board. And I think this is one of the areas where why so many people don't even bother trying to make an app store. So kudos to you attempting to do that. For those of you who, uh, aren't familiar with this major chicken egg problem, you have an app store, right? What you need to do is you need to get apps on board. So you need to get developers to upload their. But the problem is developers. Aren't going to do that. If there aren't users who are using your app store, because after all, most developers want to make money off their applications. You know, they need to earn a living or sell a product or do something with it. So if there aren't users, then they aren't going to be making any money. And if there aren't any apps, there's not going to be any developers. So it's just a vicious cycle that kind of ensures that most burgeoning app stores don't ever really take off. And it's why even app stores created by the biggest tech companies on the planet, just pale in comparison to Google's play store. There are a couple of things that Google play does particularly really well and what others have attempted to copy in order to get developers, to actually upload their apps and share them there. So Google play, for example, they offer pretty robust tools around handling rollouts and managing your storefront. And then offering various ways to promote your apps within Google play outside of Google play. Other app stores try to offer lucrative revenue, sharing fees, much lower than Google, but still, if you're trying to launch an app store and you want to compete with Google play, that's a really hard sell. But if you are trying to just launch an app store that serves a very specific purpose, just a handful of users, you know what you're targeting, then you're setting much more realistic goals for yourself. But if you're trying to launch very open app store, that steps. App submissions by any developer around the world, and you're going to deal with the problem. And it's a problem that pretty much most app repositories deal with. And it's how do you moderate to ensure that bad actors aren't abusing your platform to spread their malicious status, sucking applications or straight up malware? and if you're dealing a Google play scale or like Samsung scale, et cetera, then the answer is, in my opinion, you can't, I think it's literally impossible for a company like Google to be able to thoroughly audit each and every single app that comes out their door. There's just too much to deal with. There are a few things they do, and they do well and some not so well to raise the barrier to entry, to ensure. There aren't as many malicious apps or privacy afflicting apps that are submitted to the app store. Like the most basic thing they do is they require a fee and, you know, personnel contact information. When you sign up for developer account. I don't think it's very effective, but I'm pretty sure that does stop a lot of bot submissions, Google play, and other app stores. They also perform static analysis of manifest files to look for potential violations of privacy. So for example, Google play is a lot of policies around what permissions and app can request and how they want to use them. So if it detects that your app is trying to request the SMS permission, then Google play will reject your. If you don't submit a forum and your app doesn't meet one of the pre-approved categories of apps that can request access to SMS. Google play also does automatic dynamic analysis of apps. So it analyzes risky code or behavior such as attempts to gain root access or disabled se Linux play protect. Does this on GMs devices? Google play also has human moderators to review app submissions and updates to look for violations. And finally, Google play also forces developers to declare their use of data permissions. Although more recently, if you heard the news, Google play is actually starting to hide the list of permissions in favor of the developers submitted data safety form. You've all heard the stories. I don't know if you've ever personally come across this or dealt with this before, but you've heard horror stories about developers dealing with Google play support or that their moderation practices are ineffective, inconsistent, or straight up incompetent. And from what I've ever heard, the other major app stores aren't really any better. Do you really think there's anything Google and others could do to improve the quality of apps and limit the number of fake or malicious apps? Or do you think they're pretty much just trying to prevent the dam from. I
Logan:
00:22:27
would say similar to what you said, it's very hard for Google to make the system. They have work at their massive scale. It's hard to get frankly, enough competent reviewers and enough analysis to really prove for sure whether an app is malicious or not. It's just not really possible at that scale. They are doing quite a few things to help though, like reviewing certain permissions or making sure they're only used for legitimate purposes. For example, there's a legacy file management permission that some apps request, which basically gives them access to manage all of your files. But sometimes they request that when they don't actually need access to all of 'em, they just need one at a time when they want to send something. And so that's part of what they can do. I don't really know how to solve the problem. On Google side, because honestly I don't upload apps to the play store. So I'm not as familiar with the problems there. David might have a little more insight into that.
David:
00:23:23
Over the years we received so many meals. I'm sure you did Michelle as well from developers who were having problems with basically the content review team or content moderation team on the Google play store. And there are so many reasons that they can reject to. and often of course, they're really terrible at actually explaining the rejection reason because usually the person rejecting you is probably somebody not making very much money. And they're working through as many of these approvals as they can in a given Workday. Their goal is volume. Not quality. No, you can't just make better people for this. Like you said, that's not really a scalable solution. And to me, Google is pulling the levers a can, which are restricting permission access, making it more stringent, or at least making the declarations very obvious when it's something really sensitive, which we're seeing now in Android 13. And we've seen in previous Android versions. And I think that we'll continue to see that trend and we'll continue to be upsetting to a particular audience of hin enthusiasts who don't like that. Google is clamping down this way. At the same time, it's hard to see this being a problem that can be solved at scale without immensely powerful machine learning and really rigorous controls. And that just seems to be the direction Google is heading. That's how most of their products work. So I feel like, yeah, the horror stories from the play store are really frustrating. You hear about devs who get delisted for totally bogus reasons because the reviewer either didn't understand something about a trademark or copyright complaint, or they thought there was something in the app that had changed, but they misunderstood and seeing that kind of stuff. And like seeing people's livelihoods be affected by that. That sucks at the same time. Michelle, what you said, I think is what I've heard as well, that there is no app store. That's good at this. You hear complaints about apple on the app store all the
Logan:
00:25:21
time. This is part of why I believe it's a good idea to leverage the Android security model to enforce this level of quality control. For example, the play store requires that all apps uploaded must target a certain SDK. And basically what that means is it's tested on a certain Android version and the OS enforces certain restrictions around that app that typically get stricter as the Android version goes up. And so that's one way they can enforce quality control. The other one. Permissions, which they do quite heavily. Part of it is informing users as well. So they understand, you know, when I'm giving an app access to this thing, it has apps don't have access to a whole lot of sensitive information without user consent. And so having to rely on the app store to communicate, that can be helpful. But part of it is also up to the user.
Mishaal:
00:26:09
I wanted to ask your thoughts on the growing number of permissions that seem to be purposely designed so that their actual enforcement mechanism isn't the Android OS itself, but rather Google play. And I know like there's a lot of permission that says, oh, app stores may enforce. The use of this permission, but in actuality, you know, it was designed around Google play. So I already mentioned SMS, but there's also call log. Those two permissions existed a while ago, but I'm seeing more and more permissions being added with each new OS version and their permission level might be normal, which means that they're granted an install time, but then. There's an asterisk in the documentation that says app stores may restrict this permission. So I wanted to ask you, do you think that's a good model for Google to take and do you actually see other app stores following through and actually enforcing these permissions? Like Google? I think it can
Logan:
00:26:58
be a good thing, but I think it's also a hard balancing act. You don't wanna ask the user for every little low level permission that they might understand. For example, there's the quarry, all packages permission, which basically allows an app to list and read all the apps that you have installed in your device, which you would think is pretty sensitive stuff. But the OS doesn't have a mechanism yet to enforce that permission. Even if you don't have the qual packages permission, there are other ways to get around it and list all the apps you have installed. And so the play store restricts this, but the OS doesn't that specific example I think could be enforced by the OS later, but due to how Android works currently, it's hard to do that without breaking certain app functionality. Another example is the internet permission a long time ago, it used to be a dangerous permission, which means the user had to opt in and allow an app network access. But Google later decided that would just cause permission, fatigue. And so if the user is being requested, all these permissions that they have to give almost every single app, then they'll just start blindly accepting without really looking into what they are giving an app access to. And so it's a, it's a hard balancing act. It's definitely better for the OS to enforce a permission, but sometimes it's hard to, for very low level things that the user might not understand what they're needed for or what
Mishaal:
00:28:16
they. Yeah, we've talked about this balancing act that Google has to do before. And you know, when you're developing an operating system that serves billions of users, tens of thousands of developers around the world and hundreds of millions of regular users. It's complicated. There's so many considerations that they have to deal with that you don't really have to care about. If you're only designing a platform that's for a specific use case, if you are able to own the OS image, like if you're developing your own AOS P build for your own devices, and therefore you're able to bundle your own app store. Then you can bypass a lot of Google's or Android security restrictions on third party app installers. And this is one of the things that Google does for the benefit of the overall ecosystem. But if you can develop your own operating system based on a O S P, then you don't have to really consider the security implications of that because you can whitelist your own app store. You can grant it all the remissions that Google play store might have access. Without this permission, though, if you're a regular third party app store running on a Android device that you can buy off the shelf. Then you can't install apps on your own without holding the request, install, packages, permission, and that's the permission that's shown to the user it's install, unknown apps. What it does is it allows an app to send an intent to the system package installer, or to use the session based package installer, API, to request users, to grant approval for app installs, which is why the permission name has request in its name. So when you use. API, the user has to actually grant their approval for each and every installation and subsequent update. And that can be a real pain when they wanna do batch installations or batch updates of apps. So privileged system apps like Google play store on all GMs devices or whatever apps store you pre-B bundle on the device can request the install, packages and delete packages. Permiss. To be able to fully unattended, installs and uninstalls on first installs and all subsequent app updates, but that's not attainable by user apps unless, you know, the user app is able to gain route access, which is not possible unless the boot load is unlocked on most devices. There is. However, one way that third party app stores can do unattended updates. And you did mention this before Logan, when you were talking about a Crescent and it's through a new API that was introduced in Android. So I wanted to ask you, can you explain what this API, this new feature in Android 12 does and like, how does it exactly. Yes. So
Logan:
00:30:44
Crescent does use that API, Android 12 introduced this API that allows app stores to update apps without requesting user permission all the time. And the way it works is with this concept called the installer of record. So say AC Crescent, installs, secure chat again. Crescent is then marked as the installer of record for secure chat. When it first installs it, it brings up the normal package prompt. Do you want to install this app by the OS, but once you grant consent, then in this case, Crescent is allowed to update secure chat automatically without asking the user. And this works in tandem with other Android security features like downgrade protection and signature pining, making sure that it's signed by the same developer and that you can't downgrade it. So that's what a Crescent does. The privileged permissions you were talking about, like you said, they don't ask for user permission, but what's nice about the Android 12 unattended update permission is that's not privileged. You don't have to whitelist an app in your operating system. You can use it on any Android 12 and up device. That's basically how that API works.
Mishaal:
00:31:47
And, uh, fun fact, this whole API, this feature directly came about as a result of epic lawsuit against Google, as well as all the antitrust scrutiny that Google has been facing. If it weren't for all that scrutiny Google's been having lately, then we wouldn't have this feature or the reduction in the revenue sharing fee. So. Developers. It is really important that you keep an eye out and you watch what's happening with these lawsuits and these regulations around the world, because they're having a direct impact on the operating system and the ecosystem as a whole. They're really important, even though there's a lot of legalese and a lot of politicing back and forth, but yeah, I definitely keep an eye out on all that. If you're developing an app store, I'm sure you probably don't really consider the politics behind it. If you're just actually a developer working on the backend. But what you do care about is how do you deal with the bandwidth and storage needs of serving apps to millions of users and actually storing those apps on a server when you're at Google scale, this is a very, very important discussion to have because everybody optimization. Both on the service side to reduce storage and bandwidth and on the delivery end to improve, download speeds for users and thus user retention. There was a story I remember, I think it was in 2017. When in Google intern introduced the more efficient compression method called Broley. It was used for both OS system updates and also APK files themselves. And I was surprised by the number that Google shared. It saves them about 1.5 petabytes of data per day, just by switching to a more efficient compression. Just being able to switch one little one correction method, apply that to all the apps that they apps that they serve to save so much data and thus reduce bandwidth costs on both. That's something that, you know, it's kind of like a backend thing that most users will never most users and developers will never have to consider. But one thing that I'm sure most developers are familiar with, and at least a lot of users are, have heard of is what Logan brought up before the whole split APKs and or app bundles feature and how it's delivered to users through the Google play feature delivery feature, which used to be called dynamic delivery. This is both a really neat feature and also pretty contr. Which lo alluded to earlier and I'll get into in a second. So for those of you who don't know the idea behind the split APK and, and or at bundle is that most users don't need to have installed a monolithic APK file that contains every asset code chunk and language resource. Instead, you only need the bare minimum to get the app and its core features starting all that's contained within the base PK. And then Google play can push the remaining split APK files that contain the assets, features and language, resources that are actually needed by the user. So for example, if your assistant language set to English, then you'll get the split APK file that contains English language strings. You won't get the French language strings cuz you don't need that. So this concept is actually quite old. It's older than thera. Split a BKS were introduced with Android 5.0, but a lot of developers didn't really make use of them until the Android app bundle requirement came into effect in August, 2021. So starting August, 2021, every new app that's uploaded to Google play has to use the Android app bundle format. The bundle format was introduced with Android. And to explain a bit about what it is. It's also a valid zip file, just like APKs. So within this AAB file, you have all the resources, assets, and libraries that are needed to generate multiple configurations of the app. And it's structured in a way so that you have all of these segments kind of a segment apart from each other, you have a base module, you have dynamic feature packs and you have asset packs that this tool called bundle tool uses to generate specific split APK configurations. So what developers do is they generate the Android app bundle file in Android studio. They upload that to Google play and then Google play generates split APK files on demand when the user downloads the app. So this has a potential to save users, a lot of storage space on APKs and also thus reduce how much bandwidth is needed to download. So while the idea behind this is sound, you've mentioned Logan, that there are some issues with how it's implemented. Can you explain what some of those concerns are and what Google's done to address them? If anything. There are a few
Logan:
00:36:01
valid concerns with not necessarily app bundles themselves, but the implications of how they're used. So Google play uses this feature called play app signing. And so you upload your app bundle and then Google will sign and generate the APKs themselves. As we talked about earlier, the Android operating system. Will pin the signing keys of those AP case. But in this case, it's Google, who's signing them. And so theoretically, Google could modify the APK and deliver a malicious update either intentionally or after being compromised. And so that's one big concern with, uh, app bundles. They've mitigated this somewhat with this feature called code transparency. How that works is a developer can generate this separate signature file and upload it to the play store. And users can download that and make sure that the APK that they get was generated from the app bundle that the developer made, but it's not a perfect solution for one, the developer has to choose to generate this and upload it. Then the user has to download it themselves and verify it themselves, which already cuts out a lot of people. Most people aren't gonna try to do that for every single app. They. Another problem is it doesn't verify the entire file code transparency doesn't verify the entire APK. And so things like resources could still be modified, which could be used to change the UI or something and trick the user into doing something that they don't intend to. Those are a couple of the concerns. The way Crescent works around this is that developers upload the split APKs themselves. The tool that Google uses to generate split APKs is freely available and open source, and you can download it and use it yourself. And so AC Crescent allows developers to do that and just upload the split APKs. And so they can still be used without the trust issues with app bundles. To be fair to Google. They do a very good job of this and they're handling these keys. They have a whole white paper on their infrastructure, and there are legitimate reasons for it. For example, I didn't learn this until recently, but developers often lose their signing keys. Once you lose your signing. Key, the OS won't accept updates with a different signing key. If you didn't rotate it right. And so if you lose your signing key, all of your users are left in the dark without an update. Whereas with Google play, if you lose your upload key, which you have to use to authenticate yourself, Google still has the signing key. So they can just generate a new one for your account and you can still deliver updates as normal. So there are benefits to it, but there are definite security and trust drawback.
Mishaal:
00:38:26
What about the challenges this whole play app signing introduces when it comes to distributing apps across multiple app stores, like since Google is signing the apps that get delivered to your device, right? If you were to take that app, And you were to try to say you had Samsung galaxy app store installed as well. And you tried to install an update that was already available through Samsung's galaxy app store for the same exact app you wouldn't be able to because the app that's uploaded to Samsung store is signed with a different key than the one that's uploaded to Google play. Can you talk a bit about that? That's
Logan:
00:38:59
partially a store concern and partially a developer concern. Generally speaking, if you have an app that's being signed with different keys, say for Google play and say, you're really seeking it on GI Huber, your website or something. If you're not using the same key,
Mishaal:
00:39:13
you should
Logan:
00:39:13
change the app ID, which is basically just an internal identifier to make an app uniquely identifiable to the operating system. I know I keep bringing it up, but for example, Graphos does this with their camera. There's a version. They sign themselves and the version that Google signs. And so they have different app IDs, so they can both be installed at the same time without conflicting. So that's generally what developers should do. If they're uploading to multiple stores that are signing with different keys, they should change the app ID. So they don't conflict. But many aren't aware of that. And so that does cause a challenge, especially for apps that are say on both Google play and F Android since F Android signs with their own keys, but doesn't change the app ID. They conflict with Google play versions, and you can't have both installed at the same time or use one to update to the other.
Mishaal:
00:39:57
I've talked about this before, but the Android app bundle, there's a lot of clear and obvious benefits. It brings to both Google and to users when it comes to storage and bandwidth. But it's hard not to have like a cynical view. Oh, how does this lock users more into the Google play ecosystem? Like, is it a coincidence that it has these benefits? You tell me because I'm sure many people will think that there's no coincidence that, you know, it has this benefit of doing that. But to be fair, there are some rather unique things that can be done with Android app bundles that you can't do easily with the monolithic KPK. So, for example, at the beginning of this year, Google announced the new feature that Google play will be getting called app archiving. So basically what this does is that Google will be using the bundle that delivers upload to generate a new split APK file called an archived APK. So this archived APK is basically a. That holds nothing but code to launch the app store to redownload the full app. So this stub basically has nothing in it. It's just an empty app. That's signed by the same key because Google signed it. It has the app icon. Because it's actually still installed on your device, but it has like no resources, no nothing. It's basically like a multi kilobyte file instead of like the tens, potentially hundreds of megabytes file that the original app was using up on your device. So by introducing this app, archiving feature users will be able to archive an app and reduce its storage requirements without actually uninstalling the. And if you're interested in learning more about that, by the way, I did a deep dive on the Android buys column. I, I think that's quite an interesting feature. And I'm really curious to see if other app stores actually replicate this. Like Google did design it in such a way that other app stores could replicate it. But again, like the whole permissions data, the permission safety form stuff, I don't really know if any alarm stores actually follow through and do what Google's doing and all, everything they're doing. But before we close off this discussion, cuz we're getting close to time. I wanted to bring up a prime example. Of a market where non Google play app distribution is successful. And that's China, China, for those of you who don't know, does not allow access to any Google services, the fact that Chinese Android users are using non Google play app repository is more out of necessity than anything else. So from what I've heard, speaking to Chinese users and reading online is that Chinese users source APKs from a multitude of app stores, a multitude of websites, many third party app posting websites, and more like, it's a huge mess of where they get apps from. It's it's really decentralized and that's actually, you know, Kind of the model that desktop users have been dealing with for a long time, the decentralization, but without that centralized, Google play and GMs providing those APIs, the Chinese market kind of struggles to deal with things like push notifications, whether like 13 different push notification APIs, which requires like, if you have 13 different apps, all using different push notification implement. Then that means you have 13 different apps having these foreground services, constantly waiting for push notifications to be pushed to your device and that results in a whole bunch of apps waking up at various random times and then destroying your battery life. So if you're wondering why a lot of Chinese phone brands seem to have these really, really, really aggressive battery saving measure. It's a result of this because there's no unified centralized, Google play services. Like there is outside of China, knowing what you know about how the situation is in China. Do you prefer the centralized model we have now with Google? Or do you think a fully decentralized approach would be better or maybe something in between. At
Logan:
00:43:31
a high level, I would say the decentralized approach could be better, but it won't be. And the reason I say that is because when you have multiple parties providing updates to their own mechanisms, a lot of them aren't doing it well or doing it right. Like we've seen how a Google play store, although this largely due to GMs the place where actually does security evaluations, they do app distribution. Very, very well. It's hard to make sure that those same restrictions, uh, security and privacy are being enforced across all these different mechanisms. And most app developers won't make sure updates are delivered securely and timely and automatically. And so that's why I would say the decentralized approach could be better, but it won't be say, if you have an app that updates itself, that's like the ideal situation you can have, but most developers won't want to do that. It's extra work and it's hard to do. Decentralization in the sense of say one app updating from multiple different places actually violates Android security model there's security model mandates that one app store is just one app source. It downloads apps from one place, and there are problems if you try to change that. And then there's also a problem. If you're downloading from multiple servers with the same app, say like, does you can't use certain security features like TLS certificate pinning that would. How I would evaluate the overall situation. Centralization is unfortunate in some ways, but I think if the centralized party knows how to do things well, I would say Google does, then the situation would be much better. In that case, you don't have to trust as many parties to do the right thing. Right.
Mishaal:
00:45:08
And fortunately, you know, Android isn't as locked down as iOS, where you have no choice between centralization and decentralized. It's all centralized all the time. At least with Android. 99% is through Google play, but that 1% outside of China, at least you have the freedom to choose what apps and what app stores you're using. And of course, since a S P itself is open. You also have the freedom to customize a O S P. And then build in your own app distribution ecosystem to that distribution and to the devices that you are deploying this build on, which of course is really important because a S P as I said before, does not come with an app store by default. So you have to do something custom. If you're going to build a O S P without GMs.
David:
00:45:53
App distribution is actually something that is very near and dear toper, you might even say it's kind of the core of our business in a lot of ways, because what Esper does every day is deliver applications and updates to applications, to dedicated devices, which are things like kiosks business tablets, things that typically don't have Google mobile services to begin with. And wouldn't have a reason to other than having a way to. Apps, but obviously that's not a good solution for a lot of reasons, for a lot of different kinds of devices, some of which are cost, some of which are time, some of which are engineering effort. I mean, all of those things generally are true. Being a GMs partner versus using AOS. P. So if you're using a O S P to build a kiosk or to build a point of sale or to build display, signage, whatever, it might be a medical device, perhaps home exercise equipment, we're literally on a climbing machine. You need a way to deploy updated APKs to that Android device. And if you don't have the play store, that's a little easier said than done at scale, easy to do with a few dozen. And once you get to a few hundred, few thousand tens of thousands of devices, That gets pretty hard. And so what Asper provides along with our robust device management tools is a really, really sophisticated way to deploy your apps, stage updates, group them choose when and where they go out, set conditions for success. You. And deploy your app in a way that is highly observable and where you know exactly where it's going. You know, every device that's received it, every device that hasn't, you're getting exception reports when something goes wrong. So if you've been wondering how you can deploy applications to Android devices in a really specific, targeted, highly reliable way with an enterprise grade QoS, come talk to us at SPER we're at SPER dot. I.
Mishaal:
00:47:42
Thanks David, for the obligatory per plug in the Android bites podcast. And of course, here's the obligatory closing segment of the podcast where I give our guests the opportunity to explain where can you follow him online? So Logan, where can people find you and your work? You can find
Logan:
00:48:00
me on Twitter and on GitHub as Berry ma that's my handle on most other places. And then Crescent is on github.com/crescent, ACC, R E S C E
Mishaal:
00:48:13
NT. And you can find David and I on ESER that's blog.sper.io. And thank you all again for listening to another episode of Android bites.
