David:
00:00:02
Hello, and welcome to Android bites powered by Asper I'm, David Rudick and each week I'm joined by my cohost Michelle ramen. As we dive deep into the technical world of Android, and we're going to be talking about something that keeps everybody awake at night file access and file systems on Android, which is after reading the outline for this show. I am more confused than when we started. So I'm really excited to learn here and I'll let Michelle introduce our.
Mishaal:
00:00:29
thank you, David. And I apologize in advance to anyone who's gonna walk away more confused than before they came into this show. So, uh, before we dive into the, uh, confusing aspects, let me just offer a brief opportunity for Raymond to introduce themselves. So welcome to the show, Raymond, and tell our listeners what you do. What do you work on?
Raymond:
00:00:47
Hello everyone. I'm Raymond from Hong Kong. And currently I'm work on, uh, Android farm manager project called ma farm manager. And I joined it from 2018 when I need a farm manager that can dosh copy. So after back and forth some good requests, the team invited me to join the project. Currently my focus in the MAs project is mostly on integrations about different protocols, the FTP server, and the recent one I'm working, been working on and have been asked by. The users many time is the far FTP client
Mishaal:
00:01:30
support. Well, thank you for joining us. And for those of you who haven't used the app before it's available on Google plays, the maze file manager app. And the reason I wanted to invite a developer who's worked on file managers is because the topic of today's discussion is scope storage, or rather how scope storage changed Android file access. And it's, it's a very complicated subject. Um, one that I still. To this day, I have to verify that I know what I'm talking about by looking it up on the inter developer docs. Every single time I wanna write about scope storage. I have to go back and make sure did I get this bit right? And a lot of times it's I, you, you miss something minor detail, but the reason scope storage is confusing is not anything particularly to do with the concept itself, but rather how Google approached the scope storage model. It's because that when they introduce scope storage with Android, The way file access worked then continued to change with every Android release since. So with Android 11, it got significantly changed with Android 12. There was some behind the scene changes. And with Android 13, there are some rather significant, but not a significant changes, but still like if you're an, a developer, who's trying to figure out, okay, I want to access files. Where do I store them? How do I access them? It's gotten a lot more complicated than it used to be in the earlier days of. And because file access intersects with so many different other aspects of Android, including the Linux kernel, the underlying file systems, the way the storage ship is partitioned. I think it's an interesting topic and something that's worth discussing. So before we actually talk about scope storage, I think it's important. We actually go way, way, way back. I mean, the early days of Android, like over a decade ago to talk about SD cards. So in the very early days of Android, I'm talking like in the nexus one era. A lot of phones or most phones rather depended on having removable SD cards to hold media files as well as additional apps in their data. Since at the time internal storage ships had rather mediocre capacities. So during this time Google referred to the SD card as Android's external storage, which obviously made sense because SD cards were removable and they weren't built into the device. When Google shifted away from SD cards with the nexus S though everything then had to be stored onto the devices, built in storage ship, including all the media files and all the app data, et cetera, that were previously stored on the external SD. However, in order to maintain compatibility apps that were expecting files to be stored on the physical external SD card Mount point would instead be transparently directed to the new slash data slash media Mount point to this day, you'll see a lot of ADB commands, ADB tutorials that refer to SD slash SD card when such a thing doesn't exist on most Android phones instead slash SD card is just symbolic link to slash data slash media because of this legacy of Android used to rely on physical SD card. So the reason I'm talking about SD cards and stuff, that's over a decade old is because to this day, Android still refers to slash data slash media as external storage, even though nine outta 10 times, that's actually storage. That's internal to the device. The way that Android sees storage is that you have the underlying user data partition on phones, which is mounted at slash data it's. So the data partition is writeable media. It's where all your app data, all your removal files are stored. It's generally split into two parts. You have everything outside of slash data slash media, which Android considers internal storage. Then you have everything inside of slash data slash media. That's considered external storage. And the reason those two are separated is because the internal storage and the external storage at slash data slash media are actually mounted differently. So even though the underlying data partition may be formatted in F two Fs or X four file systems slash data slash media is actually mounted with a virtual file system. There are many reasons to do this most importantly, being that. So you don't have to unmount the entire phones data partition whenever you wanna hook it up to a PC, but there are also some privacy and security benefits to having this. So prior to Android, 4.4 kit cat, this virtual SD card partition is what soft often referred to as was mounted using the VF a T file system. But this posed some problems in the early days. This is because the slash data slash media Mount point or the external storage is what I'll refer to it pretty much from now on is not only where you'll find the shared downloads folder, your pictures, music, movies, and other sensitive media folders. But it's also home to the Android folder. And the Android folder is where apps can read or write whatever files they want within their app specific directories. So like slash Android slash data slash Android slash B for, uh, game files. So just another broad step back apps could store files in one of three primary locations. There are app specific directory in the internal storage I E and slash data slash data. There are app specific directories in the external storage. Many times you see it as slash SD card slash Android. When in actuality it's slash data slash media slash the user ID slash Android or any other directory in external storage that they have access to. So app specific directories in the internal storage are rather limited in space intentionally by. So apps that need to store lots of large files or just lots of files in general, generally picks somewhere in the external storage. So an app like telegram, for example, has a ton of cashed images and documents and videos. So it stores all of that in its app specific external storage directory. The problem with the way this was implemented in Android prior to 4.4, is that the V F a T file system didn't support the kind of permission management that Google wanted to. I. The issue was that apps that wanted to read and write files into their own app. Specific external storage directories had to request the read external storage and write external storage permissions in order to do so. But requesting those permissions also gave apps broad access to the entirety of the external storage, just by needing the permissions to access their own specific directories. They also had access to the entirety. The external storage. That was a problem because this was basically start of apps, having too much access to external storage, which also housed other shared media files. So that was a lot before we move on to how Android or how Google started to address this problem. I wanted to ask about your thoughts and experiences dealing with file management in the early days of Android. Like, what were your thoughts on SD cars? Things like apps to SD mass storage, et.
Raymond:
00:08:02
First of all. I think Google's decision to use we at is external storage because of compatibility concern, because about 10 years ago, many users are still on using windows or Mac silver, little who are using Linux as their desktop. So using we as the file system, they, they, they can Mount on, they can do this without any special drivers. So that they can just plug into the USB port and they can access the files inside the Android phone directly. And then before that they of keep cats.
Mishaal:
00:08:41
I think we mostly use
Raymond:
00:08:44
the farm manager that the Android phone bundled in. So perhaps there are many different kinds of implementations from different vendors, but generally you can see the Android phone as simply as, uh, speech fund drive. If you can, if you say otherwise, very simple,
David:
00:09:05
that's an interesting analogy. And I had not thought of it that way, but that makes a lot of sense actually, intuitively.
Mishaal:
00:09:12
Yeah. So as Raymond mentioned, using V F a T allowed for broad compatibility, because the way PC access worked was that Android would actually unmount the virtual SD card partition, and then it would Mount it onto the connected device. Obviously this led to some problems early on, because if you wanted to use your phone while it's connected to the PC, all of a sudden, a lot of apps will be missing media files or missing some data. So instead Android then shifted to using the media transfer protocol or MTP to handle a file transfer. And this made it so that it didn't matter what file system was used on the Android device. It would still be supported by the PC. So be something connected to the PC. And the reason you can look that up how MTP works, but that's not really important for this, but that was one of the ways that Google. Was able to shift away from VF a T in Android, 4.4, starting from Android 4.4 and beyond, instead of mounting the virtual SD card partition as VFA, Google decided to use Lennox's file system and user space feature or fuse for short. To emulate F a T 32. So the reason Google decided to switch to emulating F a T 32, is that it allowed Android to use F fat's directory style permissions. This made it possible for Android to let apps access their own app specific external storage directories without needing those read external storage and write external storage permiss. Eventually Google did what Google does best and deprecated using fuse switched to SD card Fest for various reasons, which is a Samsung developed kernel file system wrapper. And that, uh, was done in Android 8.0 to improve performance and fixed some issues. But then they reverted back to fuse in Android 11 after making some improvements. And in order to better support some scope, storages, new feature. But we'll get into that bit later. It's a bit of Android file system history. That is not super important to understand the nitty gritty of right now. But what's important to take away from this is that by switching from the V a T style system to using fuse, to emulate F a T 32. The end result is that fewer apps needed to hold the read external storage and write external storage permissions. And so that means that fewer apps had the ability to clutter your external storage with folders or Snoop on your private media files. However, this didn't solve everything because there's still far too many apps having broad file access. And over the years, Google tried to reduce the number. The most significant thing they tried was Android 6.0, which introduced runtime remission. Read external storage and write external storage now had to be manual granted by the user at run time. So like there would be a dialogue asking the user, do you want to grant these permissions instead of granting them automatically at install time, but this still did not do enough to limit the number of apps that had broad file access, something big needed to change to do this, which leads us to scope storage. Scope storage for those of you don't know, was introduced with Android 10 and it's described as a it's a new model for the way file access works in Android. It revamped the way fi apps could access files located on the external storage. The biggest change that scope storage made was to the right external storage and read external storage permissions. So for apps that target Android 10, right, external storage effectively became deprecated. And folded into read external storage. So whenever you grant an app access to read external storage, it implicitly has the ability to also write files or modify files. Read external storage on the other hand was significantly reduced in scope granting an app that permission no longer gave it broad access to the entirety of external storage. Instead, the permission only gave access to media files that were contributed to media store collections by other apps. I'm I'm gonna get the media store in a bit. So basically apps that contribute media files to media store collections can do so without needing any permissions to do so. But in order to read or modify files that other apps contributed, they need to hold the read external storage permiss. So you're an app developer. Raymond who's worked on file manager. Can you tell us a bit about the media store API for listeners who may not be familiar with it about
Raymond:
00:13:19
the media store API is to carry API to carry different kinds of media files on the device categorized by different video types, like image, audio documents, cetera,
Mishaal:
00:13:35
et C. Right. So like Android has like an index of media files. If you've ever opened like the files app, you have like the images filter, it shows you images from various filters on your device. That's basically the user facing access to the media store and apps can add their own image files to that media store. And then other apps can access those image files in the media store. But in order to access images that other apps contributed, they need the read external storage permiss. So that's what end enter 10 and
David:
00:14:06
just provide some context on this. I'm sure we can get into this later on as well, but there were really good reasons. Google did this. Number one is just that applications in general, the number of files, they would have broad access to the ability to read and write. There are obviously nefarious things that they could do with that permission. If a user did not understand what kind of permission they were granting that application and users. When an application, especially asks for file access, I think are very likely to say yes, because that is something that from the windows and Mac O S paradigms, um, is very common applications, always need file access because they need to see directories access, you know, documents, whatever the case may be. So users are very accustom to just being like, yeah, it's an app. It needs access files. I. Files, but, and the way the Android file system works, which is say something I don't understand very well. I do not know enough to even say what granting full file system access really would let an app do per se, compared to something like windows. And so Google was trying to define, okay, what are the very safe. File types. We can let any app, as long as the user says, okay. Access and obviously images, video, audio, because those are things that people share constantly and really do want to move between applications, whether it's touching up a photo and a photo editing app, or it's sharing something and putting a filter on it on Instagram, because you already edited somewhere else in another application. People want that traceability for their personal media files. And that's what Google I think was really trying to enable.
Mishaal:
00:15:40
Right. And just to clarify for anyone who may be concerned, when we say like broad external storage access, external storage, remember is only files contained within slash data slash media. So it's only the folder. It's only the files that are created or stuff you download to your device. You can delete the entirety of slash data slash media and your phone is still the. Because it's all files created after you install your apps and you set up your device. None of it's critical to system operation, all of that stored in different partitions. So it's not like windows or something where you grant an app administrative access, it can literally nuke your device. Um, I'm unless you're how true that is actually on anymore in modern windows, or it's also not the case that you're giving an app like root access that could truly Rick havoc, the main concern with giving an app broad external storage access is. They can potentially read sensitive media files that you don't want them to access to. So like for example, if a game wants to store files on a folder of its choice, like they create a folder on the external storage in order to do that, they needed that external storage permission. But at the same time, by giving 'em that permission, they all of a sudden had the ability to read your photos or to like to look at your photos. That's not something you want to give every game access to and you who knows what they might be doing. So that's the reason why Google wanted to crack down on this. Just there's just too much potential of like a privacy invasion. But on the other hand, there are apps that legitimately do need broad, external storage access, including file managers like a maze, like the one that Raymond works on, but because their scope of read external storage access was reduced. These apps need an alternative in order to access the same list of directors they had before. Fortunately an alternative already existed in Android in the form of the storage access framework, which is actually something that Google introduced way back in Android, 4.4. So the storage access framework or SA for short makes it possible for apps to request the user, to grant access to certain directories. So what would happen is the file manager like maze would invoke SA using an intent and that will then pull up the system provided documents. Picker also known as the files app. If, if you look it up in many places, The user would then use the files app to browse through the external storage and choose which directory to grant the app access to which on Android 10, they could even choose to give an app access to the entirety of the external storage by just choosing the root folder. This sounded fine on paper using SA to access files and directories, except for one problem. Accessing files via SFA. API was slow, really, really slow. So slow. In fact, that benchmarks done by developers showed it was orders of magnitude slower than using traditional Java file API or native file IO commands. Something that might take like, uh, mere seconds to maybe scan for music files on your external storage would take maybe two minutes to do now. Like ludicrously slower developers had raised alarms about SA F of four, because it's actually been required to use since Android six point. No, if you wanted to access files on external USB flash tribes, but with Android 10, it became front and center because it became required to use for apps targeting Android 10 in order to actually. Access or read many files because enter 10 didn't support, direct file path access via the native Java file API or native file IO commands. So, Raymond, I think this is a bit where you probably know much more than we do so many people may have heard about the controversies around skip storage, but they may not be aware of how exactly negatively affected certain types of apps or use cases. Can you tell us what exactly made file access via SA significantly worse than the previous method? The
Raymond:
00:19:15
extra effort there to obtain the info stream stream of a particular file on the storage device, wherever you say, being an SD card or attached that USB from drive or something, there is an extra effort required to obtain the stream wheel the end, we the system. So I think that's the main reason that made
Mishaal:
00:19:42
powers through the SA significant. So apart from file managers like a maze, what are some other kinds of apps that were also impacted by scope storage and SA and Android 10? I think
Raymond:
00:19:52
some game key, uh, console AORs would be affected. Actually, I didn't try myself because the last device I used to play games is still running on Android act. But I think as of Android 10, some, a embryos like retro art, Or dolphin would maybe affected because the extra require on assessing the files and the storage and because the users have to put the files in a way that Google, like, so the AMBS can smoothly assess those files.
David:
00:20:30
So I, I think, and I'll steal your lines from the outline here, Michelle, because I think this is an interesting use case and it's one that I understand. So the emulator case is really interesting because essentially Google has said, all right, well, if you want to use SAAF and you wanna store stuff out here, you can use that framework like it's accessible. But if you're using something like a, we emulator that requires many, many, many, many, many files to load a RO into memory. It's going to be so slow and for reasons that are not apparently entirely documented, apparently there are excessive IPC calls from the document via API when using SA basically just that's what slows it down so much. And so the solution that this we emulate dev had to do was basically. Everything gets stored in the app directory instead. And that means save states. It means your ROS, probably your bios files, all of it needs to go there, uh, because that's the only way they can guarantee it'll be performant. And that means if you uninstall the app. All of that goes away unless the developer has hit that flag that says, Hey, there's sensitive personal data here. Do you want to keep it when you uninstall the app? But Michelle, uh, I'm sure you can explain more about this whole idea. The Google play doesn't respect that flag, which to me is nuts. I didn't know
Mishaal:
00:21:43
this. Yeah. So there's actually a manifest flag called has fragile user data. If you set that to true, then whenever you go to Android settings and try to uninstall an app, you'll see a tick box that says keep so and so megabytes of. So when you tick that box, the app specific files in their external storage directory doesn't get deleted. So you can reinstall the app at a later time and all those files will still be there. The only problem is Google play. Doesn't read that the app actually has that flag. So if you try to uninstall an app via Google, play it, doesn't ask you if you wanna keep that data. So you have to actually go through Android settings to uninstall it, to be prompted, to keep the data, not through Google play. So, yeah, it's a particularly tricky problem. And there because there's just Andrew, such a massive platform. There are so many different apps and so many different reasons. And so many different ways apps have to access files on external storage. Of course like Google's goal is good overall, but like the way it impacts certain. Types of apps is going to be different. And interestingly, scope storage also impacts other Android platforms differently. So from what I've heard, Android TV behaves very interestingly in terms of scope storage. So Raymond, do you know anything about the way scope storage impacts Android TV? Can you tell us a bit about that?
Raymond:
00:22:59
Actually, I didn't have too much experience on this. I chatted my two boss at home just tonight and it's indeed running on Android 10. It is a new report TV report. So haven't tried much about foul assess restrictions, but on the other hand, I can point out on my Android phone, which is funny on Android 12. I used to use video land player to play video files while I'm on the go and. Power management is already affected by the new storages model because of the scope of storage. And I want to delete some media files after watching it, but as of today, video land players, storages model is still rely on assessing the roots of that storage device. Hence making it impossible to delete, delete media file, like before, like, um, Android and I, I think. Before my upload, my draw on to
Mishaal:
00:23:59
Android trial. So just to return to that Android TV example. So from what I've read, it's a very weird quirk of some Android TV devices is that they don't properly respond to the SAAF intent. So from what I said before is that apps can send an intent to SA to open the files. That the user can then go and choose which directories to give access to the app to the problem is that a lot of Android TV devices don't have the documents UI app or the files app, or they don't, haven't set it up properly. So what happens is that app says, Hey, I want to have access to this directory. Then they try to send an intent. And then the app doesn't exist on the enter TV device. So you effectively, literally cannot do anything with that app or like for a dolphin example, dolphin asked the user, where did you store your. You try to open the folder, you can't do it. So you literally cannot choose where to store your games or you'd have to manually move it to its app specific directory. So it's a very weird situation. And from what I read, it's just because Google didn't mandate it on Android TV, even though SAAF like the proper implementation is mandated on Android for handhelds. For some reason, it wasn't mandated on Android for TV. And, uh, I have no idea why, but there is several bug reports about this, but it does work on shield TV at least. Yeah. And
Raymond:
00:25:15
actually some reports about bar manager also from devices that doesn't have documents, UI install. Most are because they are using custom realms.
David:
00:25:27
Yeah. And I mean, that just is a, Michelle is a good example of how Google has to consider platforms that may or may not get as much attention as mainline Android does. And the effects that certain requirements will have on them. Android. TV's a great example because it SPER. We deal with a lot of Android TV devices, and it is a wild west in terms of what manufacturers choose to implement and not implement and allow you to even access Google, apparently just does not enforce things as strictly or care as much. And so when you have that kind of situation, yeah. You're gonna have unintended consequences and I'm sure Google knew that like going in, I'm sure they could see like, well, this is gonna break
Mishaal:
00:26:07
that. So another interesting story while we're on the topic of Android TV and paying attention to other platforms. When Nvidia rolled out the Android TV, 11 update to the shield TV. So on the initial release, the TV did have the documents UI. So you could open the files app and be prompted to choose which directory to access. However, the documents UI didn't provide a way to actually navigate the app using the. So even though it opened the app and asked you to choose a directory or create a directory, you could do neither. So you instead after like ADB into it and then use your mouse, or like send inputs that was fixed with a subsequent release, but it just goes to show, you know, these kinds of things aren't tested as equitably on all platforms and all devices. But fortunately, you know, that's why with Android. Google actually did respond to feedback. There's a lot of backlash from developers who were worried about the changes of about the amount of development work they'd have to. To update their apps to prepare for scope storage. So Google actually backtracked on some scope, storage related changes in Android 10. The first thing they backtracked on was their plan to split, read external storage into three separate permissions, which they later did in Android 13. They also made it so Android 10 wouldn't enforce scope storage at all for apps that target Android nine. And they also made it possible for apps targeting Android 10 to just opt out of script storage entirely by setting a specific flag in the. So even though scope storage was technically introduced in Android 10, it didn't really go into effect at all until the next release. So with Android 11 scope storage is really in effect now, or when Android 11 was released with Android 11, Google made a huge amount of changes to address developer feedback. One of the most important things they did was introduce the new permission called manage external storage. Which is shown to the user as quote, all files access. Uh, and I'm, I'm putting that in quotes for a reason. I'll explain in a second. So all files access or manage external storage permission grants, the app access to quote all files on external storage, except for files under the Android directory. So it's everything except for the Android direct. At the same time, they also updated storage access tray mix for that apps, couldn't be given access to the route directory of the external storage. In other words, they couldn't just broadly ask for access to all of external storage. They couldn't ask for the download directory and they couldn't ask for the Android directory either. So this means that the most practical way for apps like file managers to gain access to most of external storage would be through the new managed external storage permiss. So this is something you're probably familiar with. Raymond. Can you tell our listeners how Google was able to introduce, manage external storage without negating the efforts that they went through to reduce the scope of read external storage? Like how exactly is manage external storage protected?
Raymond:
00:29:04
I think you are probably talking about the Google plays approval on using this manage external storage flag in manifests. So they required some apps who need to access to all the files on the storage to raise this rack in the manifest and send it to Google, play for reveal. And once they got their approval, you can publish it to Google play with this flag, enable and devices running on, running on enjoy 11 or onwards. When they seen this flag, they will ask user, if you want to allow the app to you to look at or access all the files on the storage. So this is how Google play.
Mishaal:
00:29:52
So yeah, there's only certain types of apps that Google play will allow to request this permission. And you have to go through a permission, declaration, forum and whatnot. So not just any app can request it and be granted access to. So even, so even though this, this permission is shown to the user as all files access, as I mention. It doesn't actually give you access to all files on the external storage apps with its permission. Aren't allowed to access files under the Android directory. However, amazed as well as many other file managers discovered a workaround to access sub directories under the Android folder. Anyways. So I wanted to ask you, Raymond, can you describe for our listeners, how did this loophole work? Like how did a maze get access to Android slash data and Android slash B and Android 11?
Raymond:
00:30:40
Okay. About some time user report, such problem. And we also saw some app which actually can access these to directories and we Googled and Googled until we find some developers talking about using SA and by declaring the intent to assess this particular path directly in the intent. When is the intent sent to the documents? UI, uh, documents, UI will ask you. If you want access this Android data or Android be directory after you press yes. Amaz or your code can access these tools directories directly using SA
Mishaal:
00:31:22
right. So to summarize basically Android made it. So you can't ask for access to the Android folder in its entirety. Developers figured out that you can just ask users to grant access to its sub directories individually. So even though you can't access all of Android, you can access all of Android slash data and all of Android slash B individually, which effectively makes the restriction kind of useless for now until Android 13, which will get on a little bit. Moving on to the other changes that Android 11 introduced in terms of scope storage. As I mentioned before, Android 11 was the first release to actually start enforcing scope storage on apps. Whereas apps targeting Android 10, but running on Android 11 could still opt out using the manifest attribute that I talked about before apps targeting Android 11 could not opt out. So no matter what Android 11 is the release where skip storage is really starting to be. And because of Google plays, shifting target API level requirement, eventually all new apps and apps that are being updated would have to target Android 11. So scope storage is the law of the land and Android. Now, by this point in time, Fortunately Google did make some other concessions to make script stories more relatable in Andrew 11. It restored direct file system access through the Java file API and native file IO commands. This was beneficial because there were a lot of Java libraries that relied on these commands and it would, it would be a huge hassle to have to update those libraries in on top of your app. I actually didn't know this before, but while I was researching the reason Android 10 didn't support, direct file system access through the Java file API and native file IO commands wasn't because of any intentional restriction by Google, but it was because it would've required significant effort in intercepting kernel calls, whenever they were developing scope storage, they weren't able to implement enforce scope, storage, and implement the Nel call interceptions. So that's why Andrew 10 didn't support direct file path access. With Android 11, when they switched back to the fuse implementation for emulating F ad 32 on the external storage, they also tuned it for better performance to bring it on par with SD card Fs. But one of the other things they did, one of the other reasons they switched back to fuse is that it enabled better support for a script storage feature that I didn't mention before and Android 10 Google added a new permission called access media location. This is a permission that apps have to hold in order to access. Unredacted XF metadata from photos. So for example, if an app wanted to access the GPS tag in a photo, they'd have to hold the new access media location permission. The problem is that in Android 10 direct storage access through SD card Fs didn't support this feature. So like if an app were to use direct file system command, In order to access files, they'd still be able to get access to unredacted excess metadata. So that's probably partially why it wasn't allowed in Android 10 and why Android 11. It was restored because fuse does support this feature. So it is able to gate access to excess metadata that apps hold. And it's able to check that actually actually actually have the permission. That's. There's a bit of, a little complicated under the hood behind the scenes kind of business going on with how Google actually implements the higher level entered features based on lower level file system changes. And it seems really confusing that Google keeps switching back and forth between these file systems and partitions like every release, but there's a reason for every change they make.
David:
00:34:51
and just, uh, on the subject of X's metadata and why that is something that Google would get hung up on. You could use that to track somebody's location. Like it'd be very easy using photos from the camera roll would be trivial.
Mishaal:
00:35:04
Yeah. So I switching away from the SD card Fs, which was all inker and not having to intercept those kernel calls. They were able to support this feature by moving back to fuse, which was all user. So yeah, enough of the complicated under the hood talk. Now let's just go back to the actual discussion scope storage, actually, LA there is one more bit I'm gonna talk about later, but before we move on to what was introduced in Android 12, I wanted to ask Graham, what are your thoughts overall on the state of file access since Android 11? Do you think Google really did address most of the issues that were raised by developers when they first introduced skip storage? Or are there still some lingering issues maybe around perform. I think
Raymond:
00:35:44
the most important complaint is that I can no longer use Java file API as I did in back in the kit days. I still miss that because I, I can still freely navigate to the direct as I like. But that's my problem because I still use the concept of desktop computer into an Android device and ignoring the those privacy concerns. Uh, this that's what I can think of, but I think Google is tried to close the loopholes, the chaos since the kit cat days, because of so many different types of kinds of APIs, frameworks, they Google asked the ramps to use.
Mishaal:
00:36:33
Yeah. And one of the other problems still with storage access on Android. The overhead introduced by views itself. So this is actually one of the reasons why Google originally shifted away from views to SD card Fs was to reduce overhead and file IO operations, but they did address some of those performance concerns when they brought back views. Because they, uh, actually tuned it for better performance, but there's still some performance overhead. The reason why this performance overhead is because every single time there's a reader write request. All these requests have to be forwarded back and forth between the fuse Damon and the fuse driver instead of an app. Directly making calls that get sent to the underlying file system. So like, there's this middle man that's checking each command and passing each command. So that introduces some overhead on commands that increases the amount of time it takes to traverse files on certain directory, et cetera. And this is especially felt when you're trying to. Do bulk file operations. So like you're trying to mass delete or mass move or mass copy, a whole bunch of files, especially small files, like a ton of small files. You're really gonna feel that performance degradation with fuse. But some of the optimizations that Google did to reduce the impact that fuse has is that any file operations that are done within the external app specific directories, AKA within Android slash data, Android slash OPB bypass fuse, so that all those commands are sent directly to the underlying file system. So those are pretty much native performance. And then supposedly from what I read, there was also an update to the media provider mainline module. So for those of you who know media provider is one of the modular system components that contains the fuse Damon that contains a lot of the Android framework, APIs and classes for actually managing file access. And October, 2020 update, supposedly brought improvements to bulk file operation performance for apps that held the manage external storage permission, but still there are still performance overheads that are unavoidable. And the only way to avoid them is to really just rewrite how your app works instead of like doing operations that would invoke fuse. You can use, uh, bulk APIs and content provider or use media provider APIs. I. there is one thing that changed Android 12, that that should improve the performance of file operations through fuse. And it's actually called fuse pass through. So it's a feature that relies on specific support from the kernel and the framework specific kernel versions. I think Android 12 5.1 and above 5.10 and above. And then the framework that comes through an update to the media provider module. So what happens with fuse pass through is that when the fuse Damon is handling a file open request, and it determines that the requesting app has full access to the requested file, then the system can then subsequently forward all read and write request directly to the lower file system. So, as I mentioned before, what would normally happen? Would that each write, read and write request is forwarded from the fuse driver to the fuse Damon, which introduces a lot of overhead. So by bypassing that forwarding, it improves performance for reading and writing large files in theory, because once you open that large file, you can continue to read and write to it without having to go through the middle man of the fuse. this, of course won't make a lot of difference for reading and writing lots of small files because you still have to read and open it and then do a small read and write command and then open another one. And, you know, they'll still add up, but if you're reading and writing data to a large file on the external shared storage directory, then views pass through should make a difference. It's probably something you haven't heard of before, cuz it's not something really Google announced, but it's something that's there in the Android docs and it's configured in the. But something you may have heard is what Google introduced in Android 13 and what I alluded to before. So in Android 13, they're actually splitting up the read external storage permission into three new permissions called read media audio, read media video, and read media images. So what happens is that apps that are targeting Andrew 13, have to request these individual permissions to access the media files of each. That are owned by other apps. So for example, an app targeting Android 13 that wants to read or modify image files, it doesn't own would need to request the read media images, permiss. Similarly, if they wanted to modify an audio file, it didn't own they'd need the read media audio permission. If you were tracking Android APIs across the years, you might have noticed that these APIs were actually planned or these permissions were actually planned to be introduced in Android 10, but they were removed in Android 10 beta three, because Google wanted to give developers more time to adapt their apps for script storage, but we're several years out. So Google feels comfortable in reintroducing this change into Android. One of the other things that Android 13 did is that it closed the loophole that file managers like a maze used to gain access to the sub directories under Android. So you can no longer use storage access framework to request for access to slash Android slash data or slash Android slash B anymore. I wanted to ask you Raymond, for what are your thoughts on Android 13 closing this loophole? And if you found a workaround that you can tell us,
Raymond:
00:41:46
Right now I, I, or our team haven't came up with a solution on solving this problem. And personally, as a mobile game director in my previous job, actually, I will welcome the decision to close this local because hackers can no longer use the Android phone for long to modify the game. Actually our team is so an amazed for which teaches people how to hack the Android games by using MAs and open JDK on an Android device. On the negative side, it will mean more work and more work around, even more additions, negoti, uh, traditions, negotiation to Google because we want our. Direct via assess API back, but on the positive side, I think if you think in an LS way, LS from the ground up had restricted to almost all files except designated media types, like photos, like new sales videos, except the apple requests use, use their own proprietary. But perhaps some Android, maybe we can still use TP or something like, yeah. MTP is on the USB side. That means you still plug on the computer. You used to plug in the computer to access the whole Android device as like accessing USB fund drive. But on device, on the go, you are seeing the Android cloud system as like a iPhone or something. You only look at your music, your photos, your videos, this on the positive side, but there's still one complain. I personally this, uh, I still miss the ability to play music in the folder. I used to organize music in as photos and rather than scattered all them into one folder and screw it by and four drag bank to form a pay list. I just, I still want to just organize it photo nicely and just throw into music player. And there you go. Yeah, that's all my that's one of my thoughts.
David:
00:44:04
I mean, I think we can all relate to that too. I'm sure we all grew up using either windows or some other traditional file management system and that's just how our brains work. It it's how we think about computers and how they organize data.
Mishaal:
00:44:17
I wanted to ask you a question, David, actually hearing about all the changes that have come so far with Android in relation to file access and file management and all the restrictions that have come. Do you think Android will continue to offer relatively open access to the underlying external file system? Or do you think will become more and more like iOS where you don't really have a sense of what actually how the, the structure is actually.
David:
00:44:45
Yeah, iOS is so totally abstracted. That way apps can create folders in this protected media space, essentially what it is. It's the same thing as Androids media collections. I'm sure in principle, I'm sure it works differently, but if apps can't see anything outside that very, very narrowly scoped storage area. I've run into this so many times on my iPad where I'm just like, I want to see the files, this app I know has access to, but for whatever reason, I can't, because probably those two developers would need to cooperate and say, Hey, I can't see your stuff. Why can't you see it? And they would have to figure out okay. Um, what do I need to change my application to see like Google photos, for example, one of the strange limitations on Google photos, you can only put one Google photo in one iMessage at a time. You can't do multiple Google photos, probably because iOS doesn't support that for third party gallery style applications in the file picker. So you can see the problems, this approach introduces because it inherently limits what you can do with the files too. And explaining that to users. Like, you're never going to get them to understand the nuances, why one application can do one thing and why one application can't do another. And on iOS, that is something I run into all the time. But like even me being fairly casual for an Android enthusiast, this is not the kind of stuff most people run into. And I think that is where Google. Kind of wants to take, at least the Android phone experience is to a place where it's very hard for people to break. That's probably the most important thing. They don't want people in these directories because the chance that they could cause an app to misbehave and then either crash or lose their data, God forbid there's something they can mess up. So yeah, I do see Google continuing to move this direction. And the sad truth is that there's not a lot of incentive for them to preserve. This more traditional file structure access. I mean, can you imagine how many developers out there, especially for things like mass market mobile games, get emails from people trying to like mess around with directory structure and they break something. I'm sure it happens pretty frequently. And not to say that that is Google's problem. Like, you know, users go doing things that break devices all the time, but at the end of the day, Google's concern is of a massive ecosystem. And so they're. What's best for their partners in that ecosystem, their partners are Samsung. Show me all of these major smartphone vendors. And they may say, Hey, we're getting a lot of customer support issues around vis people are really having trouble using file apps. How do we make this simpler? And so Google has to digest that feedback. And as much as we like to think that the Android team is above the fray, in terms of deciding how the platform works. I don't think they totally are. They have to consider these things. And they also have to consider the fact that there are hundreds of millions of people out there who use this and who they would like to use an Android phone one day. And the more they can make it feel seamless in terms of that experience, the easier it is to get somebody to switch platforms. So there's a lot of stuff they have to think about. That's like, I mean, totally beyond the engineering level there, but. Yeah, I'm sure they're considering this. And I would say that the direction we've seen Android go that is they're going toward a more restrictive
Mishaal:
00:48:10
model. Oh. And for those of you at home who didn't immediately realize what David meant when he said this healed up an iPhone to show us. So, uh, that's all the questions that I actually had for this discussion. I mean, I know we went through an absolute ton of information and I hope this is starting to make sense. And I haven't actually confused you more about going through all of this. I mean, trust me, it took me a lot to actually digest and put all this information down and there's still stuff that I could talk about, like more information, but you gotta stop somewhere. And I think a one hour mark is a pretty good place to.
David:
00:48:47
And if this is an interesting topic to you, the entire subject of how storage works on Android and how applications interact with the operating system. That's a great question for a company like Asper to answer because we deal with apps on Android devices, especially unusual ones of varying OS distro, or excuse me, varying OS versions, underlying platform versions all the time. So if you're wonder. Hey, can I have an app that like behaves nicely on Android 12, but also works on my Android nine devices, which I already know it does. We can help you figure some of this out. We can help you understand how the platform has changed. What kind of devices you may need to create the sort of experience you want. And when it comes to things like Android TV, well, how do you get around the issue of not having access to the files app on an Android TV device? That is also one of those sticky questions that you can come to us. And we can see if there's a solution there because we deal with very strange Android devices all the time. We are not a first customizing firmware. We have our own distribution called foundation available for arm and X 86 devices. And this is the kind of thing that if you need an Android device to behave in a particular way, because it has a very specific task associated with it. That's what ER's here for. Check us out@esper.io and Michelle, I'll let you go ahead and, uh,
Mishaal:
00:50:01
do the outro. Thanks, David. And, uh, thank you again, Raymond for joining us. So if people want to follow you in your work, where can they find you? Oh, you can find me
Raymond:
00:50:11
at hab.com/trans love because I love trans music. And you can also find me on our apps, telegram channel ma file manager, which you can find it at the ma
Mishaal:
00:50:26
file manager app. Yeah. And if you're looking for a material themed open source file manager app check out maze file manager on the Google play store. And if you're looking for the content produced by the editorial team at Esper, check us out at blog, doper.io, where you'll find all of our content. Not just our blog articles, not just our newsletter, but also episodes of this podcast, which I'd say thank you again for joining us and listening through an hour long discussion of the nerdiest final like topic I could think of right now. Um, hopefully this really helped you out. It helped me out in piecing this together. So thank you for joining us on another episode of Android bites.
00:00:02
Hello, and welcome to Android bites powered by Asper I'm, David Rudick and each week I'm joined by my cohost Michelle ramen. As we dive deep into the technical world of Android, and we're going to be talking about something that keeps everybody awake at night file access and file systems on Android, which is after reading the outline for this show. I am more confused than when we started. So I'm really excited to learn here and I'll let Michelle introduce our.
Mishaal:
00:00:29
thank you, David. And I apologize in advance to anyone who's gonna walk away more confused than before they came into this show. So, uh, before we dive into the, uh, confusing aspects, let me just offer a brief opportunity for Raymond to introduce themselves. So welcome to the show, Raymond, and tell our listeners what you do. What do you work on?
Raymond:
00:00:47
Hello everyone. I'm Raymond from Hong Kong. And currently I'm work on, uh, Android farm manager project called ma farm manager. And I joined it from 2018 when I need a farm manager that can dosh copy. So after back and forth some good requests, the team invited me to join the project. Currently my focus in the MAs project is mostly on integrations about different protocols, the FTP server, and the recent one I'm working, been working on and have been asked by. The users many time is the far FTP client
Mishaal:
00:01:30
support. Well, thank you for joining us. And for those of you who haven't used the app before it's available on Google plays, the maze file manager app. And the reason I wanted to invite a developer who's worked on file managers is because the topic of today's discussion is scope storage, or rather how scope storage changed Android file access. And it's, it's a very complicated subject. Um, one that I still. To this day, I have to verify that I know what I'm talking about by looking it up on the inter developer docs. Every single time I wanna write about scope storage. I have to go back and make sure did I get this bit right? And a lot of times it's I, you, you miss something minor detail, but the reason scope storage is confusing is not anything particularly to do with the concept itself, but rather how Google approached the scope storage model. It's because that when they introduce scope storage with Android, The way file access worked then continued to change with every Android release since. So with Android 11, it got significantly changed with Android 12. There was some behind the scene changes. And with Android 13, there are some rather significant, but not a significant changes, but still like if you're an, a developer, who's trying to figure out, okay, I want to access files. Where do I store them? How do I access them? It's gotten a lot more complicated than it used to be in the earlier days of. And because file access intersects with so many different other aspects of Android, including the Linux kernel, the underlying file systems, the way the storage ship is partitioned. I think it's an interesting topic and something that's worth discussing. So before we actually talk about scope storage, I think it's important. We actually go way, way, way back. I mean, the early days of Android, like over a decade ago to talk about SD cards. So in the very early days of Android, I'm talking like in the nexus one era. A lot of phones or most phones rather depended on having removable SD cards to hold media files as well as additional apps in their data. Since at the time internal storage ships had rather mediocre capacities. So during this time Google referred to the SD card as Android's external storage, which obviously made sense because SD cards were removable and they weren't built into the device. When Google shifted away from SD cards with the nexus S though everything then had to be stored onto the devices, built in storage ship, including all the media files and all the app data, et cetera, that were previously stored on the external SD. However, in order to maintain compatibility apps that were expecting files to be stored on the physical external SD card Mount point would instead be transparently directed to the new slash data slash media Mount point to this day, you'll see a lot of ADB commands, ADB tutorials that refer to SD slash SD card when such a thing doesn't exist on most Android phones instead slash SD card is just symbolic link to slash data slash media because of this legacy of Android used to rely on physical SD card. So the reason I'm talking about SD cards and stuff, that's over a decade old is because to this day, Android still refers to slash data slash media as external storage, even though nine outta 10 times, that's actually storage. That's internal to the device. The way that Android sees storage is that you have the underlying user data partition on phones, which is mounted at slash data it's. So the data partition is writeable media. It's where all your app data, all your removal files are stored. It's generally split into two parts. You have everything outside of slash data slash media, which Android considers internal storage. Then you have everything inside of slash data slash media. That's considered external storage. And the reason those two are separated is because the internal storage and the external storage at slash data slash media are actually mounted differently. So even though the underlying data partition may be formatted in F two Fs or X four file systems slash data slash media is actually mounted with a virtual file system. There are many reasons to do this most importantly, being that. So you don't have to unmount the entire phones data partition whenever you wanna hook it up to a PC, but there are also some privacy and security benefits to having this. So prior to Android, 4.4 kit cat, this virtual SD card partition is what soft often referred to as was mounted using the VF a T file system. But this posed some problems in the early days. This is because the slash data slash media Mount point or the external storage is what I'll refer to it pretty much from now on is not only where you'll find the shared downloads folder, your pictures, music, movies, and other sensitive media folders. But it's also home to the Android folder. And the Android folder is where apps can read or write whatever files they want within their app specific directories. So like slash Android slash data slash Android slash B for, uh, game files. So just another broad step back apps could store files in one of three primary locations. There are app specific directory in the internal storage I E and slash data slash data. There are app specific directories in the external storage. Many times you see it as slash SD card slash Android. When in actuality it's slash data slash media slash the user ID slash Android or any other directory in external storage that they have access to. So app specific directories in the internal storage are rather limited in space intentionally by. So apps that need to store lots of large files or just lots of files in general, generally picks somewhere in the external storage. So an app like telegram, for example, has a ton of cashed images and documents and videos. So it stores all of that in its app specific external storage directory. The problem with the way this was implemented in Android prior to 4.4, is that the V F a T file system didn't support the kind of permission management that Google wanted to. I. The issue was that apps that wanted to read and write files into their own app. Specific external storage directories had to request the read external storage and write external storage permissions in order to do so. But requesting those permissions also gave apps broad access to the entirety of the external storage, just by needing the permissions to access their own specific directories. They also had access to the entirety. The external storage. That was a problem because this was basically start of apps, having too much access to external storage, which also housed other shared media files. So that was a lot before we move on to how Android or how Google started to address this problem. I wanted to ask about your thoughts and experiences dealing with file management in the early days of Android. Like, what were your thoughts on SD cars? Things like apps to SD mass storage, et.
Raymond:
00:08:02
First of all. I think Google's decision to use we at is external storage because of compatibility concern, because about 10 years ago, many users are still on using windows or Mac silver, little who are using Linux as their desktop. So using we as the file system, they, they, they can Mount on, they can do this without any special drivers. So that they can just plug into the USB port and they can access the files inside the Android phone directly. And then before that they of keep cats.
Mishaal:
00:08:41
I think we mostly use
Raymond:
00:08:44
the farm manager that the Android phone bundled in. So perhaps there are many different kinds of implementations from different vendors, but generally you can see the Android phone as simply as, uh, speech fund drive. If you can, if you say otherwise, very simple,
David:
00:09:05
that's an interesting analogy. And I had not thought of it that way, but that makes a lot of sense actually, intuitively.
Mishaal:
00:09:12
Yeah. So as Raymond mentioned, using V F a T allowed for broad compatibility, because the way PC access worked was that Android would actually unmount the virtual SD card partition, and then it would Mount it onto the connected device. Obviously this led to some problems early on, because if you wanted to use your phone while it's connected to the PC, all of a sudden, a lot of apps will be missing media files or missing some data. So instead Android then shifted to using the media transfer protocol or MTP to handle a file transfer. And this made it so that it didn't matter what file system was used on the Android device. It would still be supported by the PC. So be something connected to the PC. And the reason you can look that up how MTP works, but that's not really important for this, but that was one of the ways that Google. Was able to shift away from VF a T in Android, 4.4, starting from Android 4.4 and beyond, instead of mounting the virtual SD card partition as VFA, Google decided to use Lennox's file system and user space feature or fuse for short. To emulate F a T 32. So the reason Google decided to switch to emulating F a T 32, is that it allowed Android to use F fat's directory style permissions. This made it possible for Android to let apps access their own app specific external storage directories without needing those read external storage and write external storage permiss. Eventually Google did what Google does best and deprecated using fuse switched to SD card Fest for various reasons, which is a Samsung developed kernel file system wrapper. And that, uh, was done in Android 8.0 to improve performance and fixed some issues. But then they reverted back to fuse in Android 11 after making some improvements. And in order to better support some scope, storages, new feature. But we'll get into that bit later. It's a bit of Android file system history. That is not super important to understand the nitty gritty of right now. But what's important to take away from this is that by switching from the V a T style system to using fuse, to emulate F a T 32. The end result is that fewer apps needed to hold the read external storage and write external storage permissions. And so that means that fewer apps had the ability to clutter your external storage with folders or Snoop on your private media files. However, this didn't solve everything because there's still far too many apps having broad file access. And over the years, Google tried to reduce the number. The most significant thing they tried was Android 6.0, which introduced runtime remission. Read external storage and write external storage now had to be manual granted by the user at run time. So like there would be a dialogue asking the user, do you want to grant these permissions instead of granting them automatically at install time, but this still did not do enough to limit the number of apps that had broad file access, something big needed to change to do this, which leads us to scope storage. Scope storage for those of you don't know, was introduced with Android 10 and it's described as a it's a new model for the way file access works in Android. It revamped the way fi apps could access files located on the external storage. The biggest change that scope storage made was to the right external storage and read external storage permissions. So for apps that target Android 10, right, external storage effectively became deprecated. And folded into read external storage. So whenever you grant an app access to read external storage, it implicitly has the ability to also write files or modify files. Read external storage on the other hand was significantly reduced in scope granting an app that permission no longer gave it broad access to the entirety of external storage. Instead, the permission only gave access to media files that were contributed to media store collections by other apps. I'm I'm gonna get the media store in a bit. So basically apps that contribute media files to media store collections can do so without needing any permissions to do so. But in order to read or modify files that other apps contributed, they need to hold the read external storage permiss. So you're an app developer. Raymond who's worked on file manager. Can you tell us a bit about the media store API for listeners who may not be familiar with it about
Raymond:
00:13:19
the media store API is to carry API to carry different kinds of media files on the device categorized by different video types, like image, audio documents, cetera,
Mishaal:
00:13:35
et C. Right. So like Android has like an index of media files. If you've ever opened like the files app, you have like the images filter, it shows you images from various filters on your device. That's basically the user facing access to the media store and apps can add their own image files to that media store. And then other apps can access those image files in the media store. But in order to access images that other apps contributed, they need the read external storage permiss. So that's what end enter 10 and
David:
00:14:06
just provide some context on this. I'm sure we can get into this later on as well, but there were really good reasons. Google did this. Number one is just that applications in general, the number of files, they would have broad access to the ability to read and write. There are obviously nefarious things that they could do with that permission. If a user did not understand what kind of permission they were granting that application and users. When an application, especially asks for file access, I think are very likely to say yes, because that is something that from the windows and Mac O S paradigms, um, is very common applications, always need file access because they need to see directories access, you know, documents, whatever the case may be. So users are very accustom to just being like, yeah, it's an app. It needs access files. I. Files, but, and the way the Android file system works, which is say something I don't understand very well. I do not know enough to even say what granting full file system access really would let an app do per se, compared to something like windows. And so Google was trying to define, okay, what are the very safe. File types. We can let any app, as long as the user says, okay. Access and obviously images, video, audio, because those are things that people share constantly and really do want to move between applications, whether it's touching up a photo and a photo editing app, or it's sharing something and putting a filter on it on Instagram, because you already edited somewhere else in another application. People want that traceability for their personal media files. And that's what Google I think was really trying to enable.
Mishaal:
00:15:40
Right. And just to clarify for anyone who may be concerned, when we say like broad external storage access, external storage, remember is only files contained within slash data slash media. So it's only the folder. It's only the files that are created or stuff you download to your device. You can delete the entirety of slash data slash media and your phone is still the. Because it's all files created after you install your apps and you set up your device. None of it's critical to system operation, all of that stored in different partitions. So it's not like windows or something where you grant an app administrative access, it can literally nuke your device. Um, I'm unless you're how true that is actually on anymore in modern windows, or it's also not the case that you're giving an app like root access that could truly Rick havoc, the main concern with giving an app broad external storage access is. They can potentially read sensitive media files that you don't want them to access to. So like for example, if a game wants to store files on a folder of its choice, like they create a folder on the external storage in order to do that, they needed that external storage permission. But at the same time, by giving 'em that permission, they all of a sudden had the ability to read your photos or to like to look at your photos. That's not something you want to give every game access to and you who knows what they might be doing. So that's the reason why Google wanted to crack down on this. Just there's just too much potential of like a privacy invasion. But on the other hand, there are apps that legitimately do need broad, external storage access, including file managers like a maze, like the one that Raymond works on, but because their scope of read external storage access was reduced. These apps need an alternative in order to access the same list of directors they had before. Fortunately an alternative already existed in Android in the form of the storage access framework, which is actually something that Google introduced way back in Android, 4.4. So the storage access framework or SA for short makes it possible for apps to request the user, to grant access to certain directories. So what would happen is the file manager like maze would invoke SA using an intent and that will then pull up the system provided documents. Picker also known as the files app. If, if you look it up in many places, The user would then use the files app to browse through the external storage and choose which directory to grant the app access to which on Android 10, they could even choose to give an app access to the entirety of the external storage by just choosing the root folder. This sounded fine on paper using SA to access files and directories, except for one problem. Accessing files via SFA. API was slow, really, really slow. So slow. In fact, that benchmarks done by developers showed it was orders of magnitude slower than using traditional Java file API or native file IO commands. Something that might take like, uh, mere seconds to maybe scan for music files on your external storage would take maybe two minutes to do now. Like ludicrously slower developers had raised alarms about SA F of four, because it's actually been required to use since Android six point. No, if you wanted to access files on external USB flash tribes, but with Android 10, it became front and center because it became required to use for apps targeting Android 10 in order to actually. Access or read many files because enter 10 didn't support, direct file path access via the native Java file API or native file IO commands. So, Raymond, I think this is a bit where you probably know much more than we do so many people may have heard about the controversies around skip storage, but they may not be aware of how exactly negatively affected certain types of apps or use cases. Can you tell us what exactly made file access via SA significantly worse than the previous method? The
Raymond:
00:19:15
extra effort there to obtain the info stream stream of a particular file on the storage device, wherever you say, being an SD card or attached that USB from drive or something, there is an extra effort required to obtain the stream wheel the end, we the system. So I think that's the main reason that made
Mishaal:
00:19:42
powers through the SA significant. So apart from file managers like a maze, what are some other kinds of apps that were also impacted by scope storage and SA and Android 10? I think
Raymond:
00:19:52
some game key, uh, console AORs would be affected. Actually, I didn't try myself because the last device I used to play games is still running on Android act. But I think as of Android 10, some, a embryos like retro art, Or dolphin would maybe affected because the extra require on assessing the files and the storage and because the users have to put the files in a way that Google, like, so the AMBS can smoothly assess those files.
David:
00:20:30
So I, I think, and I'll steal your lines from the outline here, Michelle, because I think this is an interesting use case and it's one that I understand. So the emulator case is really interesting because essentially Google has said, all right, well, if you want to use SAAF and you wanna store stuff out here, you can use that framework like it's accessible. But if you're using something like a, we emulator that requires many, many, many, many, many files to load a RO into memory. It's going to be so slow and for reasons that are not apparently entirely documented, apparently there are excessive IPC calls from the document via API when using SA basically just that's what slows it down so much. And so the solution that this we emulate dev had to do was basically. Everything gets stored in the app directory instead. And that means save states. It means your ROS, probably your bios files, all of it needs to go there, uh, because that's the only way they can guarantee it'll be performant. And that means if you uninstall the app. All of that goes away unless the developer has hit that flag that says, Hey, there's sensitive personal data here. Do you want to keep it when you uninstall the app? But Michelle, uh, I'm sure you can explain more about this whole idea. The Google play doesn't respect that flag, which to me is nuts. I didn't know
Mishaal:
00:21:43
this. Yeah. So there's actually a manifest flag called has fragile user data. If you set that to true, then whenever you go to Android settings and try to uninstall an app, you'll see a tick box that says keep so and so megabytes of. So when you tick that box, the app specific files in their external storage directory doesn't get deleted. So you can reinstall the app at a later time and all those files will still be there. The only problem is Google play. Doesn't read that the app actually has that flag. So if you try to uninstall an app via Google, play it, doesn't ask you if you wanna keep that data. So you have to actually go through Android settings to uninstall it, to be prompted, to keep the data, not through Google play. So, yeah, it's a particularly tricky problem. And there because there's just Andrew, such a massive platform. There are so many different apps and so many different reasons. And so many different ways apps have to access files on external storage. Of course like Google's goal is good overall, but like the way it impacts certain. Types of apps is going to be different. And interestingly, scope storage also impacts other Android platforms differently. So from what I've heard, Android TV behaves very interestingly in terms of scope storage. So Raymond, do you know anything about the way scope storage impacts Android TV? Can you tell us a bit about that?
Raymond:
00:22:59
Actually, I didn't have too much experience on this. I chatted my two boss at home just tonight and it's indeed running on Android 10. It is a new report TV report. So haven't tried much about foul assess restrictions, but on the other hand, I can point out on my Android phone, which is funny on Android 12. I used to use video land player to play video files while I'm on the go and. Power management is already affected by the new storages model because of the scope of storage. And I want to delete some media files after watching it, but as of today, video land players, storages model is still rely on assessing the roots of that storage device. Hence making it impossible to delete, delete media file, like before, like, um, Android and I, I think. Before my upload, my draw on to
Mishaal:
00:23:59
Android trial. So just to return to that Android TV example. So from what I've read, it's a very weird quirk of some Android TV devices is that they don't properly respond to the SAAF intent. So from what I said before is that apps can send an intent to SA to open the files. That the user can then go and choose which directories to give access to the app to the problem is that a lot of Android TV devices don't have the documents UI app or the files app, or they don't, haven't set it up properly. So what happens is that app says, Hey, I want to have access to this directory. Then they try to send an intent. And then the app doesn't exist on the enter TV device. So you effectively, literally cannot do anything with that app or like for a dolphin example, dolphin asked the user, where did you store your. You try to open the folder, you can't do it. So you literally cannot choose where to store your games or you'd have to manually move it to its app specific directory. So it's a very weird situation. And from what I read, it's just because Google didn't mandate it on Android TV, even though SAAF like the proper implementation is mandated on Android for handhelds. For some reason, it wasn't mandated on Android for TV. And, uh, I have no idea why, but there is several bug reports about this, but it does work on shield TV at least. Yeah. And
Raymond:
00:25:15
actually some reports about bar manager also from devices that doesn't have documents, UI install. Most are because they are using custom realms.
David:
00:25:27
Yeah. And I mean, that just is a, Michelle is a good example of how Google has to consider platforms that may or may not get as much attention as mainline Android does. And the effects that certain requirements will have on them. Android. TV's a great example because it SPER. We deal with a lot of Android TV devices, and it is a wild west in terms of what manufacturers choose to implement and not implement and allow you to even access Google, apparently just does not enforce things as strictly or care as much. And so when you have that kind of situation, yeah. You're gonna have unintended consequences and I'm sure Google knew that like going in, I'm sure they could see like, well, this is gonna break
Mishaal:
00:26:07
that. So another interesting story while we're on the topic of Android TV and paying attention to other platforms. When Nvidia rolled out the Android TV, 11 update to the shield TV. So on the initial release, the TV did have the documents UI. So you could open the files app and be prompted to choose which directory to access. However, the documents UI didn't provide a way to actually navigate the app using the. So even though it opened the app and asked you to choose a directory or create a directory, you could do neither. So you instead after like ADB into it and then use your mouse, or like send inputs that was fixed with a subsequent release, but it just goes to show, you know, these kinds of things aren't tested as equitably on all platforms and all devices. But fortunately, you know, that's why with Android. Google actually did respond to feedback. There's a lot of backlash from developers who were worried about the changes of about the amount of development work they'd have to. To update their apps to prepare for scope storage. So Google actually backtracked on some scope, storage related changes in Android 10. The first thing they backtracked on was their plan to split, read external storage into three separate permissions, which they later did in Android 13. They also made it so Android 10 wouldn't enforce scope storage at all for apps that target Android nine. And they also made it possible for apps targeting Android 10 to just opt out of script storage entirely by setting a specific flag in the. So even though scope storage was technically introduced in Android 10, it didn't really go into effect at all until the next release. So with Android 11 scope storage is really in effect now, or when Android 11 was released with Android 11, Google made a huge amount of changes to address developer feedback. One of the most important things they did was introduce the new permission called manage external storage. Which is shown to the user as quote, all files access. Uh, and I'm, I'm putting that in quotes for a reason. I'll explain in a second. So all files access or manage external storage permission grants, the app access to quote all files on external storage, except for files under the Android directory. So it's everything except for the Android direct. At the same time, they also updated storage access tray mix for that apps, couldn't be given access to the route directory of the external storage. In other words, they couldn't just broadly ask for access to all of external storage. They couldn't ask for the download directory and they couldn't ask for the Android directory either. So this means that the most practical way for apps like file managers to gain access to most of external storage would be through the new managed external storage permiss. So this is something you're probably familiar with. Raymond. Can you tell our listeners how Google was able to introduce, manage external storage without negating the efforts that they went through to reduce the scope of read external storage? Like how exactly is manage external storage protected?
Raymond:
00:29:04
I think you are probably talking about the Google plays approval on using this manage external storage flag in manifests. So they required some apps who need to access to all the files on the storage to raise this rack in the manifest and send it to Google, play for reveal. And once they got their approval, you can publish it to Google play with this flag, enable and devices running on, running on enjoy 11 or onwards. When they seen this flag, they will ask user, if you want to allow the app to you to look at or access all the files on the storage. So this is how Google play.
Mishaal:
00:29:52
So yeah, there's only certain types of apps that Google play will allow to request this permission. And you have to go through a permission, declaration, forum and whatnot. So not just any app can request it and be granted access to. So even, so even though this, this permission is shown to the user as all files access, as I mention. It doesn't actually give you access to all files on the external storage apps with its permission. Aren't allowed to access files under the Android directory. However, amazed as well as many other file managers discovered a workaround to access sub directories under the Android folder. Anyways. So I wanted to ask you, Raymond, can you describe for our listeners, how did this loophole work? Like how did a maze get access to Android slash data and Android slash B and Android 11?
Raymond:
00:30:40
Okay. About some time user report, such problem. And we also saw some app which actually can access these to directories and we Googled and Googled until we find some developers talking about using SA and by declaring the intent to assess this particular path directly in the intent. When is the intent sent to the documents? UI, uh, documents, UI will ask you. If you want access this Android data or Android be directory after you press yes. Amaz or your code can access these tools directories directly using SA
Mishaal:
00:31:22
right. So to summarize basically Android made it. So you can't ask for access to the Android folder in its entirety. Developers figured out that you can just ask users to grant access to its sub directories individually. So even though you can't access all of Android, you can access all of Android slash data and all of Android slash B individually, which effectively makes the restriction kind of useless for now until Android 13, which will get on a little bit. Moving on to the other changes that Android 11 introduced in terms of scope storage. As I mentioned before, Android 11 was the first release to actually start enforcing scope storage on apps. Whereas apps targeting Android 10, but running on Android 11 could still opt out using the manifest attribute that I talked about before apps targeting Android 11 could not opt out. So no matter what Android 11 is the release where skip storage is really starting to be. And because of Google plays, shifting target API level requirement, eventually all new apps and apps that are being updated would have to target Android 11. So scope storage is the law of the land and Android. Now, by this point in time, Fortunately Google did make some other concessions to make script stories more relatable in Andrew 11. It restored direct file system access through the Java file API and native file IO commands. This was beneficial because there were a lot of Java libraries that relied on these commands and it would, it would be a huge hassle to have to update those libraries in on top of your app. I actually didn't know this before, but while I was researching the reason Android 10 didn't support, direct file system access through the Java file API and native file IO commands wasn't because of any intentional restriction by Google, but it was because it would've required significant effort in intercepting kernel calls, whenever they were developing scope storage, they weren't able to implement enforce scope, storage, and implement the Nel call interceptions. So that's why Andrew 10 didn't support direct file path access. With Android 11, when they switched back to the fuse implementation for emulating F ad 32 on the external storage, they also tuned it for better performance to bring it on par with SD card Fs. But one of the other things they did, one of the other reasons they switched back to fuse is that it enabled better support for a script storage feature that I didn't mention before and Android 10 Google added a new permission called access media location. This is a permission that apps have to hold in order to access. Unredacted XF metadata from photos. So for example, if an app wanted to access the GPS tag in a photo, they'd have to hold the new access media location permission. The problem is that in Android 10 direct storage access through SD card Fs didn't support this feature. So like if an app were to use direct file system command, In order to access files, they'd still be able to get access to unredacted excess metadata. So that's probably partially why it wasn't allowed in Android 10 and why Android 11. It was restored because fuse does support this feature. So it is able to gate access to excess metadata that apps hold. And it's able to check that actually actually actually have the permission. That's. There's a bit of, a little complicated under the hood behind the scenes kind of business going on with how Google actually implements the higher level entered features based on lower level file system changes. And it seems really confusing that Google keeps switching back and forth between these file systems and partitions like every release, but there's a reason for every change they make.
David:
00:34:51
and just, uh, on the subject of X's metadata and why that is something that Google would get hung up on. You could use that to track somebody's location. Like it'd be very easy using photos from the camera roll would be trivial.
Mishaal:
00:35:04
Yeah. So I switching away from the SD card Fs, which was all inker and not having to intercept those kernel calls. They were able to support this feature by moving back to fuse, which was all user. So yeah, enough of the complicated under the hood talk. Now let's just go back to the actual discussion scope storage, actually, LA there is one more bit I'm gonna talk about later, but before we move on to what was introduced in Android 12, I wanted to ask Graham, what are your thoughts overall on the state of file access since Android 11? Do you think Google really did address most of the issues that were raised by developers when they first introduced skip storage? Or are there still some lingering issues maybe around perform. I think
Raymond:
00:35:44
the most important complaint is that I can no longer use Java file API as I did in back in the kit days. I still miss that because I, I can still freely navigate to the direct as I like. But that's my problem because I still use the concept of desktop computer into an Android device and ignoring the those privacy concerns. Uh, this that's what I can think of, but I think Google is tried to close the loopholes, the chaos since the kit cat days, because of so many different types of kinds of APIs, frameworks, they Google asked the ramps to use.
Mishaal:
00:36:33
Yeah. And one of the other problems still with storage access on Android. The overhead introduced by views itself. So this is actually one of the reasons why Google originally shifted away from views to SD card Fs was to reduce overhead and file IO operations, but they did address some of those performance concerns when they brought back views. Because they, uh, actually tuned it for better performance, but there's still some performance overhead. The reason why this performance overhead is because every single time there's a reader write request. All these requests have to be forwarded back and forth between the fuse Damon and the fuse driver instead of an app. Directly making calls that get sent to the underlying file system. So like, there's this middle man that's checking each command and passing each command. So that introduces some overhead on commands that increases the amount of time it takes to traverse files on certain directory, et cetera. And this is especially felt when you're trying to. Do bulk file operations. So like you're trying to mass delete or mass move or mass copy, a whole bunch of files, especially small files, like a ton of small files. You're really gonna feel that performance degradation with fuse. But some of the optimizations that Google did to reduce the impact that fuse has is that any file operations that are done within the external app specific directories, AKA within Android slash data, Android slash OPB bypass fuse, so that all those commands are sent directly to the underlying file system. So those are pretty much native performance. And then supposedly from what I read, there was also an update to the media provider mainline module. So for those of you who know media provider is one of the modular system components that contains the fuse Damon that contains a lot of the Android framework, APIs and classes for actually managing file access. And October, 2020 update, supposedly brought improvements to bulk file operation performance for apps that held the manage external storage permission, but still there are still performance overheads that are unavoidable. And the only way to avoid them is to really just rewrite how your app works instead of like doing operations that would invoke fuse. You can use, uh, bulk APIs and content provider or use media provider APIs. I. there is one thing that changed Android 12, that that should improve the performance of file operations through fuse. And it's actually called fuse pass through. So it's a feature that relies on specific support from the kernel and the framework specific kernel versions. I think Android 12 5.1 and above 5.10 and above. And then the framework that comes through an update to the media provider module. So what happens with fuse pass through is that when the fuse Damon is handling a file open request, and it determines that the requesting app has full access to the requested file, then the system can then subsequently forward all read and write request directly to the lower file system. So, as I mentioned before, what would normally happen? Would that each write, read and write request is forwarded from the fuse driver to the fuse Damon, which introduces a lot of overhead. So by bypassing that forwarding, it improves performance for reading and writing large files in theory, because once you open that large file, you can continue to read and write to it without having to go through the middle man of the fuse. this, of course won't make a lot of difference for reading and writing lots of small files because you still have to read and open it and then do a small read and write command and then open another one. And, you know, they'll still add up, but if you're reading and writing data to a large file on the external shared storage directory, then views pass through should make a difference. It's probably something you haven't heard of before, cuz it's not something really Google announced, but it's something that's there in the Android docs and it's configured in the. But something you may have heard is what Google introduced in Android 13 and what I alluded to before. So in Android 13, they're actually splitting up the read external storage permission into three new permissions called read media audio, read media video, and read media images. So what happens is that apps that are targeting Andrew 13, have to request these individual permissions to access the media files of each. That are owned by other apps. So for example, an app targeting Android 13 that wants to read or modify image files, it doesn't own would need to request the read media images, permiss. Similarly, if they wanted to modify an audio file, it didn't own they'd need the read media audio permission. If you were tracking Android APIs across the years, you might have noticed that these APIs were actually planned or these permissions were actually planned to be introduced in Android 10, but they were removed in Android 10 beta three, because Google wanted to give developers more time to adapt their apps for script storage, but we're several years out. So Google feels comfortable in reintroducing this change into Android. One of the other things that Android 13 did is that it closed the loophole that file managers like a maze used to gain access to the sub directories under Android. So you can no longer use storage access framework to request for access to slash Android slash data or slash Android slash B anymore. I wanted to ask you Raymond, for what are your thoughts on Android 13 closing this loophole? And if you found a workaround that you can tell us,
Raymond:
00:41:46
Right now I, I, or our team haven't came up with a solution on solving this problem. And personally, as a mobile game director in my previous job, actually, I will welcome the decision to close this local because hackers can no longer use the Android phone for long to modify the game. Actually our team is so an amazed for which teaches people how to hack the Android games by using MAs and open JDK on an Android device. On the negative side, it will mean more work and more work around, even more additions, negoti, uh, traditions, negotiation to Google because we want our. Direct via assess API back, but on the positive side, I think if you think in an LS way, LS from the ground up had restricted to almost all files except designated media types, like photos, like new sales videos, except the apple requests use, use their own proprietary. But perhaps some Android, maybe we can still use TP or something like, yeah. MTP is on the USB side. That means you still plug on the computer. You used to plug in the computer to access the whole Android device as like accessing USB fund drive. But on device, on the go, you are seeing the Android cloud system as like a iPhone or something. You only look at your music, your photos, your videos, this on the positive side, but there's still one complain. I personally this, uh, I still miss the ability to play music in the folder. I used to organize music in as photos and rather than scattered all them into one folder and screw it by and four drag bank to form a pay list. I just, I still want to just organize it photo nicely and just throw into music player. And there you go. Yeah, that's all my that's one of my thoughts.
David:
00:44:04
I mean, I think we can all relate to that too. I'm sure we all grew up using either windows or some other traditional file management system and that's just how our brains work. It it's how we think about computers and how they organize data.
Mishaal:
00:44:17
I wanted to ask you a question, David, actually hearing about all the changes that have come so far with Android in relation to file access and file management and all the restrictions that have come. Do you think Android will continue to offer relatively open access to the underlying external file system? Or do you think will become more and more like iOS where you don't really have a sense of what actually how the, the structure is actually.
David:
00:44:45
Yeah, iOS is so totally abstracted. That way apps can create folders in this protected media space, essentially what it is. It's the same thing as Androids media collections. I'm sure in principle, I'm sure it works differently, but if apps can't see anything outside that very, very narrowly scoped storage area. I've run into this so many times on my iPad where I'm just like, I want to see the files, this app I know has access to, but for whatever reason, I can't, because probably those two developers would need to cooperate and say, Hey, I can't see your stuff. Why can't you see it? And they would have to figure out okay. Um, what do I need to change my application to see like Google photos, for example, one of the strange limitations on Google photos, you can only put one Google photo in one iMessage at a time. You can't do multiple Google photos, probably because iOS doesn't support that for third party gallery style applications in the file picker. So you can see the problems, this approach introduces because it inherently limits what you can do with the files too. And explaining that to users. Like, you're never going to get them to understand the nuances, why one application can do one thing and why one application can't do another. And on iOS, that is something I run into all the time. But like even me being fairly casual for an Android enthusiast, this is not the kind of stuff most people run into. And I think that is where Google. Kind of wants to take, at least the Android phone experience is to a place where it's very hard for people to break. That's probably the most important thing. They don't want people in these directories because the chance that they could cause an app to misbehave and then either crash or lose their data, God forbid there's something they can mess up. So yeah, I do see Google continuing to move this direction. And the sad truth is that there's not a lot of incentive for them to preserve. This more traditional file structure access. I mean, can you imagine how many developers out there, especially for things like mass market mobile games, get emails from people trying to like mess around with directory structure and they break something. I'm sure it happens pretty frequently. And not to say that that is Google's problem. Like, you know, users go doing things that break devices all the time, but at the end of the day, Google's concern is of a massive ecosystem. And so they're. What's best for their partners in that ecosystem, their partners are Samsung. Show me all of these major smartphone vendors. And they may say, Hey, we're getting a lot of customer support issues around vis people are really having trouble using file apps. How do we make this simpler? And so Google has to digest that feedback. And as much as we like to think that the Android team is above the fray, in terms of deciding how the platform works. I don't think they totally are. They have to consider these things. And they also have to consider the fact that there are hundreds of millions of people out there who use this and who they would like to use an Android phone one day. And the more they can make it feel seamless in terms of that experience, the easier it is to get somebody to switch platforms. So there's a lot of stuff they have to think about. That's like, I mean, totally beyond the engineering level there, but. Yeah, I'm sure they're considering this. And I would say that the direction we've seen Android go that is they're going toward a more restrictive
Mishaal:
00:48:10
model. Oh. And for those of you at home who didn't immediately realize what David meant when he said this healed up an iPhone to show us. So, uh, that's all the questions that I actually had for this discussion. I mean, I know we went through an absolute ton of information and I hope this is starting to make sense. And I haven't actually confused you more about going through all of this. I mean, trust me, it took me a lot to actually digest and put all this information down and there's still stuff that I could talk about, like more information, but you gotta stop somewhere. And I think a one hour mark is a pretty good place to.
David:
00:48:47
And if this is an interesting topic to you, the entire subject of how storage works on Android and how applications interact with the operating system. That's a great question for a company like Asper to answer because we deal with apps on Android devices, especially unusual ones of varying OS distro, or excuse me, varying OS versions, underlying platform versions all the time. So if you're wonder. Hey, can I have an app that like behaves nicely on Android 12, but also works on my Android nine devices, which I already know it does. We can help you figure some of this out. We can help you understand how the platform has changed. What kind of devices you may need to create the sort of experience you want. And when it comes to things like Android TV, well, how do you get around the issue of not having access to the files app on an Android TV device? That is also one of those sticky questions that you can come to us. And we can see if there's a solution there because we deal with very strange Android devices all the time. We are not a first customizing firmware. We have our own distribution called foundation available for arm and X 86 devices. And this is the kind of thing that if you need an Android device to behave in a particular way, because it has a very specific task associated with it. That's what ER's here for. Check us out@esper.io and Michelle, I'll let you go ahead and, uh,
Mishaal:
00:50:01
do the outro. Thanks, David. And, uh, thank you again, Raymond for joining us. So if people want to follow you in your work, where can they find you? Oh, you can find me
Raymond:
00:50:11
at hab.com/trans love because I love trans music. And you can also find me on our apps, telegram channel ma file manager, which you can find it at the ma
Mishaal:
00:50:26
file manager app. Yeah. And if you're looking for a material themed open source file manager app check out maze file manager on the Google play store. And if you're looking for the content produced by the editorial team at Esper, check us out at blog, doper.io, where you'll find all of our content. Not just our blog articles, not just our newsletter, but also episodes of this podcast, which I'd say thank you again for joining us and listening through an hour long discussion of the nerdiest final like topic I could think of right now. Um, hopefully this really helped you out. It helped me out in piecing this together. So thank you for joining us on another episode of Android bites.
