Hi,
So, I'm trying to make my own custom ROM. Nothing too fancy, just a couple of simple and easy modifications to fit my needs but there's somethings I don't know how they work nor I understand them and I need some help. The only thing I was able to achieve as of yet was to remove some apps (I only tried to remove "Learn More" and QuickOffice to be honest-- but it worked). I just removed the .apk and .odex file directly from the .zip file, signed it and flashed it as usual. It worked fine.
Now, a few questions...
1) I also tried to add a few apps, just like in MoDaCo Custom ROM (MCR), to the /data/app folder instead of /system/app. Actually, I just copied all of .apk files from MCR /data/app into my own (with same folder structure of course). But they didn't get installed... I know I'm missing something and I can almost bet it has something to do with the /META-INF/com/google/android/update-script file. I tried countless times to search for some guide, how-to or just a piece of documentation about this file but I can't find anything. I don't want to just change it and see if it works and then screw things up. Can anyone give me some hints about understanding this file? It has to do with the fact that my /data/apps weren't installed, right?
2) I don't use the Gmail app, just the HTC Mail one. Is it safe to delete Gmail.apk and GmailProvider.apk? I believe Gmail.apk will be fine, but the provider one, I guess it's used by the data synchronization thing. Maybe it's ok if I don't sync my mail (that's what I want actually) but maybe there's something else that I'm missing and the system will not work properly, somehow, without the GmailProvider.apk or is it safe to remove it?
3) When we first install a new ROM, starting clean, there's an introductory guide allowing to setup some things initially. One of those is the language. There is a bunch of languages, specially like English (A), English (B), English © where the letters represent other languages. I don't need all that, I only need English (Portugal). Is there anyway I could remove the ones I don't want from the ROM or I would need to dig into the Android source for that?
3.1) How about adding my own language? Is there a way I could do that to my ROM?
NOTE: Funny thing, the Android system is localized on my country's language when I first start the device but as soon as I change it to English, I cannot change it back to Portuguese cause that option doesn't exist. How come the device is in Portuguese when I first start it up after flashing a new ROM? This also happens every time I swap the SIM card, but I can never select Portuguese language by myself. Why?
4) I noticed that after flashing my custom ROM, after booting the device for the first time, the mobile network was active. This is not good... For now I have 1 month free of mobile networking but that won't last forever. I don't want to waste money every time I'm trying a new ROM. Is there anyway to disable this thing from automatically starting up connected when I first boot?
For now that's it...
1. You need to edit "META-INF/com/google/android/update-script"
That's what I said... :/ But I also said:
I tried countless times to search for some guide, how-to or just a piece of documentation about this file but I can't find anything. I don't want to just change it and see if it works and then screw things up. Can anyone give me some hints about understanding this file?
Click to expand...
Click to collapse
If anyone knows anything helpful of course...
Nazgulled said:
That's what I said... :/ But I also said:
If anyone knows anything helpful of course...
Click to expand...
Click to collapse
I am also interested in a guide or walk through for making a ROM for the Hero.
1) I also tried to add a few apps, just like in MoDaCo Custom ROM (MCR), to the /data/app folder instead of /system/app. Actually, I just copied all of .apk files from MCR /data/app into my own (with same folder structure of course). But they didn't get installed... I know I'm missing something and I can almost bet it has something to do with the /META-INF/com/google/android/update-script file. I tried countless times to search for some guide, how-to or just a piece of documentation about this file but I can't find anything. I don't want to just change it and see if it works and then screw things up. Can anyone give me some hints about understanding this file? It has to do with the fact that my /data/apps weren't installed, right?
Click to expand...
Click to collapse
You're missing copy_dir PACKAGE:data DATA:, so you're really not writing /data/app even if it's on the package. If you read through the update-script, you'll see that it starts by formating BOOT: and SYSTEM: and later on writes the contents of /system to SYSTEM: (with copy_dir) and writes the raw boot.img to BOOT: (with write_raw_image).
Add the line to the very end of the permissions settings (but before format and write boot, most likely, before the show_progress line). You also need to set a specific permission after writing anything to /data/app from an update-script, otherwise your market won't be able to install anything (as you'll have changed the permissions for /data/app through the update-script).
In all, your update script will look like this (assuming a regular update-script, not the mcr 2.9 one):
Code:
...
symlink toolbox SYSTEM:bin/xxxxyyyy
...
set_perm 0 0 04755 SYSTEM:bin/su
copy_dir PACKAGE:data DATA:
set_perm 1000 1000 0771 DATA:app
show_progress 0.1 10
show_progress 0.2 0
write_raw_image PACKAGE:boot.img BOOT:
show_progress 0.2 10
2) I don't use the Gmail app, just the HTC Mail one. Is it safe to delete Gmail.apk and GmailProvider.apk? I believe Gmail.apk will be fine, but the provider one, I guess it's used by the data synchronization thing. Maybe it's ok if I don't sync my mail (that's what I want actually) but maybe there's something else that I'm missing and the system will not work properly, somehow, without the GmailProvider.apk or is it safe to remove it?
Click to expand...
Click to collapse
It's safe to remove both, though the provider doesn't do much without the app, so it's also safe to leave it there.
3) When we first install a new ROM, starting clean, there's an introductory guide allowing to setup some things initially. One of those is the language. There is a bunch of languages, specially like English (A), English (B), English © where the letters represent other languages. I don't need all that, I only need English (Portugal). Is there anyway I could remove the ones I don't want from the ROM or I would need to dig into the Android source for that?
Click to expand...
Click to collapse
Hero roms take their settings from xml files in /system/customize. You'll see a bunch of xmls in there. Some are references to other xmls to be used (mns_map and cid_map), some are default settings (default.xml, COMMON.xml) and some are carrier specific settings (/CID/HTC_*.xml and /MNS/Lan_*.xml). The files are very redundant, and it's hard to tell which one is being used when, though, as a rule, you should understand that Hero loads a default from the /customize folder and then looks at mns_map and cid_map for the file to use inside /CID and /MNS. I'm not yet aware how it chooses the file, but, if you narrow down it's choices, then you can control which ones it uses and modify the settings there. If you crack open the xmls, you'll immediately see the languages defined in there (en_US, en_GB, etc), so removing languages is a matter of deleting the lang/region and adding your own (en_PL?). Again, there's so many xmls in there that it's hard to know which one to use, so either make your own and delete the rest, or edit them all. I'll give you the batch that I use in my /customize, reading through them should help you understand what's going on with those xmls.
3.1) How about adding my own language? Is there a way I could do that to my ROM?
NOTE: Funny thing, the Android system is localized on my country's language when I first start the device but as soon as I change it to English, I cannot change it back to Portuguese cause that option doesn't exist. How come the device is in Portuguese when I first start it up after flashing a new ROM? This also happens every time I swap the SIM card, but I can never select Portuguese language by myself. Why?
Click to expand...
Click to collapse
Look above, it's because the language is obtained from the carrier and used during first boot, but it's not defined in an xml, so it's unavailable, make it available by adding it to the list.
4) I noticed that after flashing my custom ROM, after booting the device for the first time, the mobile network was active. This is not good... For now I have 1 month free of mobile networking but that won't last forever. I don't want to waste money every time I'm trying a new ROM. Is there anyway to disable this thing from automatically starting up connected when I first boot?
Click to expand...
Click to collapse
This I'm not sure about. You could try emptying your /system/etc/apns-conf.xml so that it won't be able to register to any data service (just leave the <apns version="x"> </apns> tags, remove everything in between).
Good luck.
jubeh said:
You're missing copy_dir PACKAGE:data DATA:, so you're really not writing /data/app even if it's on the package. If you read through the update-script, you'll see that it starts by formating BOOT: and SYSTEM: and later on writes the contents of /system to SYSTEM: (with copy_dir) and writes the raw boot.img to BOOT: (with write_raw_image).
Add the line to the very end of the permissions settings (but before format and write boot, most likely, before the show_progress line). You also need to set a specific permission after writing anything to /data/app from an update-script, otherwise your market won't be able to install anything (as you'll have changed the permissions for /data/app through the update-script).
Click to expand...
Click to collapse
That's what I thought but I'm still missing one thing...
The generic HTC "update.zip" has format: DATA for the last line of the update script, which I suppose will delete everything becuse the copy_dir command was executed before.
Should I not use the format data option? Won't that leave traces of a previous installation if I want to start clean? What's the recommendations or suggestions?
jubeh said:
It's safe to remove both, though the provider doesn't do much without the app, so it's also safe to leave it there.
Click to expand...
Click to collapse
What will happen to the data-sync when i tries to sync the data and finds the provider for gmail is not there? Will it simply be ignored? Will there be any errors changing any of the options (specially the gmail one) on the data synchronization settings?
jubeh said:
Hero roms take their settings from xml files in /system/customize. You'll see a bunch of xmls in there. Some are references to other xmls to be used (mns_map and cid_map), some are default settings (default.xml, COMMON.xml) and some are carrier specific settings (/CID/HTC_*.xml and /MNS/Lan_*.xml). The files are very redundant, and it's hard to tell which one is being used when, though, as a rule, you should understand that Hero loads a default from the /customize folder and then looks at mns_map and cid_map for the file to use inside /CID and /MNS. I'm not yet aware how it chooses the file, but, if you narrow down it's choices, then you can control which ones it uses and modify the settings there. If you crack open the xmls, you'll immediately see the languages defined in there (en_US, en_GB, etc), so removing languages is a matter of deleting the lang/region and adding your own (en_PL?). Again, there's so many xmls in there that it's hard to know which one to use, so either make your own and delete the rest, or edit them all. I'll give you the batch that I use in my /customize, reading through them should help you understand what's going on with those xmls.
Click to expand...
Click to collapse
I'm familiar with XML files and have no problems editing them but this whole thing was a little confused specially because there are no documentation and like you said, it's hard to tell which one is which.
I know you kinda suggested that but I'm no seeing how to do it (maybe with the file you attached?) but, how could I start clean, only have default XML files and nothing else, no carrier specific stuff, no bulk, just plain default settings that I can tweak to my own liking.
Any hints on how to do that, or it's hard to tell and you don't know how to do it either?
jubeh said:
This I'm not sure about. You could try emptying your /system/etc/apns-conf.xml so that it won't be able to register to any data service (just leave the <apns version="x"> </apns> tags, remove everything in between).
Click to expand...
Click to collapse
Unfortunately, this did not work
The generic HTC "update.zip" has format: DATA for the last line of the update script, which I suppose will delete everything becuse the copy_dir command was executed before.
Should I not use the format data option? Won't that leave traces of a previous installation if I want to start clean? What's the recommendations or suggestions?
Click to expand...
Click to collapse
I've never understood why the default update script does that, but yeah, format DATA: will erase userdata available. It might not be such a bad thing if you hadn't written anything to it before. For a wipe before install, move format DATA: (and, optionally, also add format CACHE: ) right after format SYSTEM: and before you start writing anything to disk. It should give you a clean install without having to wipe from the recovery menu.
What will happen to the data-sync when i tries to sync the data and finds the provider for gmail is not there? Will it simply be ignored? Will there be any errors changing any of the options (specially the gmail one) on the data synchronization settings?
Click to expand...
Click to collapse
I've not tried it yet. Worst case scenario, you'll get the current problem we have with eclair builds (sync will just keep running and eat out battery because it's looking for a service). That's why I'd suggest leaving gmail provider there, it's just a broadcast service, but without an app to receive the data, it'll do nothing. If your concern is data usage due to sync, then you should turn off gmail sync in settings, that way only calendar and contacts are synchronized (although you'll still be using data).
I'm familiar with XML files and have no problems editing them but this whole thing was a little confused specially because there are no documentation and like you said, it's hard to tell which one is which.
I know you kinda suggested that but I'm no seeing how to do it (maybe with the file you attached?) but, how could I start clean, only have default XML files and nothing else, no carrier specific stuff, no bulk, just plain default settings that I can tweak to my own liking.
Any hints on how to do that, or it's hard to tell and you don't know how to do it either?
Click to expand...
Click to collapse
The files I attached are pretty generic. It's meant to replace your /customize folder, not add to it, with the exception being /resources inside /customize, that one I use a lighter version (no footprints) but I didn't include it in the archive for size. I have no carrier stuff there, just settings for language (I think I made it all en_US, replace or add what you need) and layouts for the different scenes, some mail app behaviors (added back the Yahoo! mail and AOL mail options, I think), some footprint, stock, and weather information. Again, it's meant to replace your /customize directory and the smaller amount of files makes it way easier to edit.
Unfortunately, this did not work
Click to expand...
Click to collapse
Blame the xmls in /customize for that. I mentioned earlier I don't exactly know how the system picks which xmls to use, I think it looks at your carrier, which is then assigned a number (for example, let's assume T-Mobile is given number 238, so xmls with 238 on them will be loaded). Problem is inside those xmls are also apn settings (which I've removed from the files I provided so that the system looks instead for apn settings in apns-conf.xml in /etc).
You could flash your system erasing the whole /customize directory (and removing all mentions of it and it's sub-directories from the update-script so that install doesn't fail). Doing that will give you ALL languages available to the rom, but you'll also lose all your pre-loaded settings (like scenes, weather locations, footprints, stocks) and HTC Setup will take an ungodly amount of time to start (about 4-5 minutes after it's done dexopting) while it builds it's own default settings. It's another route you could take, though. Experiment and you'll eventually hit what you want/need.
OK, I guess I'll have to waste some time testing all this stuff, but I suppose I'll leave that to some other day and do more basic things for now.
But just to be sure, if I use your "customization" I'm basically starting from scratch (clean, defaults) and I can use the "old" customization stuff to understand how am I supposed to configured everything and adapt the clean customization folder to my needs. Right?
jubeh said:
You could flash your system erasing the whole /customize directory (and removing all mentions of it and it's sub-directories from the update-script so that install doesn't fail). Doing that will give you ALL languages available to the rom, but you'll also lose all your pre-loaded settings (like scenes, weather locations, footprints, stocks) and HTC Setup will take an ungodly amount of time to start (about 4-5 minutes after it's done dexopting) while it builds it's own default settings. It's another route you could take, though. Experiment and you'll eventually hit what you want/need.
Click to expand...
Click to collapse
This gave me an idea, dunno if it could work...
What if I did that, erase everything inside customize, let the system build its own default settings and then access that directory (through ADB maybe?) and see what the system has created. It should be much more cleaned, without all those carrier XMLs and stuff. Then I could start from there, what do you think?
I have a few more questions about the update-script if you don't mind, but I'm very tired to write them and discuss them now. I'll leave them for another day if you don't mind coming back here and help me out some more.
Thanks for all your help so far
I should probably have mentioned, some of the stuff on those xmls is pretty straightforward (setting language, region, weather, footprints, etc), but a lot of it (setting up default shortcuts, default widgets, system settings) requires working knowledge of android, or at least of how activities are handled and how xmls come into play and how android references the different activities in a package.
On a different note, the xmls are read-only configuration files, actually, anything in system is read-only data (program data, configuration data, resource data), so if you don't have a customize directory the system won't recreate it. What the system does do during normal boot (factory) is to take those carrier-specific settings and apply them to a file(s) in /data where configuration settings are ultimately written/re-written and accessed at will. The xmls are just a database of the settings that will ultimately get picked. When you don't have a /customize directory the system writes the settings straight to -rw enabled /data partition based on things like your current carrier, carrier provided information for locale, etc.
I don't think you can take what's written to /data and use it on a default install package, well, you could, but it's much easier (and a more correct approach) to learn how those xmls in /customize work.
jubeh said:
(...)but it's much easier (and a more correct approach) to learn how those xmls in /customize work.
Click to expand...
Click to collapse
I wouldn't mind doing that if there was some kind of documentation... As it is right now, it's too confusing and there are too many XMLs to begin with...
jubeh said:
I've not tried it yet. Worst case scenario, you'll get the current problem we have with eclair builds (sync will just keep running and eat out battery because it's looking for a service). That's why I'd suggest leaving gmail provider there, it's just a broadcast service, but without an app to receive the data, it'll do nothing. If your concern is data usage due to sync, then you should turn off gmail sync in settings, that way only calendar and contacts are synchronized (although you'll still be using data).
Click to expand...
Click to collapse
But if I leave the gmail provider there and do not turn off gmail sync in settings, my e-mail will still be synced right? There's just no Gmail app to view them but they are still going to be downloaded, correct?
Ok, I finally got a change to test the Gmail/GmailProvider thing and it didn't look like there any problems like trying to sync and draining the battery. Actually, if auto-sync was enabled, there was always a message saying the mail was never synced and no matter how many times you check/uncheck it, it wouldn't sync. If auto-sync was disabled, you could press the gmail button many times but nothing happened, no sync at all (of course).
I think this behavior is fine and I can safely delete the Gmail and provider from the phone without worrying that something will go wrong
Ok, now my other questions that I said I was going to post later...
As we talked about, you suggested to add a format data and cache right after a format to system. Comparing that to the MoDaCo ROM (and other ROMs too) it's common for them to, instead of formatting data, they do this:
Code:
delete DATA:app
delete DATA:init.sh
delete DATA:local
delete DATA:dalvik-cache
copy_dir PACKAGE:data DATA:
set_perm 1000 1000 0771 DATA:app
Why are they doing this? If they are not formatting, it means there's something inside data that they want to keep. Have any idea what and why?
The other question I had about the update-script is that the following two lines are found in the original HTC update but not on MoDaCo's:
Code:
symlink dumpstate SYSTEM:bin/bugreport
symlink dumpstate SYSTEM:bin/dumpcrash
Are they relevant somehow? Should I just ignore and leave them there or remove them just as MCR did, for some reason?
There's a file named build.prop inside the system folder. I've already searched countless times and can't find any proper documentation about this file... It seems it can hold various default properties for the system and it would be nice to know what I can configure there. Maybe I could workaround problem #4 (top of the topic) in this file? Maybe some file in the Android source has comments about the possible properties on this file? Do you have any idea?
One last thing...
I've read about odex and dex files but found it a little confusing... All I know is that original HTC updates come mostly with .apk files and a corresponding .odex file. But most (if not all) custom ROM's, do not have the .odex file but have a classes.dex inside the apk. What's the real difference between the two? And why custom ROMs prefer to have the .dex file instead of the .odex, is it better somehow for some reason? If so, what should I do to "change" the .odex files into classes.dex ones and put them inside the .apk files?
As we talked about, you suggested to add a format data and cache right after a format to system. Comparing that to the MoDaCo ROM (and other ROMs too) it's common for them to, instead of formatting data, they do this:
Code:
delete DATA:app
delete DATA:init.sh
delete DATA:local
delete DATA:dalvik-cache
copy_dir PACKAGE:data DATA:
set_perm 1000 1000 0771 DATA:app
Why are they doing this? If they are not formatting, it means there's something inside data that they want to keep. Have any idea what and why?
Click to expand...
Click to collapse
Yeah, I mentioned on the earlier post that doing "format DATA:" and "format CACHE:" would be the equivalent of running "Wipe data/factory reset" from the recovery menu since you mentioned about making a build that installs fresh. Paul's (MoDaCo) roms are no-wipe, so they preserve that program data and instead format all other directories inside /data that might be problematic during an upgrade; his roms use apps2sd, so he assumes that the user's downloaded apps are in mmcblk0p2 (the memory card's ext partition), so by formatting /data/app, he's only erasing the apps he included in his update package, if the user doesn't use a2sd, well, this erases their downloaded apps (one of the reasons I dislike the current a2sd implementation, it's not suited for all situations). Then he deletes data/init.sh, which is basically a script he includes in /data that runs during a rom's boot and starts things like a2sd, compcache, linux-swap, and other custom rom's "advanced" features. DATA:local is also app storage (I believe) and functions much like in windows' /users/x/application data/local, it can be problematic if not done between builds. Last, he erases /data/dalvik-cache, this is where the temporary application code resides (the extracted classes.dex from inside an apk) for rapid execution (for apks that haven't been pre-odexed). If you wish to make your rom no-wipe, add those lines instead of "format DATA:", but I'd remove "delete DATA:app" since it's a bit problematic for people who don't use a2sd.
The other question I had about the update-script is that the following two lines are found in the original HTC update but not on MoDaCo's:
Code:
symlink dumpstate SYSTEM:bin/bugreport
symlink dumpstate SYSTEM:bin/dumpcrash
Are they relevant somehow? Should I just ignore and leave them there or remove them just as MCR did, for some reason?
Click to expand...
Click to collapse
dumpstate is a debugging tool that android uses, but I don't know exactly where or how (I think it's an HTC-specific log for their HTCLog.apk). It contains two tools, dumpcrash and bugreport, much like toolbox is a binary that contains all the tools you see symlinked to it in update-script (bin, df, cp, mv, ls, insmod, etc.). It's not a problem at all if you remove it, but it's not a problem either to leave it there. Whenever something is symlinked, though (if it's something you added, for example, busybox tools), you have to make sure that the binary or a link to it or a previous symlink to it doesn't exist, otherwise your update-script will fail (like symlinking toolbox's insmod and then busybox's insmod, or busybox's ping if a ping binary already exists in the target symlink directory (for example /system/bin)).
There's a file named build.prop inside the system folder. I've already searched countless times and can't find any proper documentation about this file... It seems it can hold various default properties for the system and it would be nice to know what I can configure there. Maybe I could workaround problem #4 (top of the topic) in this file? Maybe some file in the Android source has comments about the possible properties on this file? Do you have any idea?
Click to expand...
Click to collapse
I guess that could work if you modify network type (there's a property somewhere in there about HSXPA, I think), but like you said, there's no documentation so it calls for some experimenting.
Those properties are also overrides, so, for example, if your rom doesn't do adb root (because ro.secure is set to 1 in init.rc), you can add ro.secure=0 in build.prop to enable adb root again.
Most properties are self explanatory, and pretty much, if you can't tell what it is, you probably shouldn't change it (for example, build.fingerprint allows you to change what kind of apps Market presents to you).
I'd say, don't mess around too much with the first batch (default build properties), with the exception of build.fingerprint.
One last thing...
I've read about odex and dex files but found it a little confusing... All I know is that original HTC updates come mostly with .apk files and a corresponding .odex file. But most (if not all) custom ROM's, do not have the .odex file but have a classes.dex inside the apk. What's the real difference between the two? And why custom ROMs prefer to have the .dex file instead of the .odex, is it better somehow for some reason? If so, what should I do to "change" the .odex files into classes.dex ones and put them inside the .apk files?
Click to expand...
Click to collapse
The classes.dex files is the dalvik bytecode for the application. It contains everything inside /appnamespace/src and /appnamespace/R if you're working from the source. It's basically all the java code already compiled for the dalvik virtual machine.
Dalvik doesn't do jit, so the classes.dex have to be extracted (if they're not already) and run through the virtual machine in a process called dexopt that produces code that can be much more rapidly excecuted by the dvm (dalvik virtual machine). The extracted code is then placed in /data/dalvik-cache (mentioned earlier) as [email protected]@[email protected]. All this pre-cached code eats away at your space in your /data partition (where you store your system data, like your email, messages, browser cache, and downloaded applications) and can be quite a big chunk of space gone (framework classes specially, since all jars inside /system/framework are nothing but java code, about 20MB for all framework code).
The solution to prevent usage of /data storage on an initial build is to have the code pre-extracted. This is done at compile-time running the created system through an emulator and then creating the odex files, which is almost exactly the same as the *@classes.dex found inside /data/dalvik-cache except that it's much better aligned to the device's memory configuration and offers (marginally) better excecution, plus when the package manager first encounters the apk, it's first step is to check if an .odex of it is available, if it is, then it skips checking and dexopting a classes.dex inside it and proceeds to the next package.
Odex files increase the space in your /system partition (it was already read-only, so it's not like you're missing out on something) so those 20 MB of framework that would have been tossed into your userdata partition are now out of the way into the useless (in an available-storage sense) /system partition so you have more free space for your apps.
There's a tool that can create odex files for apks that don't have them, it's called dexopt-wrapper and it has to be run from a running phone. However, this tool won't strip the (now unnecessary) classes.dex from inside an apk, so it has to be done manually. Removing the classes.dex once you've created an odex is a good idea merely on the basis of space savings. However, once a classes.dex has been odexed, it's very hard (used to be impossible) to turn it back to a classes.dex and place it inside an apk.
Usually, us rom cooks preffer the classes.dex inside the apk because odexes create conflicts when you try to mix and match apps between builds. Since the odex files are tied to the framework where they were created, adding an apk with it's odex to another framework (even one that's only ever-so-slightly different) will be very problematic (the odex code is no longer protected or verified, so the rom won't even boot because it will fail verification for all available packages).
The tool for reverting odex files is called deodexerant by jesus freke. It requires working knowledge of how the system goes through the dexopting process so you can run it in the proper order.
Another thing to know is that even if your /system is pre-odexed, your downloaded apps will still be run through the package manager and have their classes.dex dexopted and placed in /data/dalvik-cache.
Oh, and never, ever put odexes anywhere but /system/framework or /system/app (or /data/app_s if you use a2sd). The package manager currently can't handle odexes in /data/app or /data/app_private (the other two locations where it looks for packages other than /system/app).
In the end, whether to use/have odexes comes down to personal choice (and available space). There's two hero builds, those based in 2.73.405.61 and 2.73.405.66, that have not been pre-odexed, so they're perfect for experimenting, mix/matching, etc...
jubeh said:
Yeah, I mentioned on the earlier post that doing "format DATA:" and "format CACHE:" would be the equivalent of running "Wipe data/factory reset" from the recovery menu since you mentioned about making a build that installs fresh. Paul's (MoDaCo) roms are no-wipe, so they preserve that program data and instead format all other directories inside /data that might be problematic during an upgrade; his roms use apps2sd, so he assumes that the user's downloaded apps are in mmcblk0p2 (the memory card's ext partition), so by formatting /data/app, he's only erasing the apps he included in his update package, if the user doesn't use a2sd, well, this erases their downloaded apps (one of the reasons I dislike the current a2sd implementation, it's not suited for all situations). Then he deletes data/init.sh, which is basically a script he includes in /data that runs during a rom's boot and starts things like a2sd, compcache, linux-swap, and other custom rom's "advanced" features. DATA:local is also app storage (I believe) and functions much like in windows' /users/x/application data/local, it can be problematic if not done between builds. Last, he erases /data/dalvik-cache, this is where the temporary application code resides (the extracted classes.dex from inside an apk) for rapid execution (for apks that haven't been pre-odexed). If you wish to make your rom no-wipe, add those lines instead of "format DATA:", but I'd remove "delete DATA:app" since it's a bit problematic for people who don't use a2sd.
Click to expand...
Click to collapse
So, to make it clear, I could:
Option A) Add format DATA and format CACHE to my ROM and this would make my ROM to automatically wipe everything and start clean.
Option B) I could not add anything (no formats and no deletes) and warn the users to explicitly perform a wipe before flashing.
Option C) Add delete DATA:local and DATA:dalvik-cache and have a no-wipe ROM where the apps would be preserved from the last ROM, right?
Ok, I don't use a2sd and for now have no intention in doing so. My best guess is to start with option A for now and maybe opt by using some app to backup everything and restore everything back after flashing a wipe ROM. Or I could go with option C and everything could work fine but there's a chance I could run into some issues in the long run.
jubeh said:
The classes.dex files is the dalvik bytecode for the application. It contains everything inside /appnamespace/src and /appnamespace/R if you're working from the source. It's basically all the java code already compiled for the dalvik virtual machine.
Dalvik doesn't do jit, so the classes.dex have to be extracted (if they're not already) and run through the virtual machine in a process called dexopt that produces code that can be much more rapidly excecuted by the dvm (dalvik virtual machine). The extracted code is then placed in /data/dalvik-cache (mentioned earlier) as [email protected]@[email protected]. All this pre-cached code eats away at your space in your /data partition (where you store your system data, like your email, messages, browser cache, and downloaded applications) and can be quite a big chunk of space gone (framework classes specially, since all jars inside /system/framework are nothing but java code, about 20MB for all framework code).
The solution to prevent usage of /data storage on an initial build is to have the code pre-extracted. This is done at compile-time running the created system through an emulator and then creating the odex files, which is almost exactly the same as the *@classes.dex found inside /data/dalvik-cache except that it's much better aligned to the device's memory configuration and offers (marginally) better excecution, plus when the package manager first encounters the apk, it's first step is to check if an .odex of it is available, if it is, then it skips checking and dexopting a classes.dex inside it and proceeds to the next package.
Odex files increase the space in your /system partition (it was already read-only, so it's not like you're missing out on something) so those 20 MB of framework that would have been tossed into your userdata partition are now out of the way into the useless (in an available-storage sense) /system partition so you have more free space for your apps.
There's a tool that can create odex files for apks that don't have them, it's called dexopt-wrapper and it has to be run from a running phone. However, this tool won't strip the (now unnecessary) classes.dex from inside an apk, so it has to be done manually. Removing the classes.dex once you've created an odex is a good idea merely on the basis of space savings. However, once a classes.dex has been odexed, it's very hard (used to be impossible) to turn it back to a classes.dex and place it inside an apk.
Usually, us rom cooks preffer the classes.dex inside the apk because odexes create conflicts when you try to mix and match apps between builds. Since the odex files are tied to the framework where they were created, adding an apk with it's odex to another framework (even one that's only ever-so-slightly different) will be very problematic (the odex code is no longer protected or verified, so the rom won't even boot because it will fail verification for all available packages).
The tool for reverting odex files is called deodexerant by jesus freke. It requires working knowledge of how the system goes through the dexopting process so you can run it in the proper order.
Another thing to know is that even if your /system is pre-odexed, your downloaded apps will still be run through the package manager and have their classes.dex dexopted and placed in /data/dalvik-cache.
Oh, and never, ever put odexes anywhere but /system/framework or /system/app (or /data/app_s if you use a2sd). The package manager currently can't handle odexes in /data/app or /data/app_private (the other two locations where it looks for packages other than /system/app).
In the end, whether to use/have odexes comes down to personal choice (and available space). There's two hero builds, those based in 2.73.405.61 and 2.73.405.66, that have not been pre-odexed, so they're perfect for experimenting, mix/matching, etc...
Click to expand...
Click to collapse
Well, I'm currently basing my ROM on 2.73.405.38 as the .66 one has "test" on the name and it feels like a beta version, which I'd rather not use. The .38 version uses .odex files for most of the packages (only a few has classes.dex inside) and I guess I'll leave them like that. If future ROMs, like that .66 one, comes with classes.dex vs .odex files, well, then I'll just leave it like that...
That looks like a very important thing in Android apps and at the same time something that I should be very knowledgeable about before messing with it, so I'll just stay put
Thanks once again for taking your time with such complete answers
So, to make it clear, I could:
Option A) Add format DATA and format CACHE to my ROM and this would make my ROM to automatically wipe everything and start clean.
Option B) I could not add anything (no formats and no deletes) and warn the users to explicitly perform a wipe before flashing.
Option C) Add delete DATA:local and DATA:dalvik-cache and have a no-wipe ROM where the apps would be preserved from the last ROM, right?
Ok, I don't use a2sd and for now have no intention in doing so. My best guess is to start with option A for now and maybe opt by using some app to backup everything and restore everything back after flashing a wipe ROM. Or I could go with option C and everything could work fine but there's a chance I could run into some issues in the long run.
Click to expand...
Click to collapse
Sounds like you got it.
Well, I'm currently basing my ROM on 2.73.405.38 as the .66 one has "test" on the name and it feels like a beta version, which I'd rather not use. The .38 version uses .odex files for most of the packages (only a few has classes.dex inside) and I guess I'll leave them like that. If future ROMs, like that .66 one, comes with classes.dex vs .odex files, well, then I'll just leave it like that...
That looks like a very important thing in Android apps and at the same time something that I should be very knowledgeable about before messing with it, so I'll just stay put
Click to expand...
Click to collapse
Test only refers to the build being signed with test-keys instead of release-keys. In my testing, it makes no difference whatsoever for a rooted user, and there's many theories floating around about signatures, I haven't researched signatures a whole lot because test-keys have worked fine enough for me, so I'm not so sure, but they seem to be working fine for all.
The reason those releases are un-odexed (and signed with test-keys) is because of the problem with odexes I mentioned earlier. When HTC is testing their rom packages and they need to make, say, a framework change, all the apps with odexes would fail as the framework has changed, so, for testing purposes, it's best to leave the apks and jars in the build full (with classes.dex inside). Once you know your build is solid and you won't be changing much, it's safer to odex the whole thing (sort of like me, I'm waiting for the final release of eclair for ADP1 so I can put out a Dream rom odexed and fully working).
Thanks once again for taking your time with such complete answers
Click to expand...
Click to collapse
No problem, I get bored at work
So, version .66 is not beta or anything? I know MoDaCo has his latest ROM based on that and you're saying they are only different in the signing key, still, it's somewhat a "leaked" build, not quite "official".
But hey, if others are using it without any issues, maybe I should be using it too
On second thought...
1) I can't find a pre-rooted .66 version and I don't know how to root it :/
2) There's lots of files in system/xbin that I don't see referenced anywhere (not even in the update-script) and I don't know what should I do with them.
I'm finding it much more confusing to base my ROM on .66 than on .38...
It is a beta, of sorts, but between it and the final shipping product there's not much difference. It's just much, much easier to work with it.
1.) The simplest form of root is to change ro.sercure to 0 in default.prop in the boot ramdisk. Adding a su binary/Superuser.apk is optional, those you can pull from one of paul's builds.
2.) The system/xbin is added to the boot classpath in the boot ramdisk in init.rc. The only mention of it in in update-script is the permissions for the whole folder and then just a couple permissions for some tools, not all of them.
Hi there,
I'm programming something for Windows Mobile 2003. It basically removes and creates files around. One of the files it deletes is special file.reg, which is normally picked up during the hard-reset.
The file gets deleted using "DeleteFile" and very soon after, I force a hard-reset.
The problem is, the special file.reg is deleted from the file system tree, but it is apparently still available from the hard-reset.
I was wondering if there was another function I had to call to "flush" the file system? If not, I need to find a good trick to 1) enforce the file deletion (like rename first, then delete) and to 2) flush current ghost files left around.
The file system on those persistent drives are fat12 and/or fat16.
Thanks in advance for any input,
Simon
The hard reset returns the machine to its first power up state. If 'file.reg' was part of the original build, then the hard reset will restore it from ROM.
Also any programs you have installed to run on startup, will also be lost, so it is going to be a little difficult, if not impossible, to get rid of this file programatically.
Right, but I'm not talking about the ROM.
I'm talking about the persistent memory which are mounted as \Platform and \Application on this device. Those are FAT12 and/or FAT16, and files deleted (normally) do not come back.
In this case, the files are not coming back, they are not hidden either, but the hardreset process is able to pick them up somehow.
I mean, I call it the problem of the "Ghost files", because they are supposed to be nowhere, but they are found during during the hardreset.
(The files are not recreated, they are still not there, but their contents gets loaded. The info in them cannot be placed in ROM as it contains stuff that changes often)
They come back after a hard reset because during cold boot, they're being copied there from the rom or being created by the system. You may be able to delete them afterwards, but the only way to prevent them from being formed will be to re-cook the rom and stop them from being copied/created during boot-up.
The files DO NOT come back, it's gone, I cannot re-delete it. But somehow it is "read" by a program during the hard-reset.
This file is not part of the ROM, it's part of the persistent memory that doesn't get wiped out upon hardreset but is read-write.
I need to wipe out the ghost file that is stuck on the read/write partition... and I need a way to avoid these being created!
I understand that, but clearly the file is being created (or copied from rom) during cold re-boot, otherwise you wouldn't see it coming back. There isn't going to be any easy way to prevent that, unless you can re-cook the rom, or include some sort of user customization that would delete the files prior to using the device. There are lots of ways that the rom could create the files and put them onto persistant storage.
Hi Farmer Ted,
this is not a ROM issue, recooking the ROM will not help. The problem is a FAT12 or FAT16 filesystem that has bogus data in it.
The problem is most likely a bug in the program that reads the persistent folders... It probably reads it in a way that goes around the change made by DeleteFile()...
Changing that program is not possible (in ROM and I don't have the source, it's also necessary). I just need to make sure it can't find the file I've deleted on the persistent directory (not in ROM).