Special Thanks
Abusalza (for the most initial start off guide)
Cmonex (for the “MOST” important finishing touches)
!Aman! (for all the testing and Hex edit helping)
Noonski (for being the inspiration to keep going )
Ervius (for developing the kitchen tool to perform all the operations)
In this forum there are many many tools from experts and likes for porting XIP, rebuilding dumped ROMs etc. This threads aims at showing or sharing what goes in the background of these automated tools and also aims at answering all the many unanswered questions about various factors of ROM cooking / editing I have come across in this forum
Suggestions / comments always welcome to make these tutorials even better
Index of Tutorials
Manual XIP Porting Guide: CLICK HERE
XIP Porting Updates from members: CLICK HERE
XIP Porting for Himalaya devices & others (Nokser): CLICK HERE
Misc XIP Updates: CLICK HERE
PagePool Changing Guide (for Diamond & Raphael): CLICK HERE
Gain More Storage Memory (Increase imgfs size) Guide: CLICK HERE
ULDR Partition Size Reduction Guide: CLICK HERE
MBR and MSFLSH50 Regions Screenshots: CLICK HERE
Gain More Storage Memory (compress imgfs) with LZX algorithm: CLICK HERE
Get High File System Index (!Aman!): CLICK HERE
Ervius's GUI kitchen thread to perform all operations, Noonski's amazing RunCC & AutoRun & SDAutorun tutorial thread
Ervius's post on patching nk.exe to change the EndRam address for more available RAM in device (original credits to cmonex )
Da_G's amazing new initiative to utilise the ULDR partition to upgrade ROM without re-flash
All the above guides and updates are compiled in pdf file also for offline reading, attached in this post as All Guides.zip
The imgfs Gain.zip is actually the 5th guide with pictorial seperatly put up for members who would want to refer only to that process
The Pictorial.zip is the 7th picture reference for offline reading
Donations for this hard work and research are much appreciated. Below are the links whom you may choose to provide those to
Donation to Abusalza, Donation to Cmonex, Donation to !Aman!, Donation to Ameet
Index of Threads (Manila related)
Ameet's Mode9 script editing ideas thread
l3v5y's tutorial thread for editing Manila files
NisseDILLIGAF's Manila Hash tool
Manual Full XIP Porting
Tools you need: (attached the tools in this thread for easy access)
HEX Calculator (recommended – HEX workshop (Not Free)), suggested Windows Calculator
XIPPort.exe
M’Reloc.exe
NBMerge.exe
Insert.exe
OS.nb.payload from 19965 build (shipped ROM)
Cup of nice strong Coffee (A Must)
Brief:
There are many different ways to port the XIP. Few mention of using the 723*.dsm for the build number, few others mention of using the coredll.dll module to have the latest build numbers. As my friends, Noonski and !Aman! always say “Only numbers are just eye wash, core system is what matters the most” Based on this as inspiration, I am posting this guide for manual XIP porting. A few places you may find colors in this guide, these are to visually link the data for easy understanding
The only files removed from the ported XIP are (these are removed to keep the new XIP within the original size):
osaxst0.dll + osaxst0.dll.imageinfo.txt
hd.dll + hd.dll.imageinfo.txt
bmui.nb0 + bmui.nb0.imageinfo.txt
Process:
Prepare OEMXip base
Dump your original XIP.bin (from 19965 build) with XIPPort.exe, and click “write maps” to get MAP.txt in the OUT folder
Open the MAP.txt and go through what you will need to achieve for a full port. I advice to keep this MAP.txt as a backup, just in case
Click “make pkgs” to get “OEMXipKernel” and “MSXipKernel” folders inside \Files and \Modules
Delete MSXipKernel folders from \Files and \Modules both
Now our base OUT folder is ready with OEMXipKernel
Prepare MSXip donor
Dump your donor XIP.bin (from 20758 build) with XIPPort.exe, and click “make pkgs” to get “MSXipKernel” folder inside \Files and \Modules
Delete osaxst0.dll + osaxst0.dll.imageinfo.txt, hd.dll + hd.dll.imageinfo.txt and bmui.nb0 + bmui.nb0.imageinfo.txt to get the new XIP within the original RAM size. If you don’t wish to delete these files, then you will need to increase the “physlast” in ROMHDR.txt. Process of which is not covered under this guide
Copy the MSXipKernel folders from \Files and \Modules both to the \Files and \Modules in the base OUT folder
Now our OUT folder is ready to be ported with the OEMXipKernel and MSXipKernel
Now to proceed with the reallocing you need to re open the packages which have been created. Open XIPPort.exe and click "undo" then click “realloc P” to re calculate the reallocation addresses. Then click “write maps” to get the new MAP.txt file
Open this MAP.txt and look in the o32_realaddr and e32_vbase addresses. Busenum.dll must be the last entry in both tables. Here you may find overlaps of the modules in a few or most places (seen as !!!!!!!!!!!!!!!!!!)
These are the overlaps which need to be taken care of by reallocating the modules in Initialized Data and Virtual Base addresses
You need to work our way up from the bottom of the list since the busenum.dll is reallocated at the last address of the memory
For example:
03f4c000 03fe3000 L00097000 Virtual base address of coredll.dll
03fe2000 03fe3000 L00001000 !!!!!!!!!!!!!!!!!!
03fe2000 03ff0000 L0000e000 Virtual base address of certmod.dll
03ff0000 03ffb000 L0000b000 Virtual base address of cachefilt.dll
03ffa000 03ffb000 L00001000 !!!!!!!!!!!!!!!!!!
03ffa000 04000000 L00006000 Virtual base address of busenum.dll
Meaning, e32_vbase address of cachefilt.dll is overlapping that of busenum.dll by 1000 (L00001000) Similarly e32_vbase address of coredll.dll is overlapping that of certmod.dll by 1000 (L00001000)
I recommend you use M’Reloc.exe for reallocating the addresses in imageinfo.bin and Notepad to reallocate the addresses in the corresponding imageinfo.txt files. Since the binaries (S000, S001...) must actually be relocated using M'Reloc, it is not enough to just adjust the values in the imageinfo.txt files
To calculate the revised addresses (in below example, the e32_vbase) of the overlapping module, open Hex Calculator. To do that you will need to know the e32_vsize of the overlapped module. To find that out open overlapped module (for e.g. cachefilt.dll) in M’Reloc.exe and see the e32_vsize (0000B000)
Now to correct the e32_vbase of cachefilt.dll, follow this calculation as a base (e32_vbase busenum.dll - e32_vsize cachefilt.dll = e32_vbase cachefilt.dll)
Meaning, (03FFA000 – B000 = 03FEF000) hence the correct e32_vbase address is 03FEF000
03ff0000 03ffb000 L0000b000 Virtual base address of cachefilt.dll
03ffa000 03ffb000 L00001000 !!!!!!!!!!!!!!!!!!
03ffa000 04000000 L00006000 Virtual base address of busenum.dll
Now since the cachefilt.dll is reallocated using the above calculation, the modules next in line above that will also have to be reallocated. Namely, certmod.dll (although not overlapping yet above the cachefilt.dll). To calculate the e32_vbase of certmod.dll you will need the revised e32_vbase address of cachefilt.dll which you got just now
I recommend writing down the e32_vbase, e32_vsize, o32_realaddr and o32_vsize of each module so it will be easier to calculate the correct addresses for reallocation)
Remember, you need to work our way up from the bottom of the list since the busenum.dll is reallocated at the last address of the memory
To reallocate the addresses for o32_realaddr, follow the above calculation, only this time replace the e32_vbase busenum.dll with o32_realaddr and e32_vsize with o32_vsize
Now open the corresponding imageinfo.txt file for each module and change the e32_vbase and o32_realaddr address values in the txt file of the values mentioned with V= and D=, seen for e.g. like this
Module name: cachefilt.dll
e32_vbase: V=03FEF000
o32[1].o32_realaddr: D=01FFE000
You will notice that the FLASHDRV.DLL module has the realaddr at 2 regions. Although I have not found a way to calculate the difference between both regions but I change the values as per Abusalza’s MAP.txt
o32[1].o32_realaddr: D=01FCC000
o32[3].o32_realaddr: D=01FD4000
Since the OEMXipKernel modules never change, I only correct values of the ported MSXipKernel modules
This is helpful if the MSXipKernel modules ported from donor ROMs are similar in the sizes. If not then you will need to do the calculation and correction of values
Once through with the address reallocation, open XIPPort.exe and click “realloc P” to re calculate the addresses for writing maps. It will show you errors regarding some regions, ignore those and click “write maps”. Open the new MAP.txt and recheck for (!!!!!!!!!!!!!!!!!!) If none found that means the XIP has been ported well
Now click “build xip_out.bin” to create the resulting XIP to be inserted into the ROM .payload file. Use this command for inserting the xip_out.bin into the .payload (presuming you already have the shipped OS.nb.payload file in the same working folder
insert.exe -i xip_out.bin -o OS.nb.payload -d 0x00320000 -s 0x004C0000
Check these values with your device imgfs since in Diamond the XIP starts at 0x00320000 and the imgfs starts at 0x007A0000, but for some reason the imgfs signature in Diamond is at 0x007E0000
Build OS.nb for use in the ROM folder from the .payload you just updated with the new XIP. Please note these commands are for Diamond device. Please check with your device on the same before building
nbmerge.exe –kaiser OS.nb
Now put this OS.nb file in the ROM, put the boot.rgu from 19965 (shipped ROM) into the \ROM\XIP folder and do not include any of the OEMXipKernel or MSXipKernel folders in OEM & SYS folder while cooking. I observed for some reason, WinCeNls_WWE folder cannot be taken out of XIP and included in SYS. Device wont boot, so keep that in XIP (found a working solution by spocky12: Here (last quote)
Please note the insertion of xip_out.bin can also be done through XIPPort.exe directly
Before clicking “write xip_out.bin to:” replace the name “nk.nb” with “OS.nb.payload” and the address to “00320000” all without quotes
IMP: There may be chances that although the XIP is working fine, but the windows are seen as QVGA versions. The solution to that is either of the below
XIP & SYS of the same builds or
XIP and the OS\Gwes.exe from same build
Cook the new ROM with your favorite kitchen (whichever doesn’t do anything with the XIP) and use this OS.nb file as base template for the ROM with the new XIP
With this note, I hope this guide will serve many as a guiding light and answer many questions on manual full XIP porting. Happy porting
Members Porting Updates
This is where we showcase the updates on XIP porting provided by our kind forum members
Original quote - Cmonex
Code:
[COLOR=royalblue][B]Quote=ababrekar[/B] - Busenum.dll must be the last entry in both tables[/COLOR]
Actually the values are arbitary, even though Microsoft decided to place coredll.dll as the last entry, i.e. at the highest memory address, it doesn't really matter. So, the values are arbitrary, but of course only within limits: the addresses must be divisible by 0x1000 (pagesize of the platform), and they must be inside the memory range reserved for XIP. part of that is the dllfirst and dlllast values in ROMHDR.txt. The other part (the higher addresses, 0x03xxxxxx) are determined by the following way: IMGFS .VM tells you the limits for IMGFS memory range, and XIP is beyond that range. So, if your OS doesn't want to boot, you can check if IMGFS .VM is overlapping with XIP memory range as per your MAP.txt for xip and dump_memorymap.txt (or .VM folder, etc) for IMGFS.
For example if IMGFS ends at 0x03DE0000, then the higher part of your XIP must start later than 0x03DE0000. You can of course modify this to make more space for XIP
If xipport crashes on writing maps it means you definitely have some overlaps left in. So yes, best to work with the maps from the original XIPs and only use the final XIP map to verify you got everything right
[COLOR=red]Btw, XIPPort's insertion function was found buggy on one device once, but cannot remember the details. It wasn't my device, so just posting this as a possible warning[/COLOR]
Oh, same applies to ROMMaster.exe, it is buggy when you try to use that to extract the XIP some ROMs
[COLOR=royalblue][B]Quote=ababrekar[/B] - Few mention of using the 723*.dsm for the build number, few others mention of using the coredll.dll module to have the latest build numbers[/COLOR]
Btw, coredll.dll replacement only works for that pre-WM6.1
And a last tip for [B]debugging [/B]if your OS doesn't want to boot: if you already checked that the maps are all ok and IMGFS doesn't overlap, etc., then if you have a new enough HTC device (for example HTC Athena and later is new enough), then go to SPL using mtty or putty or qmat and there the "task 37" command (without the quotes) will show KITL log, with lots of debug messages, that can be very helpful. (first you must issue "task 32", for "task 37" to work) - this doesn't appear to work on some Raphaels
Original quote - cruzzmz
Code:
If porting for [B][COLOR=teal]Zinc[/COLOR][/B]. After finish with all the MReloc, you need to Hex the S000 of nk.exe in the MODULES folder. The value can be found in MAP.TXT under the Modules
[COLOR=royalblue][B]Quote=ababrekar[/B] - 802FAA9C - 802faaf0 L00000054 rom_00 header: dlls=01f901fd-02000000 phys=80180000-803dc4fa, 24 modules, 10 files, 2 copyentries ext=8018265c ram=803dd000-83c00000 cputype=000001c2[/COLOR]
Open S000 in ur fav hex editor, then go to [B]Offset 1658[/B]
Change the original value i.e: [COLOR=red][B]802FAA9C [/B][/COLOR]and Hex edit it to [COLOR=blue][B]9CAA2F80[/B][/COLOR]
Original quote - DupinBJK
Code:
The addresses on the 80xxxxxx range should be on a [B]WORD [/B]boundary - Divisible by [B]4[/B]
Original quote - spocky12 (how to move wincenls to IMGF from XIP)
Code:
This is related to BootPhase key in boot.rgu. [URL="http://msdn.microsoft.com/en-us/library/ms885267.aspx"]According to Microsoft[/URL]
If this value is 0, then related filesystem is loaded prior to initialization of locale. But for this to work, the filesystem has to be loaded in Autoload key, like this :
[B][COLOR=red][HKEY_LOCAL_MACHINE\System\StorageManager\AutoLoad\FLASHDRV][/COLOR][/B]
[B][COLOR=red] "DriverPath"="Drivers\\BuiltIn\\FLASHDRV"[/COLOR][/B]
[B][COLOR=red] "LoadFlags"=dword:1[/COLOR][/B]
[B][COLOR=red] "Order"=dword:0[/COLOR][/B]
[B][COLOR=red] "MountAsRoot"=dword:1[/COLOR][/B]
[B][COLOR=red] "MountAsBootable"=dword:1[/COLOR][/B]
[B][COLOR=red] "BootPhase"=dword:0[/COLOR][/B]
With this, autoload will regsiter access to the imgfs filesystem before wince.nls is loaded. Then, when it'll be required, if it's not present in xip, it should be found in imgfs
Here is where we showcase miscellanous updates on XIP porting / MBR / MSFLSH50 which doesnt fall under the above categories. These are the updates which are not harming the system in any ways if left as is. Yet just a know how or just in case
Removing modules from XIP: Original quote - Cmonex
Code:
You can always remove osaxst0.dll, osaxst1.dll, hd.dll, kd.dll, and also bmui.nb0 - the latter is just a SplashScreen saying your OS can't boot and reflash or something (I forget the exact text)
The other files are Kernel debuggers and similar, best to remove them, because it just takes up space and can also cause problems if you somehow manage to use the wrong versions of them. They are mapped directly to the Kernel memory space, and if your device uses a different range (i.e. you didn't keep your original debugger dlls), it will prevent the rom from booting
Also I found it's ok to remove (m)encfilt.dll and cachefilt (put them in IMGFS if you want them)
[I][B]Physlast [/B][/I]can be changed up to [I]ulramstart [/I]value without problems (of course not if you don't have enough space in the flash, but that's not really a real life possibility). Of course that also assumes we are not talking about some older devices that have the xip mapped to a different memory range than [I]ulramstart[/I]
You can move [I]ulramstart/ulramfree [/I]too if you relocate nk.exe data section (usually S002) with M'Reloc-nk. Also relocation is needed for any other modules (such as giisr.dll, on some non HTC devices) that have mappings similar to nk.exe (so they have a data section in the map that points into [I]ulramstart/ulramfree [/I]range). on HTC devices I didn't really see such modules so not a real problem usually
Increasing the free RAM (Part 1): Detailed explanation here by DupinBJK
Simple explaination for easy understanding: (the below values are from a sample MAP.txt and dump_memoryMaps.txt (View attachment Examples.zip)) for trying to explain what comes from where and the actual values may differ from your files
Code:
8019e9a4 - 8019e9f8 L00000054 rom_00 header: dlls=[COLOR=red]01f801fc[/COLOR]-[COLOR=black]02000000[/COLOR] phys=[COLOR=black]80000000[/COLOR]-[COLOR=blue]8030c7b3[/COLOR], 28 modules, 10 files, 1 copyentries ext=80002b4c ram=8030d000-83000000 cputype=000001c2
[COLOR=blue]8030c7b3 [/COLOR]- 8030c7b3 L00000000 End: highest physical address
The blue value that mentioned is the physlast value. In the dump_memoryMaps.txt, you will find:
Code:
01F7F000 - 01F7FFFF (4095 bytes): bthasplugin.dll
after which the dllfirst starts in MAP.txt with a difference of 1FD length and
Code:
03D66000 - 03D6FFFF (40959 bytes): bthasplugin.dll
after which the e32_vbase starts (03dcb000 - 03dd4000 L00009000 Virtual base address of wce_rex.DLL) in MAP.txt with a difference of 5B001 length
Increasing the free RAM (Part 2): by Ameet
Changed the size of ROM in G'Reloc from 83000000 to 83400000 and increased the ulRAMEnd to the same value (83400000) getting free RAM space of L0306C000 (I dont know how to translate this into the actual size in % or in bytes) but with the original of 83000000 the RAM space was L02C6C000
Having done this, I get about 62% free memory without TF3D and approx 54% free memory with TF3D at system start
Extracting the XIP from any ROM: Detailed explanation here by boggsie
More explaination about XIP processes & editing OS version on 1st splash screen: by FormerPalmOS
Code:
[B][COLOR=red]1) [/COLOR][/B]The Initial Program Loader copies the XIP partition from the FLASH to SRAM - in Diamond and Touch Pro there is a custom Samsung chip that includes both NAND FLASH and SRAM. The overall physical RAM space where this is loaded is also hard-coded - see below. The amount of RAM used is variable - this info comes from a header in the XIP section - basically how much RAM does the XIP need? What's left is what you get for program memory.
[B][COLOR=red]2) [/COLOR][/B]IPL executes a jump to a hard-coded address within the SRAM - this should be busenum.dll, which is why busenum.dll has to be at a specific physical and virtual memory location.
[COLOR=red][B]3) [/B][/COLOR]busenum.dll does its thing (not sure entirely what) but eventually calls nk.exe. nk.exe is the kernel. nk.exe loads the other modules, initializes the hardware (that's why nk.exe is device-specific), and initializes the filesystem and basic device drivers (again why those are device-specific).
[COLOR=red][B]4) [/B][/COLOR]Once this process has completed, the filesystem proceeds to load the imgfs filesystem and turn control over to the full OS.
The virtual memory map for WM consists of a number of slots. The memory management unit in the CPU translates virtual memory references into physical memory addresses. Every loaded dll or exe must occupy a portion of virtual memory for its code and will likely also use some of the available RAM for its data. The location within virtual memory where the code for a dll or exe is loaded is determined at load time unless the dll or exe is a module (everything in the XIP is a module) in which case the virtual memory location is specified during cooking. In the XIP, the location of the RAM used is also specified - the process of relocating a module in the XIP specifies the virtual memory location for the code and and data in the case of nk.exe, the physical RAM location.
There are four VM sections we care about (note - I'm taking some liberty here - these don't exactly correspond with what Microsoft refers to as a VM slot). Slot 0 runs from 0x00000000 to 0x01FC0000 (in the CDMA Touch Pro). The end of slot 0 is a function of the number of and size of the data regions for the DLL modules in the XIP. This number plus 0x1FC is stored in the ROM header (and can be examined in ROMHRD.txt) - it is referred to as dllfirst. This is also the slot 0 you see when you do G'Reloc.exe (the value in G'Reloc.exe is the last address of slot 0 plus one). These two must match!!! What the XIP uses must not overlap with what your ROM uses.
The next slot is the XIP DLL initialized data. This runs from dllfirst to dlllast. dlllast is fixed (in the Touch pro) at 0x02000000. The XIP DLL data sections are loaded starting at 0x02000000 and working backwards.
The next slot is again available for the OS and runs from dlllast to wherever the code in the XIP starts. You can see this in your XIP memory map - this again must match (the end of slot 1 in G'reloc.exe must match the first DLL virtual base address in your XIP - in mine this is 0x03DC0000). The XIP DLL and EXE code occupies from this virtual memory address to 0x03FFFFFF.
The OS will load DLLs and EXEs (other than XIP) into this slot starting at 0x03DC0000 and working backwards, then will move to the slot below 0x01FC0000. Recall, I'm using my numbers here. Any modules in the ROM will have their virtual memory slot and address pre-assigned. Any non-module DLL or EXE will be relocated to an available slot and VM address at load (this is why modules load quicker).
So in summary, my VM map looks like this:
0x00000000 - 0x01FC0000 - OS available (G'Reloc.exe slot 0)
0x01FC0000 - 0x01FFFFFF - XIP data
0x02000000 - 0x03DC0000 - OS available (G'Reloc.exe slot 1)
0x03DC0000 - 0x3FFFFFFF - XIP modules code
The actual physical XIP RAM address starts at 0x80000000 in the Touch Pro (this is physfirst in the ROMHDR.txt) and ends at 0x83400000 (in the Verizon Touch Pro - this is ulRAMEnd). The XIP is copied from the NAND flash starting here with the ROM header occupying 0x80000000 - 0x80001000. Then come the various XIP components, hopefully none of which overlap. The XIP should end at or before a ROMHDR.txt value called physlast. Thus physlast - physfirst is the size your XIP has to fit into.
Following physlast comes ulRAMStart - this is where the RAM required for nk.exe is located. This RAM ends at ulRAMFree. What remains after ulRAMFree until ulRAMEnd is available for your OS. Shrinking your XIP and relocating nk.exe will allow you to recover wasted space and give you more program memory, but it buys you nothing to move a module out of the XIP if it is required by the system. Only things that aren't required (like debuggers and hard drive drivers) should be removed.
Also, the least significant 16 bits must be zero (lower four hex digits) of the end of vm slot 0 and slot 1 in G'reloc.exe and in your ROMHDR.txt. The least significant 14 bits must be zero (the lower four digits can only be 0000, 4000, 8000 or C000) of the RAM address (ulRAMStart and ulRAMFree).
Code:
Hex edit the S000 file in the nk.exe module folder and search for the revision string. You can find it by doing a search for the unicode string "[B][COLOR=red]Kernel Built[/COLOR][/B]" (Hex String [B][COLOR=red]4B 00 65 00 72 00 6E 00 65 00 6C 00 20 00 42 00 75 00 69 00 6C 00 74 00[/COLOR][/B]). Shortly after that will be the revision that is displayed on the phase 1 boot screen (small red letters in the lower right corner of the device on CDMA Touch Pro). Change that (make sure to overwrite, not to insert, and limit it to 12 characters in unicode format.
When you rebuild your xip.bin and cook with it, you should see this value on the screen during phase 1 boot. The only other way would be to insert a marker into the boot registry
Change PagePool through Hex editing (for Diamond & Raphael)
I'm putting this up here so to answer one more unanswered question about this especially for Diamond & Raphael ROMs
To change PP of Diamond ROMs:
Open the OS.nb in Hex editing software
1. Go to offset 0x37AD68 to find 03 25 A0 E3 03 15 A0 E3 00 20 83 E5 hex string (If this string is not found at the 37AD68 offset, then search for this hex string)
Replace the string with 20 83 E5 with 00 A0 E1
This will make the string NOP (No Operation) meaning, the ROM wont set the PP to default 12MB but will allow the change in below offset
2. Now go to offset 0x3A7F94 to find E0 E2 04 80 00 00 60 00 hex string
again, if this hex string not found at the 3A7F94 offset, then search for the hex string. Just as a hint, this string is after the second NKKD8 (search for text string)
60 is the size of PP that you can now modify to suit your liking
e.g. I made mine 00 to get 0MB PP. Or change it to 80 to get 8MB PP, so forth and so on
With changing the first hex string and making the Kernel NOP, you can also use the tool to change PagePool and it does work
Also to make it a permanent change you can hex edit the first mentioned string in S000 of nk.exe module in XIP and then modify the PP with the program or by hex on OS.nb
To change PP of Raphael ROMs:
Search for hex string: 03 15 A0 03 02 15 A0 13 00 10 82 E5 and change the last 4 bytes to 03 15 A0 03 02 15 A0 13 00 00 A0 E1 then the normal PP Changer tool will work
This is the 2nd string, ignore the 1st one coz that's in ULDR
Gain more Storage Memory (increase imgfs size)
There are 4 partitions in Diamond ROMs
part00 – ULDR
part01 – XIP
part02 – IMGFS
part03 – FAT (This partition exists only on few devices)
We all port XIP from different devices to exclude few modules to gain space and to upgrade the kernel and make the XIP partition smaller in size. Although the new XIP is smaller in size but because of the insertion addresses of XIP & imgfs, there is a gap of wasted space filled with FF between end of XIP & start of imgfs. Although there is no way we can include this space into XIP as free RAM but make use of this space in imgfs and gain whatever storage space we can
Files used as example for this tutorial
xip_out.bin: My own ported XIP of size (30CA12 in Hex, 3195154 in bytes)
os.nb.payload: My own cooked payload (since I also wanted the final ROM to be a cleaner ROM)
imgfs start: in my payload at 0x7A0000 (unedited)
XIP start: in my payload at 0x320000 (unedited)
Before we move into hex editing, let me give an overall outlook of the MBR & MSFLSH regions of the ROM
MBR is the Master Boot Record of the ROM (512 bytes) from 0x0 to 0x1FF. The infomation of partitions types Flags in hex offsets are called from the registry entry mentioned in boot.rgu below
The starting block (LBA) and number of sectors for each partition are defined as shown below
part00. 1C6 – 1C9 (starting block) 1CA – 1CD (number of sectors)
part01. 1D6 – 1D9 (starting block) 1DA – 1DD (number of sectors)
part02. 1E6 – 1E9 (starting block) 1EA – 1ED (number of sectors)
part03. 1F6 – 1F9 (starting block) 1FA – 1FD (number of sectors)
[HKEY_LOCAL_MACHINE\System\StorageManager\PartitionTable]
"04"="FATFS" ; (hex: 1F2)
"20"="BOOT" ; (hex: 1C2)
"23"="RAWFS" ; (hex: 1D2)
"25"="IMGFS" ; (hex: 1E2)
MSFLSH50 is the Flash region of imgfs from 0x800 (see post #8 for screenshots, shown here is for Diamond) to 0xFFF. The starting block of imgfs is located in MSFLSH at 81C
e.g. if your device ROM's sector size is 200 then the MSFLSH50 region will starts at 0x200 and so on
Moving into the hex editing mode for making use of the wasted space between the actual XIP end & start of imgfs partitions
The new xip_out.bin is 30CA12 in total size (check your actual xip_out.bin size, shown here is just example) starting at 0x320000 (check you device XIP start, shown here is for Diamond) and ideally should end at 62CA12. But since the starting block of imgfs must be divisible by 20000 (see post #8 for screenshots, shown here is for Diamond) the imgfs needs to start at 640000. So the new XIP will have to be inserted into the payload at 0x320000 till 0x640000 with XIP size of 320000 and reduced wastage of 135EE bytes
The imgfs can also start at 630000 since this is directly after the XIP and also divisible by 20000, used here is 640000 as expansion for future xip_out.bin
Open the existing os.nb.payload in hex editor. Delete everything from 0x640000 till 0x79FFFF. This will move the imgfs from 0x7A0000 to 0x640000. Since we are now moving the imgfs partition next to new XIP, the number of sectors in new XIP and new LBA of imgfs needs to be edited to the revised value in the MBR region
To calculate the new starting block of imgfs partition we need the number of sectors in new XIP. To calculate that, use the following method
In Hex calc
Number of sectors = size of partition / sector size
e.g. (new XIP) 320000 (shown above) / 800 (see post #8 for screenshots, shown here is for Diamond) = 0640
since the coding is in little endian, we have to reverse these values to 40 06 00 00
Go to offset 0x1DA and change the values to 40 06 till 1DB and then 00 00
Now realloc the LBA of imgfs since we revised the number of sectors in XIP and to calculate that, use this method
In Hex calc
Logical Block Address (LBA) = Previous Partition LBA + Previous Partition number of sectors
e.g. (XIP LBA) 0640 + (XIP no of sectors) 0640 = 0C80
since the coding is in little endian, we have to reverse these values to 80 0C 00 00
Go to offset 0x1E6 and change the values to 80 0C till 1E7 and then 00 00
Logical Block Address (LBA) should be equal to (Previous Partition LBA + Previous Partition number of sectors * Sector Size)
e.g. (XIP LBA) 0640 + (XIP no of sectors) 0640 * 800 (see post #8 for screenshots, shown here is for Diamond) = 640000 (size of imgfs partition)
Similarly to imgfs calculate and change the LBA of FAT at 1F6 and 1F7 using the default imgfs no of sectors (use these since the cooking tools will change these as per actual size)
We have changed the LBA and number of sectors in MBR, but the OS needs to know the block address of imgfs in MSFLSH50 region
To calculate that, use this method
In Hex calc
MSFLSH50 Block Address = imgfs partition starting address / 20000 (see post #8 for screenshots, shown here is for Diamond)
e.g. (imgfs starting address) 640000 (shown above) / 20000 = 32
Go to offset 0x81C and change the value to 32
Save and close the os.nb.payload file in hex editor. Insert the new XIP into this file using this command
“insert.exe -i xip_out.bin -o OS.nb.payload -d 0x00320000 -s 0x00320000” (check your insert start address, shown here is for Diamond)
To calculate the size of XIP from MBR, use this method
In Hex calc
Size of XIP = Number of Sectors * Sector Size
e.g. (if the no of sectors in little endian) 0640 (shown above) * 800 (see post #8 for screenshots, shown here is for Diamond) = 320000 (sector size for diamonds)
This value shall be the "-s" while using insert.exe tool and to calculate the start of the XIP, use this method
In Hex calc
XIP Start = imgfs Start + "-s"
Reduce ULDR Partition Size
“ULDR” stands for “Update Loader”, and is part of the Image Update system. This system allows deployed devices to be updated with new software after they ship. The Update Loader reads a configuration stored in persistent memory and downloads and installs new versions of operating system or OEM files
Also known as part00 in the ROM, is something we all wish to get rid of and use the space as additional storage memory. This tutorial currently aims at reducing the size of this partition by 3.0 MB
Tools you need
NBSplit.exe
NBMerge.exe
Hex editor
Ervius's Payload Reducer
IMPORTANT NOTES
The template OS.nb used is the same OS.nb in which the XIP is inserted at 320000 and of size 320000
For best results use Ervius's Payload Reducer to reduce the size of payload from shipped ROM use nbmerge.exe to cook OS.nb as template for further cooking
This ROM is assumed to be from Diamond and check your device values as per the guide below
The hex offsets of (L)ogical (B)lock (A)ddress and number of sectors and imgfs block address are mentioned in tutorial above or in the post #8 below
Process
Extract OS.nb.payload from the OS.nb (nbsplit.exe –kaiser (check your device) OS.nb)
Run the OS.nb.payload through Ervius's Payload Reducer tool to remove all files from the imgfs and keep only the partition headers
Open this OS.nb.payload in your hex editor. We need to change LBA values of the partitions and number of sectors of ULDR partition since we are reducing the size
In the MBR region (partition Flag 20) LBA of ULDR partition remains same since we are not moving it anywhere. The existing number of sectors of ULDR is 3E 06 from little endian it will be 063E. We are removing 0600 sectors from this partition (0600 * 800 (size of sector, see post #8 for screenshots) = 300000) so, 063E – 0600 = 00 3E. Write it in little endian at hex offset 1CA and 1CB to 3E 00
To physically reduce the partition, remove all data between hex offsets 0x20000 till 0x31FFFF. This will make the XIP start from hex offset 0x20000 till 0x33FFFF and the imgfs partition start at 0x340000
Now since we have reduced the size of ULDR partition, the LBA values of XIP and imgfs partitions will have to be changed in the MBR region
Now change the LBA of XIP. To calculate that, use this method
In Hex calc
Logical Block Address (LBA) = Previous Partition LBA + Previous Partition number of sectors
e.g. (ULDR LBA) 00 00 00 02 + (ULDR no of sectors) 00 00 00 3E = 00 00 00 40
since the coding is in little endian, we have to reverse these values to 40 00 00 00
Go to offset 0x1D6 and change the values to 40 00 00 00 till 1D9
Now change the LBA of imgfs. To calculate that, use this method
In Hex calc
Logical Block Address (LBA) = Previous Partition LBA + Previous Partition number of sectors
e.g. (XIP LBA) 00 00 00 40 + (XIP no of sectors) 00 00 06 40 = 00 00 06 80
since the coding is in little endian, we have to reverse these values to 80 06 00 00
Go to offset 0x1E6 and change the values to 80 06 00 00 till 1E9
We have changed the LBA and number of sectors in MBR, but the OS needs to know the block address of imgfs in MSFLSH50 region
To calculate that, use this method
In Hex calc
MSFLSH50 Block Address = imgfs partition starting address / 20000 (see post #8 for screenshots, shown here is for Diamond)
e.g. (imgfs starting address) 340000 (shown above) / 20000 = 1A
Go to offset 0x81C and change the value to 1A
Save and close the os.nb.payload file in hex editor. Insert the new XIP into this file using this command
“insert.exe -i xip_out.bin -o OS.nb.payload -d 0x00020000 -s 0x00320000” (check your insert start address, shown here is for Diamond) (ignore this if the XIP is already inserted and shifted to this location with this size)
The value (02) seen at hex offset 0x1BF should not be changed or touched since that value tells the OS that first partition starts from the third Sector of the ROM (0x800 (sector size) + 0x800 = hex offset 0x1000) Currently the reduced ULDR partition starts from third sector
Now create the OS.nb from the edited OS.nb.payload to be used as template for cooking using this command
“nbmerge.exe –kaiser (check your device) OS.nb” (without -conservative switch)
NOTE
For best results directly use the OS.nb.payload as template for cooking without merging it into OS.nb. For this you will need to edit the CreateROM.bat
Note the change in red and delete the blue lines from this bat file
copy ROM\OS.nb.payload temp\OS.nb.payload
..\TOOLS\NBSplit -kaiser OS.nb
Rem rename os.nb.extra os-new.nb.extra
!Aman!'s awesome tutorial on removing ULDR partition from devices which don't have the FAT partition (part03) can be refered here: http://forum.xda-developers.com/showthread.php?t=446506
Screenshots of MBR and MSFLSH50 Regions
MBR Region
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
MSFLSH50 Region
Attached these images in Pictorial.zip with post #1 for offline reference
Gain more Storage Space with LZX compression
Thanks to:
spocky12 for cecompr_nt.dll (attached)
ivanmmj for cecompr.dll (attached) This module supports LZX compression as well as the default XPR algorithm
Replace the cecompr.dll found in the OEMXipKernel (or whichever folder you have your XIP modules) with the attached cecompr.dll module that supports LZX compression
The LZX compression takes a load of your RAM while cooking which makes the continuing IMGFSFromDump tool crash. To avoid that replace the attached cecompr_nt.dll file found in “Tools/IMGFS” folder of your kitchen
Pause your kitchen process right after it extracts the IMGFS.bin and before it inserts the files into it. (A simple “PAUSE” in the batch file will suffice). Then open up your IMGFS.bin in hex editor of your choice and search for the string "XPR". Replace the FIRST one, FIRST & ONLY ONE with "LZX". Close the hex editor, save the file and let your kitchen continue with the cooking
After flashing the ROM cooked with this module should give you approx. 10MB more space in the storage memory
Original Posts:
ivanmmj: http://forum.xda-developers.com/showpost.php?p=3678382&postcount=877
spocky12: http://forum.xda-developers.com/showpost.php?p=3690996&postcount=904
Space for Donors
High value real estate space for donors for all my work on XDA
Special Thanks
Piranha1 $5
Vippie $5
Steven Ellis $10
Guitarguy $5
Ckpv5 $5
Letama $10
!Aman! $5
LoriInWa $10
Noonski $15
Kevin $5
Dogmale $1
Nazaliyah $1
Miscellanous uploads related to XIP porting
Kitchen Files
I use "DiamondKitchen_v0.4" kitchen to cook Diamond ROMs. The XIPInsert file is something that I made to automate the insertion and nbmerge process (well, something automatic is better than complete manual )
If you select to use the XIPInsert batch file then you must have DiamondKitchen_v0.4\XIP\xip_out.bin and DiamondKitchen_v0.4\XIP\OS.nb.payload to use this option, else the existing OS.nb file in \ROM folder will get deleted
Note: The insert values used in the batch file is for Diamond ROMs. Please check and edit these as per your devices
Code:
[B][U][I]!COOK.cmd[/I][/U][/B]
Modified to provide options to include the below batch file or to continue without it
Also included necessary warning
Code:
[B][U][I]8a.XIPInsert.bat[/I][/U][/B]
@echo off
cd [COLOR=green]XIP[/COLOR]
..\TOOLS\insert.exe -i [COLOR=green]xip_out.bin[/COLOR] -o [COLOR=green]OS.nb.payload[/COLOR] -d 0x00320000 -s 0x004C0000
..\TOOLS\nbmerge -kaiser OS.nb
copy OS.nb ..\ROM\OS.nb
del OS.nb
del *.payload
del *out.bin
exit
Deprotecting ROMs: My friend, Ervius has made a small tool to "deprotect" a protected ROM , here: http://forum.xda-developers.com/showthread.php?t=465642
First to say THANKS
Cheers
and second to say this is real hard work...but u have done it bro..
congrats
another amazing work by the XIP Master
Thanks for sharing the info, ababrekar! I'm sure this will help out many people (myself included ).
Bite Down and Don't Give Up.
Sounds like someone i know from.... hey, it is you!
Good Job,
ababrekar said:
Use this command for inserting the xip_out.bin into the .payload (presuming you already have the shipped OS.nb.payload file in the same working folder
insert.exe -i xip_out.bin -o OS.nb.payload -d 0x00320000 -s 0x004C0000
Check these values with your device imgfs since in Diamond the XIP starts at 0x00320000 and the imgfs starts at 0x007A0000, but for some reason the imgfs signature in Diamond is at 0x007E0000
Click to expand...
Click to collapse
just to make it more clear, the value for "-s"= (starting offset of imgfs - starting offset of XIP)
PS: wonderful job writing this guide brother
Ouch - too much like work, but it is nice to know how to do it.
Thanks for your effort!
Best regards,
-boggsie
So, how does one get hooked into the flow of new releases?
Perhaps would be a good idea to use one of your reserved posts as a repository of good XIP.BIN files with versions and info about the ROM extracted, so all we can use in our ROMs... only a tought...
jcespi2005 said:
Perhaps would be a good idea to use one of your reserved posts as a repository of good XIP.BIN files with versions and info about the ROM extracted, so all we can use in our ROMs... only a tought...
Click to expand...
Click to collapse
That is a good idea but I'll do that for posting the unedited XIP.bin files from dumped ROMs since the xip_out.bin I build would be for my device. People wont want to ruin their ROMs with someone else's ported XIP, right?
Related
Could you explain please (very short description) how you modified the xip chain for rom kitchen?
All I can see is the following:
- no length (0)
- no RSA1 signature
- only file entries
What I want to know:
- how to find phys. (ROM) position (do you use unused holes in rom?)
- is 0 length for ROM = autolength
- how to choose the RAM position
- why can length of RAM be 0
Please help. (I need this info for a smartphone project)
I did not bother setting the length, only the 'pvAddr' field is used.
I only make fileentries, because I have yet to implement the generation of modules. ( if I ever do ).
yes, I use unused holes in the rom.
actually, if you don't care about xip updates of other sections, you
may use addresses anywhere in the rom, where your data fits.
It does not nescesarily have to be contigous.
I just copied the ram setting from the other xip entries.
Thank you for the information.
Why don't you take romimage.exe from platformbuilder to generate a XIP block. You only have to write a little .bib file for it. This tool can handle modules and compression as well.
John
P.S. Source code for romimage.exe is available in PB 4.2 private build tree.
I hadn't found that tool yet when I wrote makexip, and then we couldn't have made the romkitchen with it, since romimage.exe runs only under windows.
Don't waste your time with this crap tool (romimage.exe). Some needed files are missing (e.g. bin2xip.exe).
How can I be sure to choose a good phys. addr.? There might be some memory mapped devices...
I have one additional difficult question:
Modules are relocated when embedded into XIP's. Even there seems to be a modification to the import table of the module (e.g. references to coredll.dll will be checked/updated?)
If I extract a module (e.g. a *.dll) from a XIP of an other phone do I have to re-relocate it / modify it's import section even if I place it in a FILES section?
Thanks
John
converting bin to xip is not that difficult. see http://www.xs4all.nl/~itsme/projects/xda/wince-flashfile-formats.html
do you mean the 'physfirst' field in the romheader? that is just the startaddress in the rom.
since the relocation information is not stored in rom, the only way to really
recover it, is to disassemble the file, and find the offsets to stuff that
needs to be reallocated.
so that is a lot of work. and dumprom only extracts nonrelocatable .exe and .dll modules.
if your extracted dll is fixed to a memory location that overlaps with an already existing dll, you will have a problem I think.
I am not even sure, if an extracted dll works at all, I only use them for reverse engineering.
Yes, I mean phys first field. But how can I be sure to choose a valid address for the new XIP block?. My idea is to use address space between existing XIP blocks. Or can I simply choose a very high address (e.g. 8F000000) and hope not to use a region where memory mapped devices are located?
Since I used (your?) dumprom to extract the *.dll files do you think they are "nonrelocated"?
John
I ask so much because I crashed my lovely smartphone a week ago. :-(
My new XIP seems to be invalid... so it doesn't boot anymore. Unfortunately I've killed the bootloader too...
When I try next time (I've ordered a new one) this must not happen!
I am sure they are nonrelocated, fixed to run from a specific memory location.
( just wrote another post about this )
maybe even the module loader does not allow non-xip modules to be loaded in xip reserved memory.
THANK YOU VERY MUCH
I've got it. My Smartphone now have a new XIP block with some files in it...
Only thing left is to rewrite some *.dll files (only resource dlls with no function exports) to extend the language of the MIO 8380.
Are you familiar with languages on smartphone? There are multiple .mui files (resource dlls containing all the dialogs and strings). I've exported all resources and (re)created the dll's as resource only. Unfortunately they don't work ... yet ...
Are there some other files for language extension? What about "wince.nls" or "mxip_lang.vol" ?
Thanks again for your great tools. I will setup a site containing detailed information about this hack soon.
John Smith
cool, I am always interested to see how things work out that I haven't actually tried myself yet.
is this how you create resource only dll's:
http://www.xs4all.nl/~itsme/projects/programming/icondll.html
Currently I'am a little bit confused. PB 4.2 docu says MUIs are resource only .dlls and sample in smartphone sdk adds a dllmain...
I will have to investigate this things a little bit more...
John
O.K.
I've tried anything. The only thing left is that the new resource dlls are not XIPed as modules...
The sample mui app works fine (regardless of resource only / normal dll).
John
P.S. I've successfully changed all other settings some things already appear in the new language. Only poutlook, homescreen and control panel will not change!
Now after some more testing (included a dllmain into the mui file which logs the loading/unloading to file) it seems that my mui.dll is never loaded by system (if I load it manually with LoadLibrary the log is written).
Here is my question:
I've looked a little bit deeper into the dumped mui.dll and found a pointer in security section (pe header) which points to nowhere (just after the [virtual] end [rva] of all of the e32/o32 sections).
Could it be, that I've missed something? Does dumprom fill in this values correctly?
One other interesting idea could be to exchange only the data section of the module (since I want to patch resource only .dlls). But since english is a very short term language all other files will be bigger...
John
>>> I've got it <<<
the new (mui-language) modules have to be REAL xip modules...
So I've build a custom.bib file and used RomImage from CE3.0 Platformbuilder. Even compression is possible now.
Note: romimage.exe does the same thing as makexip.pl
To share my results here is the content of the .bib file I've used:
Code:
MEMORY
; Name Address Size Type
MYXIP 81f00000 0013f000 RAMIMAGE
RAM 8c020000 00fe0000 RAM
CONFIG
COMPRESSION = ON
PROFILE = OFF
ROMFLAGS = 2
ROMSTART=81f00000
ROMSIZE=13f000
ROMWIDTH=32
DLLHIGHADDR=00b00000
MODULES
; Name Path Memory Type
; ------------------------- ------------------------------- ------ ----
outres.dll.0407.mui input\outres.dll.0407.mui MYXIP SHU
syncres.dll.0407.mui input\syncres.dll.0407.mui MYXIP SHU
tapres.dll.0407.mui input\tapres.dll.0407.mui MYXIP SHU
tshres.dll.0407.mui input\tshres.dll.0407.mui MYXIP SHU
wmplayer.exe.0407.mui input\wmplayer.exe.0407.mui MYXIP SHU
FILES
; Name Path Memory Type
; ------------------------- ------------------------------- ------ ----
Busy.0407.mid input\Busy.0407.mid MYXIP
mxip_lang_799.rgu input\mxip_lang_799.rgu MYXIP
ms_splash.gif input\ms_splash.gif MYXIP
carrier_splash.gif input\carrier_splash.gif MYXIP
- The MYXIP region in MEMORY section is a hole in the ROM I've found with calcgaps.pl.
- The RAM region is copied from the other sections (they all use the same)
- ROMSTART and ROMSIZE have to be the same values as defined in MYXIP
- DLLHIGHADDR has to be the !!!lowest!!! loading address found with dumprom (header: dlls=...-... ).
Example: If the lowest address found is "header: dlls=00b00000-00c90000 ..." then DLLHIGHADDR has to be 00b00000
Don't care about the warning the warning "Unable to do imports from ... to COREDLL.dll - will late bind". Thats because coredll is in another XIP.
John
P.S. Thanks a lot for all of your support.
DETAILED INFORMATION ABOUT THIS HACK CAN BE FOUND HERE:
http://smartphonerom.tripod.com (only download the "detailed information")
Hi there,
I've done some reverse engineering on the LG DZ files, and here is the layout of these files.
DZ File Format
Code:
Offset Length (bytes) Description
0x0 0x8 Magic code "MSTXMETX"
0x8 0x2 Unknown, same value in all files (value 0x01)
0xA 0x2 Separator ? (value 0x0)
0xC 0x2 Unknown, same value in all files (value 0x0B)
0xE 0x2 Separator ? (value 0x0)
0x10 0x6 Unknow, value differs from file to file
0x16 0x2 Unknown, same value in all files (value0x01C8)
0x18 0x8 String, phone model ? (value "KS20")
0x20 0x50 String, file title ?
0x70 0x1C Two null terminated string concatened ("[chipmodel]\0[osname]\0")
0x8C 0x80 String, DZ filename
0x10C 0x20 Separator (filled with 0xFF)
0x12C 0x10 Header MD5 hash
0x13C Variable Concatened subfiles (see Subfile format)
-- 0x78 Offset Table, unknown useage (see OffsetTable format)
-- -- Optionnal data, present in some DZ file (unknown)
Subfile format
Code:
Offset Length (bytes) Description
0x0 0x4 Magic code "SSTX"
0x4 0x2 Unknown, same value for all subheaders (value 0x01)
0x6 0x2 Separator ? (value 0x0)
0x8 0x2 File Type ?
0xA 0x6 Separator ? (filled with 0x0)
0x10 0x4 Data length
0x14 0x80 Filename, null terminated string
0x94 0x10 Separator (filled with 0xFF)
0xA4 0x10 Uncompressed data MD5 hash
0xB4 0x10 Subheader MD5 Hash
0xC4 Data length Gzip compressed data
File type (in subheader)
Code:
Type Filename Description
0x3 amss.mbn AMSS modem
0x8 partition.mbn Partition table
0xA qcsblhd_cfgdata.mbn QCSBL header
0xB qcsbl.mbn QCSBL
0xC oemsblhd.mbn OEM boot header
0xD oemsbl.mbn OEM boot
0xE amsshd.mbn AMSS modem header
0x13 appsboothd.mbn APPS boot loader header
0x14 appsboot.mbn APPS boot loader
0x15 FLASH.bin
0x16 apps.mbn APPS
It's not complete, but that's a start.
I also made a tool to extract the subfiles contained in the DZ file (see attached file).
Hope this helps,
JP.
Check also the tool to extract the content of the Flash.Bin file : LGFlashParser
Great news !
I tested your soft with different lg ks20 dz files downloaded on the web and files seem to be extracted as they should. I didn't try to flash though, as I'm at work and I don't have usb cable here.
Now we still need to find how to edit those files (*.mbn and especially flash.bin which contains windows file system). Anyway, it's a great step towards lg rom cooking. Thanks again!
I have updated the DZExtract tool...
It now checks all the known MD5 hash, so the extracted files can be considered valid.
Here are the command line options :
Code:
dzextract.exe [options] path [outputpath]
options : default -p
-p Print and check header information
-x extract subfiles
path : path to dz file
outputpath : path to output directory (must exists)
Hope this helps,
JP.
Wow, sounds great!
Thank you very much for this excellent work.
it's possible to extract also the files of .mbn or FLASH.bin ?
For mbn files you can try with this soft :
http://forum.modopo.com/diskussionen-rund-ums-modding/t-19197-qc-bqs-firmware-analyzer.html
Great.....a good start............
How does it work? Can it work in vista? Coz i double click the DZextract.exe, a dos window came out for 1 sec then closed and nothing happen.
raykisi said:
How does it work? Can it work in vista? Coz i double click the DZextract.exe, a dos window came out for 1 sec then closed and nothing happen.
Click to expand...
Click to collapse
This is a command line tool, so you need to open a command window and follow the instructions.
About the contained files :
The flash.bin seems to be the windows image (found old infos here : Rom file format)
The mbn files reflect the firmware structure :
- QCSBL = QC Secondary Boot Loader
- OEMSBL = OEM Secondary Boot Loader
- AMSS = Advanced Mobile Subscriber Software
...
(infos from the link posted by DomZ)
JP.
Great application !
Files identical (same CRC) from unbranded to branded roms:
apps.mbn
appsboothd.mbn
oemsbl.mbn
oemsblhd.mbn
partition.mbn
qcsbl.mbn
qcsblhd_cfgdata.mbn
All the remaining files are different, inclueding the flash but that is obvious.
(amss.mbn, amsshd.mbn, appsboot.mbn, Flash.bin)
I guess we have to compare the unidentical mbn files to find out what prevents the unbranded roms to work in phones with branded NVs.
Then if nothing is conclusive we will have to check the flash itself.
mbn files seem plain/uncompressed binaries.
In fact I believe the CID checks are performed by amss.mbn which seems to be somehow the true bootloader (It is a true ELF binary and likely the very first to be running)
This is just a guess based on my todays reading and disassembling: the msm7200 has two processor, and one is dedicated to the radio functions. This processor actually runs the AMSS file which is an arm elf executable (you can to disassemble this file with IDA).
Apparently, the AMSS part is based on an L4 microkernel, and running iguana embedded os (see here).
On the QC BQS analyzer page, the boot process is explained (it's for another phone, but I think it's the same). The QCSBL (Qualcomm secondary boot loader) load the OEMSBL (probably on the first processor) and then also load the AMSS Elf.
Both the oemsbl and the amss run in parrallel (i guess they communicate with some kind of IPC mechanism).
After that, the oemsbl start the windows kernel.
I think, we should concentrate on the flash.bin file, as i think the other files are either encrypted or at least signed.
My post is just a guess as it's very hard to find information on the msm7200 processor and as I do not have a KS20 phone (nor a qualcomm based phone) to check all these informations.
JP.
The other files are plain text. They may however indeed be signed although this is not sure, since they did not bother to add any kind of encryption (at least not that I could spot) , perhaps they did not bother adding a signature either. I think LG did not expect us to access these files/ressources from the phone, I do not expect them to be accessible through software (running on the phone that is) we can actually access it thanks to the leaked service tools from LG using their recovery mode.
I think you are right about LG as I found that the files contained in the DZ file were supposed to be crypted : when disassembling the LG sofware, I found that the fonction that extracts the subfiles could also decrypt the files, however none of the DZ files I have are encrypted.
But regarding the qcsbl file for example, I'm not so sure : this file is certainly built (or at least coded) by Qualcomm and is probably part of the SDK they send to their customers.
I've seen posts about similar phone using qualcomm chipset reporting that some of the bootloaders were signed.
JP.
Spend money...
Hi,
its cool what you have find out about the DZ Format. But the problem is that misterjp has no KS20. What do you think if we all spend a little money, so that misterjp can buy a KS20 because with this he can find more about the DZ Format? If we know a lot of the DZ format in the future someone will release new Windows Mobile Version.
Dear Dussel
in fact need to knew more about .mbn file format ... not .dz (that is simpliest pack method /container for multiple .mbn /)
on net i'm saw .mbn extractor ... but not .mbn creator ... that format more complicated than .dz - so better to spend money for that, who can work with .mbn - it's more perspectivelly, or anybosy, who can find alternate way to flash (without .mbn)
so firstly take a look to :
1. MBN-Resourcer = Qualcomm firmware resource viewer
2. QPST - Qualcomm Product Support Tools
3. QXDM - Qualcomm eXtensible Diagnostic Monitor
4. QC BS analizer - similar to mbn-resourcer, but other maintainers
5. BREW Resource editor - it's more for high level programming ...
that will be primarilly tools for decryting ... but crypting - possible need signing key for creating .mbn and that key must be trusted by device ...
@ DomZ
Thanx for info.
@ misterjp
Nice work.
@ all
Like DaLiV said. I think also. It is an good idea to collect more infos about Qualcomm. Many manufacturer use Qualcomm Hardware...
Samsung, LG, Motorola, Huawei... many more.
AMSS is ever used. If Qualcomm MSMxxxx works as heart of mobile.
Best Regards
Edit...
1. I have looked into QPST 2.7 Build 264 (good Description in Manual) See Screenshot...
2. It looks that RSA2048 Signature at End of AMSS is not used.
Check amssHD.mbn
First 2 adresses the same and means End of AMSS... And 0x1D is 00 instead 01
Why the other 2 adresses exceed AMSS? I don't know...
So in theory Manipulation of AMSS seems possible.
3. BREW is "obsolete". Not an part of AMSS. Because the WM OS is used as an "Layer"...
4. NVM folder I can see... So I think also NV items are used...
5. Funny GZIP is used also from BenQ for EF91... To hide AMSS...
I agree with you. Now that misterjp has partially reversed dz format, we need to concentrate on files contained inside it (as you may all know, lgmdp can actually flash firmware from a directory containing mbn+bin files, not only dz, so there is no point in trying to reassemble a dz from a list of mbn files (at least, it's not a priority).
On the other side, misterjp has proved to be a good reverser on a format for which he doesn't even own a phone. To be totaly honest, (and as I already said on a french forum), he did this on my demand because I was not really efficient in reversing lgmdp. I'm pretty sure he could go further (and with no doubt could cook an english firmware, for those of you waiting for this). I think the least thing would be for him to have a phone to test his development. So misterjp, if you're ok to continue your work and would like some donations, don't hesitate to update your first post with a paypal link, at least til you get a phone.
Hi there,
Thanks for the support ! I did not have many time this week end, so I do not have many informations today : the flash.bin is using a Windows flash format, very easy to extract :
Code:
Offset Length (bytes) Description
0x0 0x7 Magic code "B000FF\n"
0x7 0x4 Image start adresse
0xB 0x4 Image Length
0xF --- List of memory block
-- 0x4 Always zero
-- 0x4 Image entry point
-- 0x4 Always zero
Each memory block is composed of
Code:
Offset Length (bytes) Description
0x0 0x4 Memory destination of block
0x4 0x4 Block size
0x8 0x4 Block checksum (sum32)
0xC --- Data
JP.
So,
I have sucessfully extracted the imgfs from the flash file. So here is the tool to do it.
It's a bit late, so I'm not giving explanation now, but I will !
To use the tool, just run
Code:
LGFlashParser.exe FLASH.bin
It will generate 3 files (BOOT, RAW and IMGFS), then you can use ImgfsToDump to extract the content of the IMGFS file.
At the moment, ImgFsToDump crash at the end of the processing (but the content is extracted), this is probably due to an invalid char in my generated imgfs, but it's too late to investigate.
Apparently, some files extracted from the imgfs part have no data. It could be related to the problem exposed before. I am working on this problem.
I have also checked the part_2_RAWFS : for this file, you can use the dumprom utility to extract it's content :
Code:
dumprom.exe part_2_RAWFS.bin -5 -d dump-xip
Update November 11, 2009
CFC and CFC GUI have been updated to 0.60 and 0.60.35 respectively. CFC runtime files have been updated. This thread has also been update to add information about the new file formats.
Changes:
- (CFC GUI) Support for new filenames (and categories) used in TF3D v2.5+
- Support for new file formats (read only, "replace" write is always in old format) used in TF3D v2.5+
- Support for new compressed formats that correspond to the new formats mentioned above
- (CFC GUI) If you are using this with 2.5, please do read the 2.5 specific notes (in the CFC GUI thread).
- At this time I STRONLY recommend patching Manila files manually instead of using the auto-patching! See the tech thread referenced below.
Update June 5, 2009
CFC and CFC GUI have been updated to 0.55 and 0.55.25 respectively. The CFC runtime files have not been updated. Changes:
- Better compression ratio (backwards compatible with old runtime)
- UltraHQ en/de-coding support (CFC)
- "Patch Manila on device" function has been improved to work-around the issue of Manila no longer starting that some people have (CFC-GUI)
Update April 20, 2009
CFC and CFC GUI have been updated. Many changes! For the changelog see the CFC GUI thead. The CFC runtime has also been updated to v0.50, so there is a new libgles_mn file required. See the cfc-support-dll zip attached a few posts down.
Update Feb 26, 2009
CFC and CFC GUI have been updated to 0.46.15. Fixed an issue with PNGs without alpha channel.
Update Jan 15, 2009
CFC Live Patch 0.45.01 has been released. You can run this tool on your phone to make your current Manila installation compatible with CFC compression.
See this post.
Update Jan 14, 2009
CFC GUI 0.45.15 released. No changes to CFC core, only GUI stuff. Attached download updated.
CFC GUI thread: http://forum.xda-developers.com/showthread.php?t=470798
Changelog: http://forum.xda-developers.com/showpost.php?p=3164604&postcount=3
Update Jan 13, 2009
CFC 0.45 and CFC GUI 0.45.12 released.
CFC (core) changes:
- Heavily modified encoding algorithm. It is often slower but the quality should be much better. Please read this post!
CFC GUI changes:
- No longer freaks out if the wrong file attributes are set on some files
- Added background color selection
- Added tool to patch a complete Manila package for CFC compatibility and optimization
Also, non-technical discussion of CFC GUI (only) should go to this thread:
http://forum.xda-developers.com/showthread.php?t=470798
Update Jan 12, 2009
Yes I know, too many updates today!
CFC 0.42 attached. Was something weird going on with the encoding sometimes. Furthermore it seems like the encoding works great on the original files, but it drops the ball here and there with other files. Going to look into that ASAP and finetune the encoding algorithm!
Most important about this update is, CFC now also comes with a GUI!
Update Jan 12, 2009
CFC has been updated to version 0.4. Added features are:
- QTC -> PNG (+- 30% faster than 0.4b2)
- PNG -> QTC
- CFC -> QTC (finally)
- QTC padding
- QTC trimming
With QTC <-> PNG conversion now available from CFC, it seems the Compressonator is no longer needed!
Further EDIT: All posts updated.
Further EDIT: 0.41 added... There was a small bug in CFC compression in 0.4, it didn't always set PayloadSize correctly, which could create errors with padding/trimming.
Update Jan 8, 2009
CFC has been updated to version 0.3. It can now fully handle the RGB format as well. CFC compression has been slightly optimized. QTC and ATC headers are now completely written correctly.
This now also allows for larger than original images
Also lots of info in the first 6 posts has been updated to reflect these changes and add information.
Update Jan 7, 2009
Thanks to myself and D-MAN666, it seems the QTC format is now completely known!
Also today brings CFC compression for (W)VGA devices, if chefs choose to implement it. The needed stuff is here.
The CFC tool itself still needs an update (0.3 ?) to handle QTC/ATC_RGB conversion to ATC/ATC_RGB conversion (and back) and to decompress the CFC files. Donor headers won't be needed anymore then either in some cases (but they will remain handy in others!), and with that some parts of the first 5 posts will have to be rewritten as well (sigh...)
Note that some other parts of the first few posts are marked with changes. Look for the red letters.
- end of updates -
Intro
As some of you may have seen, me, djboo, pcarvalho, "he who shall not be named" and several other enthousiasts have ported TouchFlo3D (to be specific, the version that came on my Raphael) to QVGA. A large part of this effort involved hacking into Manila/Mode9 and even OpenGL ES itself to get it operating decently, but after that, a lot of effort went into optimizing, which was largely done by scaling the images. While working on this, I encovered a lot of information and wrote quite a bit of code to 'get it done'. As always, a lot of it ultimately redundant, but we did pull it off! (barring some issues that are driver related).
Acknowledgements
Before continuing, people need to be acknowledged for their parts in this. I could hardly have done it alone. A lot of these credits go out to people involved in the TF3D QVGA porting, but also drivers porting, they all had a hand in this information being 'discovered', and hence are mentioned here.
- djboo Keeping all this stuff going. I looked at this stuff once when herg did an attempt, never really got involved, but because of him did get involved this time around. As he seems to be forgotten in the credits here and there, he's #1 in this thread!
- pcarvalho A bit of competition that led to great stuff. In the end, our intended methods of porting complement eachother nicely - 'my' part got it going, but the QVGA port didn't really shine 'til we did the other things 'his way' as well.
- "he who shall not be named" The anonymous HTC-CA hacker, about whom probably everybody knows who he is (it aint me ) Did some cracking work on this too.
- The P3D Team A bunch of them did a lot of testing
- D-MAN666 Mentioned last but certiainly not least! Cracked the file format first, and generally found out and published a hell of a lot of information. Also the author of Manila Editor.
Requirements
For all this, you will need and/or want the following:
- "The Compressonator", image conversion/compression/viewer tool by ATI/AMD. You may need to sign up at the AMD website, but it is free of charge and I haven't received any spam from them yet. Update: The version of Compressonator on AMD's site is no longer able to do ATC. The correct version of Compressonator is attached to this post. Update: The Compressonator is now no longer strictly needed due to CFC being able to do the QTC en/decoding.
- "Manila Editor", Manila image editor. You will not be using this for the actual good stuff, but you may be using this for testing things quickly, and you will definitely want to use it for finding the files you actually want to replace. Update: no longer needed, use the CFC GUI instead!
- CFC, (attached), tool by yours truely to convert between QTC, ATC, CFC and PNG formats
- Knowledge of the Windows command line - Though CFC now comes with a GUI as well, yay!
- The files attached to this post. These are all the images from Manila (the version that came with my Raphael) converted to PNG by "The Compressonator". It's kind of a ***** to do, so if you want to save yourself some trouble, just see that post. VGA as well as (rescaled using Lanczos3) QVGA images are attached to that post. Update: this image packs need to be updated, will do some time
CFC download
Notice: you do not need the compressonator files to use CFC. They are just here in case you want to do things the old-school way
( < 0.60 : 3408 downloads )
Textures, ATC, CTES, QTC, CFC (tech background)
The imageon 3D chip in our devices support texture compression, and Manila (Mode9) uses this. The format used is a special format created by AMD/ATI for low size and lower power use on mobile devices, called ATC (ATI Texture Compression).
There are three ATC formats:
- ATC_RGB: 4 bits per pixel (4 bits RGB)
- ATC_RGBA_EXPLICIT_ALPHA: 8 bits per pixel (4 bits RGB + 4 bits Alpha)
- ATC_RGBA_INTERPOLATED_ALPHA: 8 bits per pixel (not sure on the format)
Almost all images used in Manila are of the 'ATC_RBGA_EXPLICIT_ALPHA' variant, and this article will focus on these. ATC_RGB is also used for a small number of images, though I have not further investigated its format.
The image data for these formats are stored in one of the following file formats:
- ATC: The file format generally used by AMD/ATI
- CTES: ATC related, some weirdness, see below. Seems to be forward compatible with ATC, but not backwards.
- QTC: Qualcomm adapted version of ATC, used by Manila/Mode9
The formats are very similar, though we will focus only on ATC/QTC, that's all we need.
Image data (ATC_RGBA_EXPLICIT_ALPHA) - Updated January 12, 2009
A lot of the original information comes from D-MAN666's posts here.
I will skip over the headers (32 bytes), they are listed below for ATC and QTC specifically. This is about the actual image.
The image is divided into blocks of 4x4 pixels. An 8x8 image would be stored like this: (A, B, C and D are 'pixel blocks')
AAAABBBB
AAAABBBB
AAAABBBB
AAAABBBB
CCCCDDDD
CCCCDDDD
CCCCDDDD
CCCCDDDD
A 4x4 pixel block is 16 pixels and 16 bytes. So in effect, you can see it as 8 bits per pixel. An image is always stored using these 4x4 pixel blocks. A 5x5 images would thus use 4 blocks!
bytes 0-7 are alpha values for each pixel, 4 bits per pixel (4 bits * 16 = 64 bits = 8 bytes) - this is not present for the ATC_RGB format. Scale these to the 0..255 range by multiplying each alpha value by 17.
bytes 8-11 are color keys, there are two keys of 16 bits (2 bytes). The keys are stored like this:
word 1: XRRRRRGG GGGBBBBB (1-bit method, 5-bit R, G, B)
word 2: RRRRRGGG GGGBBBBB (5-bit R, 6-bit G, 5-bit B)
Where X is the decoding method to use, there are two.
bytes 12-15 are mixdown values, 2 bits per pixel (2 bits * 16 = 32 bits = 4 bytes). The per-pixel mixdown value, combined with the color keys mentioned above determine the actual color that is outputted. You must scale the scolor keys to the 0..255 range and apply a formula to them.
Code:
if HasAlpha then begin // skip for ATC_RGB
sIn.Read(dw, 4); // read dword
for i := 0 to 7 do begin
alpha[i] := (dw and $F) * 17;
dw := dw shr 4;
end;
sIn.Read(dw, 4); // read dword
for i := 8 to 15 do begin
alpha[i] := (dw and $F) * 17;
dw := dw shr 4;
end;
// alpha[0..15] now contain the scaled 4x4 pixel block alpha values
end;
sIn.Read(w, 2); // read a word, key1
Method := (w shr 15);
Keys[iR, 0] := ((w and $7C00) shr 10) * (255/31);
Keys[iG, 0] := ((w and $03E0) shr 5) * (255/31);
Keys[iB, 0] := (w and $001F) * (255/31);
sIn.Read(w, 2); // read a word, key2
Keys[iR, 1] := ((w and $F800) shr 11) * (255/31);
Keys[iG, 1] := ((w and $07E0) shr 5) * (255/63);
Keys[iB, 1] := (w and $001F) * (255/31);
sIn.Read(mixdown, 4); // read a dword, mixdown values
for i := 0 to 15 do begin
pixels[i] := (mixdown and $3);
mixdown := mixdown shr 2;
end;
// pixels[0..15] now contain the still-encoded 4x4 pixel block mixdown values
When thinking about the color keys and mixdown values, think of the keys as a color-range start and end value. The mixdown values decide which value to pick inside the range. (for each R,G,B)
For example, let's take a key1 of 10 and a key2 of 40 for Green. Then the decoded Green values would be:
Code:
mixdown 00 01 02 03
value 10 20 30 40
This is only true, however, when the 'method' bit (X) is 0. Full decoding of both methods:
Code:
for i := 0 to 15 do begin
a := alpha[i];
if (method = 0) then begin
r := Round( Keys[iR, 0] + ((pixels[i] / 3) * (Keys[iR, 1] - Keys[iR, 0])) );
g := Round( Keys[iG, 0] + ((pixels[i] / 3) * (Keys[iG, 1] - Keys[iG, 0])) );
b := Round( Keys[iB, 0] + ((pixels[i] / 3) * (Keys[iB, 1] - Keys[iB, 0])) );
end else begin
case pixels[i] of
0: begin
r := 0;
g := 0;
b := 0;
end;
1: begin
r := Round( Keys[iR, 0] - ((1/4) * Keys[iR, 1]) );
g := Round( Keys[iG, 0] - ((1/4) * Keys[iG, 1]) );
b := Round( Keys[iB, 0] - ((1/4) * Keys[iB, 1]) );
end;
2: begin
r := Round( Keys[iR, 0] );
g := Round( Keys[iG, 0] );
b := Round( Keys[iB, 0] );
end;
3: begin
r := Round( Keys[iR, 1] );
g := Round( Keys[iG, 1] );
b := Round( Keys[iB, 1] );
end;
end;
end;
end;
Both methods have various way of formulating them. I thought the ways I have chosen here are clearest for how it works.
Update Jan 7, 2009
Image data (ATC_RGB)
The image data here is exactly the same as ATC_RGBA_EXPLICIT_ALPHA, except that the alpha bits aren't includes. So, each 16-pixel block becomes 8 bytes instead of 16, as bytes 0-7 from ATC_RGBA_EXPLICIT_ALPHA are not there. This means 4 bits per pixel.
- end of update -
ATC, CTES, QTC
This image data seems to be the same across all formats - and it should be, as this data is sent directly to the 3D chip. You would not want to have to process it first.
Let's first pick out CTES, as I have very little to say about it. It seems to be nearly the same as ATC and QTC, however, for some reason, "The Compressonator" will output CTES files we can use as ATC, but will not read our own Manila-based ATC's in CTES mode (only in ATC mode). What's up with that? I don't know. Perhaps one of you will figure it out.
QTC header
Code:
Magic: DWORD; // 0x31435451 : "QTC1"
Magic2: DWORD; // always 1 ?
Width: DWORD;
Height: DWORD;
Format: DWORD; // 0x14, 0x15
Dummy1: DWORD; // formerly known as Unknown1, may be 0 - Jan 7, 2009
PayloadSize: DWORD; // formerly known as Unknown2 - Jan 7, 2009
Dummy3: DWORD; // formerly known as Unknown3, may be 0 - Jan 7, 2009
The meaning of the unknowns has not been deciphered yet. Setting them to weird values does muck-up the decoding of the images, however, they do not seem to be actually sent to the 3D chip. Or perhaps I just have not found where and when!
For format, 0x14 is used for ATC_RGB_EXPLICIT_ALPHA. The small number of images that use 0x15, I suspect, are ATC_RGB. Either way, they do not decode using the ATC_RGB_EXPLICIT_ALPHA method and I know ATC_RGB is used some places, so it would make some sense to make this assumption.
Update Jan 7, 2009
Unknown2 has been replaced by PayloadSize, thanks to myself, D-MAN666 and eidolen.
The PayloadSize is the number of bytes after the header that contain content.
For images of type 0x14 (ATC_RGB_EXPLICIT_ALPHA) this is: Width * Height, where both Width and Height are multiples of 4, due to how the format itself works, in other words: (RoundUp(Width / 4) * 4) * (RoundUp(Height / 4) * 4).
For images of type 0x15 (ATC_RGB) this is half of type 0x14, because ATC_RGB uses 4 bits per pixel instead of 8. The multiples of 4 rule still stands, so the PayloadSize is: Round(((RoundUp(Width / 4) * 4) * (RoundUp(Height / 4) * 4)) / 2)
Note that all Manila image files (at least the ones I have) are padded to be a multiple of 512 bytes in size. This is probably a speed optimization for when these files are cooked into a ROM.
Dummy1 and Dummy3 (aptly renamed from Unknown1 and Unknown3) seem to be irrelevant. After we figured out how PayloadSize (Unknown2) was relevant, we tried blanking out Dummy1 and Dummy3 with 0's, and TF3D still works without a hitch. The original values do not seem to be related to the dimensions nor the payload size, and they are not sent to the graphics chip either.
- end of update -
Update November 6, 2009
Manila 2.5 uses 4 additional file formats:
0x01 - 8888 RGBA, 32bpp
0x02 - 888 RGB, 24bpp (I have not encountered an actual image in this format, so processing may be faulty by CFC and CFC GUI)
0x03 - 565 RGB, 16bpp
0x05 - 4444 RGBA, 16bpp
- end of update -
ATC header
Updated 08/Jan/2009
Code:
Magic: DWORD; // 0xCCC40002
Width: DWORD;
Height: DWORD;
Format: DWORD; // ATC_FORMAT, 0x01 for RGB, 0x02 for RGBA_EXPLICIT_ALPHA, 08/Jan/2009
Magic3: DWORD; // 0x20 ... mucks up colors... not clear?
Magic4: DWORD; // 0x01, 08/Jan/2009
Magic5: DWORD; // 0x01, 08/Jan/2009
FormatMagic: DWORD; // 0x8C92 for RGB, 0x8C93 for RGBA_EXPLICIT_ALPHA, 08/Jan/2009
- end of updated content -
CFC format + historic Compressonator editing
CFC format
I use the CFC format (yes, that's why the tool is called CFC) for the Manila QVGA port. It saves a lot of space and even seems to improve performance a bit. It uses standard gzip/zlib compression on the QTC image data (which compresses to about 20% on average) and hides the compressed data inside the QTC file itself. Decompression of this is over 5 MB/s on our devices, but images are only a few KB each. The proxy libgles_cm is what actually decodes this and sends the decompressed data to the 3D chip.
CFC adjusts the QTC header to the proper values. Beware when doing this yourself that Mode9 uses these values internally as well). The image data ('payload') is replaced as follows:
Code:
Magic: DWORD; // 0x31434643 : 'CFC1'
Format: DWORD; // CFC_FORMAT...
Width: DWORD;
Height: DWORD;
CompressedSize: DWORD;
UnCompressedSize: DWORD;
... compressed data ...
Format can be one of the following:
Code:
CFC_FORMAT_COMPRESSED_QTC_RGBA_EXPLICIT_ALPHA = 0x3001; // 0x14 from QTC
CFC_FORMAT_COMPRESSED_QTC_RGB = 0x3002; // 0x15 from QTC - used since CFC 0.3
CFC_FORMAT_COMPRESSED_RGBA = 0x3101; // April 20, 2009 - RGBA format - used since CFC 0.5
CFC_FORMAT_COMPRESSED_RGB = 0x3102; // April 20, 2009 - RGB format - used since CFC 0.5
Width and height are included for historic reasons, and it also opens up the possibility to do some weird mods. RGBA format is included for possibly allowing use of uncompressed textures for Manila support on non-HTC/Qualcomm/ATI/AMD based devices.
Update April 20, 2009
(A)RGB formats are gzip/zlib compressed just as QTC/CFC variants and require the CFC 0.50 runtime files. The uncompressed data is actually stored as (height x width x) BGR(A) (from x86 viewpoint) as this is the format the graphics chip can handle.
Update November 6, 2009
The following formats have been added to CFC. Note that the QTC header always says RGB or RGBA_EXPLICIT_ALPHA. This actually allows the new formats to be used on older Manila versions that do not directly support them, if you are using CFC 0.60 runtimes.
Code:
// CFC 0.60 additional formats
CFC_FORMAT_COMPRESSED_QTC_8_8_8_8 = 0x3003;
CFC_FORMAT_COMPRESSED_QTC_X_8_8_8 = 0x3004;
CFC_FORMAT_COMPRESSED_QTC_5_6_5 = 0x3005;
CFC_FORMAT_COMPRESSED_QTC_4_4_4_4 = 0x3007;
Notice: the text below in this post is here for historic reasons. It is no longer completely relevant
Right, well I did tell you to get "The Compressonator" from AMD's site, right? You should have done this by now. You should also have the CFC tool attached to the first post of this thread.
Say we want to manipulate an image from Manila. First we need to find out which image it is. Easiest way to find them is to use Manila Editor (also linked in first post), so you get the 'magic' filename.
You may want to ask why we dont simply use Manila Editor for doing these things, simply put, the image quality from The Compressonator is better than the current version of Manila Editor. Also, we can do stuff in batch in The Compressonator.
Manila to PNG (single)
Now, say this image is 7d3f1247_manila (the globe on the internet page), we use the CFC tool to convert it to ATC format:
Code:
cfc -qa 7d3f1247_manila 7d3f1247_manila.atc
You can open this file in The Compressonator. It may look a bit weird, because alpha is not displayed. Right click on the image and select "Show RGBA", there, that looks better.
Something you will not directly notice with the globe image, but you will with other images, is that the image is UPSIDE DOWN. You will need to flip the image over if you want to put it back into Manila. For some reason I haven't figured out yet, decoding goes upside down, but encoding needs to be the correct side up.
Now we may wish to edit this file, so we save it as PNG: File -> Save Original.
Open it in photoshop, flip it vertically, and save it.
Manila to PNG (batch)
We will need to use batch mode to convert back to Manila anyways, so lets just start using it for converting it to PNG as well (for some reason doing it non-batch doesn't work right).
This will assume you have a bunch of .atc files in a directory. Batch converting Manila (QTC) files to ATC files is also possible with CFC:
Code:
cfc -qaf orgfiles atcfiles
Assuming you have your original Manila files in the folder 'orgfiles' and created a new empty directory 'atcfiles'.
Open The Compressonator, and go to File -> Batch Compress (or press F4). Navigate to your folder containing all the ATC files, set the "Files of type" box to "ATC Textures (*.ATC...)". set the "Output File Format" to PNG and "Output Format" to "ARGB8888". Punch the "Compress All" button and wait a bit.
Note that some files will not decompress correctly and crash The Compressonator. You will have to look at the crash dump to find out which file was the culprit and remove it from your batch directory. IIRC, there are about 10 files that have this issue, so be prepared for 10 minutes of infuriating work.
You MUST set "Output directory" to "Use Input Directory", or you will not be able to decompress more than one file!
In the end, you will have a large bunch of PNG files. Note that these PNG files are also available already done for you, see the link in the first post of this thread.
Files known to crash The Compressonator: 08/Jan/2009
Code:
00ad7edb_manila.atc
056e5c7f_manila.atc
063f5858_manila.atc
0c175082_manila.atc
2255b55f_manila.atc
24720929_manila.atc
39064485_manila.atc
4a209508_manila.atc
PNG to Manila (single + batch)
Even for single files, we are using the batch function, as there seems to be an issue with doing this in The Compressonator normally.
The operation is exactly the same, but for single you select the file and press "Compress", and for many files you do not select a file and press "Compress All".
Note that as previously mentioned, decoding ATC to PNG files results in the PNG's being upside down, but to make ATC files from PNG files the correct side needs to be up!
This time around we set "Files of type" to "PNG Textures" (duh) and "Output File Format" to "CTES Textures". As previously mentioned CTES is compatible with ATC, but ATC is not compatible with CTES. You won't notice this though.
The magic is the "Output Format" setting. Set it to "CTES Texture Compression" and hit the "Options" button. In the "Compress Texture" dialog that pops up, select "ATC RGBA Explicit Alpha (8 bits per pixel)", or "ATC RGB (4 bits per pixel)", depending on which format you want, hit OK, and you're there. Hit "Compress" or "Compress All", and wait 'til it's done.
You MUST set "Output directory" to "Use Input Directory", or you will not be able to decompress more than one file!
Now we want to convert these CTES/ATC files back to Manila files, and for this, again, we use the CFC tool:
Code:
cfc -aqf atcfolder qtcfolder orgfolder
You can also use -aq instead of -aqf for a single file. Note that the CFC tool does NOT change filenames, so you have some renaming to do.
Update 08/Jan/2009
With CFC 0.3, donor headers are not longer necessary, and have become an optional parameter.
- end of update -
Rescaling to QVGA
Converting Manila images to QVGA is pretty simple. Just use the techniques described above.
What you want to do is scale ONLY images 32x32 and larger, and you will want to divide the width and height exactly by two. That's all there is to it.
If you have a bunch of PNG files you want to scale, the CFC tool can even do this for you, including the needed vertical flip:
Code:
cfc -nsf vgapngfolder qvgapngfolder
This will rescale using the Lanczos3 algorithm.
CFC Compression
The QVGA port supports the CFC format as mentioned above. This can save a lot of space and is the preferred way of using textures for the QVGA port.
To compress your QVGA QTC files to CFC:
Code:
cfc -cf qvgaqtcfolder qvgacfcfolder
CFC tool (commandline)
As CFC offers a lot of options, many of them related to the pre-0.4 way of converting images, I'll take a very short time to explain the most relevant CFC options, what they do, and when/why you should use them. Most of these are available in the GUI as well.
Convert QTCs to PNGs
Code:
cfc -qp in-filename out-filename
cfc -qpf in-folder out-folder
Since CFC 0.4 this should be the preferred way to convert Manila's QTC files to PNG's (it's pretty fast and saves a lot of steps and trouble compared to using other CFC options and the Compressonator). These images come out with correct side up, so no longer is there a need to flip them manually with CFC or Photoshop or whatever.
Note: This does not handle CFC compressed QTC's, you will have to decompress those first!
Convert PNGs to QTCs
Code:
cfc -pq in-filename out-filename
cfc -pqf in-folder out-folder
Obviously if you want to convert QTCs to PNGs there's a good chance you also wish to do the reverse. Introduced in CFC 0.4b2. CFC will automatically detect if RGBA_EXPLICIT_ALPHA or RGB is the most optimal QTC format to use.
Quality rivals the Compressonator, but CFC is quite a bit slower (though the saved number of steps save you more time).
Note: The output are not CFC compressed, you will have to do that manually.
Compress QTCs to CFCs
Code:
cfc -c in-filename out-filename
cfc -cf in-folder out-folder
Compressed QTCs (CFCs) are usually much smaller than the original QTCs, and provide a performance boost as well. However, you will need a patched Manila to be able to use CFCs. The QVGA port is patched to do this, and instructions on how to other Manila versions can be patched are included a few posts below this one.
Decompress CFCs to QTCs
Code:
cfc -d in-filename out-filename
cfc -df in-folder out-folder
This should speak for itself
Trim QTCs
Code:
cfc -t in-filename out-filename
cfc -tf in-folder out-folder
Saves some space on your hard-disk, removes unnecessary data from the QTC files (also works on CFCs).
Pad QTCs
Code:
cfc -p in-filename out-filename
cfc -pf in-folder out-folder
This makes the QTCs (and CFCs) a multiple of 512 in bytes in size. HTC originally did this with all their images. It seems to improve performance when Manila is cooked in.
Scale PNGs VGA->QVGA
Code:
cfc -ps in-filename out-filename
cfc -psf in-folder out-folder
For the theme and Manila porters. Note that Manila in QVGA can handle VGA textures (and vice versa) just fine, however, using the correct image size for the Manila resolution does improve performance quite a bit.
Complete QVGA port example
Updated 13/Jan/2009
You can now do this whole thing with a single push of a button in the CFC GUI: Tools, Scale QVGA -> VGA
- end of update -
Updated 12/Jan/2009
The text below has been updated to reflect changes with CFC 0.4
- end of update -
This is the complete rundown of how I converted all the VGA images from Manila to QVGA images, the same method can be applied to themes and what not.
First, create a directory somewhere, and put the CFC.exe in it. Then go to the command line and change directory to this new directory.
We will want to create a bunch of directories:
1) Setup
Code:
mkdir org png pngscale qtc cfc out
Dump all your original manila files in org (in this case, the entire manila, but you could just dump your theme in there instead)
2) Convert QTC to PNG
Code:
cfc -qpf org png
3) Scale PNG images from VGA to QVGA
Because CFC will not scale very small images, we need to make sure the output folder (pngscale) has all the images we want first. The ones that are scaled will be overwritten by CFC:
Code:
copy png\*.png pngscale\*.png /y
Do the scaling:
Code:
cfc -psf png pngscale
4) Convert PNG back to QTC
Code:
cfc -pqf pngscale qtc
5) Convert QTC to CFC (optional)
Code:
cfc -cf qtc cfc
6) Pad QTC/CFC (optional, for cooking)
Code:
cfc -pf cfc out
2-6 in one go:
Note that this does not include making the directory and placing your Manila in the org folder.
Code:
cfc -qpf org png
copy png\*.png pngscale\*.png /y
cfc -psf png pngscale
cfc -pqf pngscale qtc
cfc -cf qtc cfc
cfc -pf cfc out
--
Et voila, most of our images have now been rescaled to QVGA, are compressed for optimum filesize, and located in the qtc, cfc, or out folder, depending on which steps you skipped.
Also note we retouched some images by hand for the QVGA version.
CFC for VGA and WVGA
!IMPORTANT! Updated 11.11.2009: CFC GUI updated. Support DLLs updated to CFC 0.60!
!IMPORTANT! Updated April 20, 2009: CFC Live Patch discontinued!
I have recompiled the libgles proxy file originally made for QVGA to a version that only handles the CFC compression, and should work on any 'normal' Manila 3D version, like the ones found on the Touch Diamond, Pro and HD.
If you cooks/chefs/whomever implement this, you can reduce TF3D's size footprint by half (7 mb smaller in my test). This also makes a positive performance difference, as the on-the-fly decompression of the images is actually faster than the flashdisk access.
Patching TF3D to be able to use CFC compression also allows theme makers to make faster and smaller themes.
Instructions to modify TF3D VGA/WVGA - for users NEW!:
FOR v2.5 AND NEWER PATCH MANUALLY!
- Get CFC GUI, attached to the first post of this topic.
- Connect your device using ActiveSync
- Use the "Device->Patch Manila on device" feature
Instructions to modify TF3D VGA/WVGA - for chefs:
FOR v2.5 AND NEWER PATCH MANUALLY!
- Get CFC GUI, attached to the first post of this topic.
- Run CFC GUI and select the folder where your Manila package is stored
- Make sure you got a backup of said folder
- Go to Tools -> Patch Manila
The following things will be done to your Manila package:
- All image files will be CFC compressed (lossless, faster)
- All image files will be padded to be a multiple of 512 bytes in size (faster)
- All image files will be set to System/Hidden/Archive file attributes
- Manila.exe will be patched to use libgles_mn.dll
- Manil2.exe will be patched to use libgles_mn.dll (depending on Manila version)
- Mode9.dll will be patched to use libgles_mn.dll
- libgles_mn.dll will be placed in the package
- zlib_mn.dll will be placed in the package
If no errors occur, there is nothing else you need to do, aside from cooking the package
Instructions to modify TF3D VGA/WVGA - by hand
Attached is a zip file (cfc-support-dlls) with the two DLLs you need: libgles_mn.dll and zlib_mn.dll . These must be placed in your \Windows folder.
Next, hex edit Manila.exe, Manil2.exe and Mode9.dll to use libgles_mn.dll instead of libgles_cm.dll. Just search for "libgles_cm.dll" in the files and replace it with "libgles_mn.dll". These values may appear multiple times in the file! Make sure to search and replace for both ANSI and UNICODE variants! Your Manila version may not have both Manila and Manil2, or only one of them may contain the "libgles_cm.dll" string. This is normal.
You should also modify HKLM\Software\OEM\MASD\Manila and append _CFC to the version string. This so people can recognize if their installation supports CFC.
This will only add support for CFC to your Manila install, it will not make images and such actually use CFC. But you can now support themes that do use CFC.
Note to people who used an older version of this patch
You don't have to do it again, but the automated tool makes sure everything is done right. If you still have zlib1.dll from the old patch on your device, do not remove it unless you want to break the "Teeter" game.
You can run the automated tool over a package you used the older patch on - it will clean it up.
Wow. I can't wait. So is it possible then to run TF3D on an older VGA device? Or is this to help us decode the themes. Isn't james making a image converter from vga to qvga?
Kraize92 said:
Wow. I can't wait. So is it possible then to run TF3D on an older VGA device? Or is this to help us decode the themes. Isn't james making a image converter from vga to qvga?
Click to expand...
Click to collapse
everything has been converted to qvga
CraZyLiLbOy said:
everything has been converted to qvga
Click to expand...
Click to collapse
True, but a tool would still be nice because the skins of tf3d are all in vga. If we had a converter, I could apply the theme and everything, and then convert it to qvga.
Kraize92 said:
True, but a tool would still be nice because the skins of tf3d are all in vga. If we had a converter, I could apply the theme and everything, and then convert it to qvga.
Click to expand...
Click to collapse
You don't have to convert anything to qvga. All touchflo 3d themes work on our qvga devices. You are using touchflo 3d interface from the diamond. You can apply any themes you want for the touchflo 3d. I know the diamond is a vga device but it still works. I'm saying this because I've tried it and it works, mark my words
The QVGA port was made specifically in mind with that VGA stuff will just work - and they will. However, when I finish this guide and you have read it, you will understand exactly how you can significantly optimize the QVGA version - and make the VGA version better
I don't know man but I think I fall in love with the touchflo 3d. Thanks to Chainfire and djboo.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Great TUT very interesting. thanks for sharing your tools
yesssss thank you for your great tuto,
2Chainfire
Didn't understand how to convert single PNG to manila's VGA file
Could u give me more examples?
any progress???
Any progress... on what? This tutorial is finished. TF3D-QVGA is finished. All we're waiting for is a new driver release that fixes the power issues.
i dont mean to sound like a complete idiot, but will this ever be in cab form?
i have no idea what most of what you are talking about means in this tutorial, I am just legitimately curious if it is even possible.
also if that is not possible, wouldnt the old touchflo3d porting-to-qvga cab be necessary along with this tutorial? maybe i am just seriously confused and have no idea what is going on
disregard if this is too much ignorance in one post!
thanks
htctoucher
htctoucher said:
i dont mean to sound like a complete idiot, but will this ever be in cab form?
i have no idea what most of what you are talking about means in this tutorial, I am just legitimately curious if it is even possible.
also if that is not possible, wouldnt the old touchflo3d porting-to-qvga cab be necessary along with this tutorial? maybe i am just seriously confused and have no idea what is going on
disregard if this is too much ignorance in one post!
thanks
htctoucher
Click to expand...
Click to collapse
I'm not quite sure what you are after with this CAB you ask for. If you are looking for TF3D for QVGA, a CAB is available (but there are still some issues with the required 3D drivers for the QVGA devices).
The modifed TF3D executable files work fine on QVGA with the original graphics. This thread however explains the method to do pixel-perfect TF3D image manipulation (for themers, porters, etc) in general, which was up until the documentation of this thread impossible (read: unknown). Methods like using Manila Editor work, but degrade image quality significantly more than the method described here.
However, this thread also documents converting the (originally) VGA TF3D images to the corresponding QVGA sizes, and even adding an extra 5x compression to them. This is not necessary to do if you have the modified TF3D executables for QVGA (it will use the VGA graphics files just fine), however, it certainly makes TF3D a hell of a lot faster and significantly reduces the size it requires on disk and the memory use (both of which are more limited on the QVGA devices compared to the VGA ones). It includes this information because this method was 'discovered' in the process of porting TF3D to run on QVGA devices. Similarly, porting existing TF3D VGA themes to QVGA resolutions will increase the speed of TF3D on QVGA devices over using the original theme's VGA graphics files.
Note that the TF3D-QVGA CAB ( http://www.htcclassaction.org/download/TF3D-QVGA.cab ) already includes the optimized QVGA graphics, and some of them were even retouched by hand (not mentioned in this threaD).
I hope this explains what you want to know
Hi I have managed to convert a manila file to ATC format using cfc.exe but now I cant open it using The Compressonator. If I go to open (there is no ATC file type listed in 'Files of type' drop down menu), select All Files (*.*) and try to open the converted .atc file, it does not load. I am using The Compressonator 1.50 which seems to have been updated recently (18-12-08) so it might be an issue with the new version. If it is that can you please upload or direct me to the old version or else if I am doing something wrong, please help.
Thank you
AF241
After having done some research in changing the pagepool on the new 2.0x based diamond roms I have made some interesting discoveries.
The new roms have another 4Mb missing e.g.
2.0x ROMs with 12Mb pagepool have 107.02Mb Program Memory, 8Mb Roms have 111.02Mb, 0Mb Roms have 119.02Mb & if you see 113.02Mb then pagepool is 6Mb
In Comparison
1.93 ROMS with 12Mb pagepool have 111.02Mb Program Memory, 8Mb Roms have 115.02Mb, 0Mb Roms have 123.02Mb & if you see 117.02Mb then pagepool is 6Mb
So this means all new 2.0x Roms are actually set to 12Mb Pagepool and not 16Mb as previously thought.
The best way to definitely be sure what your pagepool is is to run devhealth.exe on your device. This will create the following file on your internal storage mem_1.txt. This file will detail how your physical memory is shared out into pagepool and other areas. 2.00Roms show up in devhealth as having 120.12Mb of total physical memory (including pagepool) and previous roms have 124.12Mb total memory.
Devhealth.exe is available at the following link:
http://forum.xda-developers.com/showthread.php?t=416334
Most of the time there is not much point in editing the pagepool on these new roms as they are already 12Mb and lowering that I feel they do not run as well.
If you would still like to change the pagepool there are two methods. the easiest is to use pagepool changer however it does not work without modifying the rom first with a hex editor as the pagepool appears to be set in two places and one can override the other.
Now how do we change the pagepool on these new roms?
Method 1 Using Pagepool Editor
Pagepool Edit 2.21 will not work on the rom properly unless a hex key is changed in the rom image file or your rom has been pagepool patched by the chef.
1. In a hex editor search for the following key 03 25 A0 E3 03 15 A0 E3 00 20 83 E5
2. If this key exists, replace 20 83 E5 with 00 A0 E1 This disables this key and lets pagepool changer work.
If the key in step 1 does not exist and you are using a cooked rom then search for the following 03 25 A0 E3 03 15 A0 E3 00 00 A0 E1. If this exists then the pagepool changer should work without any hex edits as the chef has already made the required change.
3. Save the image file.
4. The rom will now not use this key to change the pagepool and it can now be set using the key changed by pagepool editor. If you don't run pagepool editor the rom is now set to 6Mb.
5. Any changes made through the pagepool changer will now be carried out correctly and will reflect in program memory and devhealth.exe
Method 2 Using hex edits only
This method will probably not work if the rom has already had the above key changed by the chef. If this is the case use the pagepool changer.
1. Search for the following Hex key in a hex editor 389FE554089FE5. This is slightly different to the previous key.
2. You need to change the two numbers after it to the numbers shown further down. The first numbers are 02 25
3. Search for the string again (it appears twice). Again change the two numbers after it which will currently be 03 25
4. Save the image file then flash
0Mb Pagepool: 00 25
8Mb Pagepool: 02 25
12Mb Pagepool: 0C 26 (No point using this as it makes no difference already 12Mb)
16Mb Pagepool: 10 26
Method 3 Using ervius's Visual kitchen
Use the following kitchen to change the pagepool:
http://forum.xda-developers.com/showthread.php?t=469420
Thanks ervius
P.S Credits to everybody who found the edits for the previous roms. I wouldn't have found these without them.
This is excellant Thanks
Yes, great job. Definitely an interesting discovery between the 1.93 vs. 2.00 roms.
So ultimately 12pp and higher is the optimal choice?
Gone are the days of the 0 pp with these newer Roms?
band27 said:
Yes, great job. Definitely an interesting discovery between the 1.93 vs. 2.00 roms.
So ultimately 12pp and higher is the optimal choice?
Gone are the days of the 0 pp with these newer Roms?
Click to expand...
Click to collapse
I know 12pp works well however 0Mb seemed to be absolutely fine whilst I tested for a short while. 8Mb seemed a little slow however I was really testing to ensure everything worked properly not for speed so I'll leave that to you guys to test which performs better. Personally I'm more than happy with the default 12mb. In my own cooked rom thread I have 0Mb,8Mb and 12Mb to test so you can make your own choices.
Ok, fair enough...........
& thank u again for unavailing this to us...
Nice work. Also, your ROM is fantastic. Thanks for both.
Are you sure, that it works to 2.03?
mondilv said:
Are you sure, that it works to 2.03?
Click to expand...
Click to collapse
I've not fully tested however I will test tonight. It looked the same when I looked at it and I may have assumed it's the same! My bad.
mondilv said:
Are you sure, that it works to 2.03?
Click to expand...
Click to collapse
Tested both methods with 0Mb Dynamic PP on my 2.03 ROMS and they worked perfect. Devhealth shows a 0mb pp & total program memory is now 119.06 which confirms this has definitely worked.
Try to ulrd & more reduced container.
i have tried method 1 with pagepoolchanger... i have found all the values and changed them but after recook its still the same...
RideTheTube said:
i have tried method 1 with pagepoolchanger... i have found all the values and changed them but after recook its still the same...
Click to expand...
Click to collapse
No idea what went wrong. It works 100% for me everytime. Not that there is any need to change PP on this rom I would say.
cannot find the hex two times
Hi Shaks1979,
I am trying to change the pagepool on a cook ROM (CT's 6.4lite) and both method failed.
I try change the HEX using method 1 and pagepool changer cannot patch the nbh file.
I change using method 2, but i can only locate one HEX value and the two number after this is 03 25 instead of 02 25.
is that this cook ROM file have problem?
thanks.
shaks1979 said:
After having done some research in changing the pagepool on the new 2.0x based diamond roms I have made some interesting discoveries.
The new roms have another 4Mb missing e.g.
2.0x ROMs with 12Mb pagepool have 107.02Mb Program Memory, 8Mb Roms have 111.02Mb, 0Mb Roms have 119.02Mb & if you see 113.02Mb then pagepool is 6Mb
In Comparison
1.93 ROMS with 12Mb pagepool have 111.02Mb Program Memory, 8Mb Roms have 115.02Mb, 0Mb Roms have 123.02Mb & if you see 117.02Mb then pagepool is 6Mb
So this means all new 2.0x Roms are actually set to 12Mb Pagepool and not 16Mb as previously thought.
The best way to definitely be sure what your pagepool is is to run devhealth.exe on your device. This will create the following file on your internal storage mem_1.txt. This file will detail how your physical memory is shared out into pagepool and other areas. 2.00Roms show up in devhealth as having 120.12Mb of total physical memory (including pagepool) and previous roms have 124.12Mb total memory.
Devhealth.exe is available at the following link:
http://forum.xda-developers.com/showthread.php?t=416334
Most of the time there is not much point in editing the pagepool on these new roms as they are already 12Mb and lowering that I feel they do not run as well.
If you would still like to change the pagepool there are two methods. the easiest is to use pagepool changer however it does not work without modifying the rom first with a hex editor as the pagepool appears to be set in two places and one can override the other.
Now how do we change the pagepool on these new roms?
Method 1 Using Pagepool Editor
Pagepool Edit 2.21 will not work on the rom properly unless a hex key is changed in the rom image file.
1. In a hex editor search for the following key 03 25 A0 E3 03 15 A0 E3 00 20 83 E5
2. If this key exists, replace 20 83 E5 with 00 A0 E1 This disables this key and lets pagepool changer work.
If the key in step 1 does not exist and you are using a cooked rom then search for the following 03 25 A0 E3 03 15 A0 E3 00 00 A0 E1. If this exists then the pagepool changer should work without any hex edits as the chef has already made the required change.
3. Save the image file.
4. The rom will now not use this key to change the pagepool and it can now be set using the key changed by pagepool editor. If you don't run pagepool editor the rom is now set to 6Mb.
5. Any changes made through the pagepool changer will now be carried out correctly and will reflect in program memory and devhealth.exe
Method 2 Using hex edits only
This method will probably not work if the rom has already had the above key changed by the chef. If this is the case use the pagepool changer.
1. Search for the following Hex key in a hex editor 389FE554089FE5. This is slightly different to the previous key.
2. You need to change the two numbers after it to the numbers shown further down. The first numbers are 02 25
3. Search for the string again (it appears twice). Again change the two numbers after it which will currently be 03 25
4. Save the image file then flash
0Mb Pagepool: 00 25
8Mb Pagepool: 02 25
12Mb Pagepool: 0C 26 (No point using this as it makes no difference already 12Mb)
16Mb Pagepool: 10 26
P.S Credits to everybody who found the edits for the previous roms. I wouldn't have found these without them.
Click to expand...
Click to collapse
shaks1979 said:
After having done some research in changing the pagepool on the new 2.0x based diamond roms I have made some interesting discoveries.
The new roms have another 4Mb missing e.g.
2.0x ROMs with 12Mb pagepool have 107.02Mb Program Memory, 8Mb Roms have 111.02Mb, 0Mb Roms have 119.02Mb & if you see 113.02Mb then pagepool is 6Mb
In Comparison
1.93 ROMS with 12Mb pagepool have 111.02Mb Program Memory, 8Mb Roms have 115.02Mb, 0Mb Roms have 123.02Mb & if you see 117.02Mb then pagepool is 6Mb
So this means all new 2.0x Roms are actually set to 12Mb Pagepool and not 16Mb as previously thought.
The best way to definitely be sure what your pagepool is is to run devhealth.exe on your device. This will create the following file on your internal storage mem_1.txt. This file will detail how your physical memory is shared out into pagepool and other areas. 2.00Roms show up in devhealth as having 120.12Mb of total physical memory (including pagepool) and previous roms have 124.12Mb total memory.
Devhealth.exe is available at the following link:
http://forum.xda-developers.com/showthread.php?t=416334
Most of the time there is not much point in editing the pagepool on these new roms as they are already 12Mb and lowering that I feel they do not run as well.
If you would still like to change the pagepool there are two methods. the easiest is to use pagepool changer however it does not work without modifying the rom first with a hex editor as the pagepool appears to be set in two places and one can override the other.
Now how do we change the pagepool on these new roms?
Method 1 Using Pagepool Editor
Pagepool Edit 2.21 will not work on the rom properly unless a hex key is changed in the rom image file.
1. In a hex editor search for the following key 03 25 A0 E3 03 15 A0 E3 00 20 83 E5
2. If this key exists, replace 20 83 E5 with 00 A0 E1 This disables this key and lets pagepool changer work.
If the key in step 1 does not exist and you are using a cooked rom then search for the following 03 25 A0 E3 03 15 A0 E3 00 00 A0 E1. If this exists then the pagepool changer should work without any hex edits as the chef has already made the required change.
3. Save the image file.
4. The rom will now not use this key to change the pagepool and it can now be set using the key changed by pagepool editor. If you don't run pagepool editor the rom is now set to 6Mb.
5. Any changes made through the pagepool changer will now be carried out correctly and will reflect in program memory and devhealth.exe
Method 2 Using hex edits only
This method will probably not work if the rom has already had the above key changed by the chef. If this is the case use the pagepool changer.
1. Search for the following Hex key in a hex editor 389FE554089FE5. This is slightly different to the previous key.
2. You need to change the two numbers after it to the numbers shown further down. The first numbers are 02 25
3. Search for the string again (it appears twice). Again change the two numbers after it which will currently be 03 25
4. Save the image file then flash
0Mb Pagepool: 00 25
8Mb Pagepool: 02 25
12Mb Pagepool: 0C 26 (No point using this as it makes no difference already 12Mb)
16Mb Pagepool: 10 26
P.S Credits to everybody who found the edits for the previous roms. I wouldn't have found these without them.
Click to expand...
Click to collapse
you forgot point 3:
my visualkitchen, that changes pp values on xip\ correctly on 2.x roms.....
ervius said:
you forgot point 3:
my visualkitchen, that changes pp values on xip\ correctly on 2.x roms.....
Click to expand...
Click to collapse
Sorry ervius. Don't believe your kitchen was released when I originally wrote this document. Can you send the link and I will add the info on?
shaks1979 said:
Sorry ervius. Don't believe your kitchen was released when I originally wrote this document. Can you send the link and I will add the info on?
Click to expand...
Click to collapse
no problem!!!
I only saw that with xipporterex or all visualkitchen all is done automatically
btw the link for visualkitchen is:
http://forum.xda-developers.com/showthread.php?t=469420
bye bye!!!
ervius said:
no problem!!!
I only saw that with xipporterex or all visualkitchen all is done automatically
btw the link for visualkitchen is:
http://forum.xda-developers.com/showthread.php?t=469420
bye bye!!!
Click to expand...
Click to collapse
Info added to first post
Thanks shaks1979
If I understand disable hex key return PP to 6Mb. But if I search hex 38 9F E5 54 08 9F E5 I have the same 02 25 and 03 25 values that as you list they correspond to 8 PP
So, Let me get this Straight.
0MB Page Pool means that its Dynamic? So Windows Will Pick Whatever Size is Required? Whether it be 6 or 8 or 12mb etc;?
Wouldn't this be the most effective choice?
kang.t7 said:
So, Let me get this Straight.
0MB Page Pool means that its Dynamic? So Windows Will Pick Whatever Size is Required? Whether it be 6 or 8 or 12mb etc;?
Wouldn't this be the most effective choice?
Click to expand...
Click to collapse
That is my understanding of it yes. What you should also know, from my observations is, with PP set at "0", that this eats more memory than its worth.
Not sure if others have experience this as well so I steer clear of the dynamic setup.
Hey guys!
After reading through more pages than I'd like to admit and literally trying for hours I have to open this thread hoping that you guys can manage to figure out, what I did wrong trying to create my own WM6.5-ROM for my Blackstone phone.
I used osKitchen 1.1.3 and tried the following packages:
Topaz - WWE OEM/ROM/XIP/NK, 6.5 kernel
28230 [NEW] All Resolutions (DPIs), All Languages (Locales)
All_New SYS_4 - New Sys 28230/23534
I get an error stating the XPI or OEM versions seem incompatible.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Edit:
Also, how can I test my ROMs without flashing them to my precious phone?
I already got the Windows Mobile SDK 6 and the emulator v3 for win x64 but I don't see how it should work.
What am I doing wrong?
You can't use the stuff for the topaz, the kernel has to be the one for blackstone, same for the xip. Also, the folder structure for the XIP is incorrect, the xip must go in a folder with the number of windows mobile build as name. Try importing a blackstone ROM and starting from that.
So, I experimented some further and using
All_New SYS_1 - New Sys 24001/23519/23102/23096
and
-HTC Touch HD-MR_2009-RUU_BlackStone_HKCSL_WWE_1.57.831.1_Radio_52.64.25.34_1.14.25.24_Ship
I even got to cook my own ROM.
But on flashing my Blackstone phone, after 1%, it stops, stating the ROM is not suitable for the device.
View attachment 285098
What did go wrong?
You want to flash without HardSPL?
I used USLP. Is that wrong?
Why does BlackstonecustomRUU state that my ROM consisting only of files for Blackstone is not suitable for the phone?
Ok now... since finally IoDa00 helped me get the what I hope are the right ingredients (for 24001), I managed to cook my own ROM AND even got to flash it to my Blackstone phone successfully (that is without any error messages).
But now, when the Windows Screen should be visible, there is just nothing and the phone is unresponsive...
What did go wrong?
Same prob here!
Even I am stuck with the same issue. I cooked a ROM for my Artemis using the files from Shady's 28005 original microsoft kitchen. I was able to cook the ROM and it loaded on my phone without any errors. But since the, the phone hangs at first screen.
Any advice!
Here's the log of rom cooked:
======================================
Cleaning up
Starting a new build process. Actual Locale: 0409
Configuring kitchen to USE:
DPI: 96
ResH: 240
ResV: 320
Device name: Artemis
Native 6.5 Kernel: False
Creating folders
Copying OEMs...
Copying XIP...
Copying SYS...
Copying EXT and scanning for all add2* files
Applying ROM tweaks...
OK, Letting Device-specific preparations...
Deleting unneeded resolutions in Titanium
Running Platformrebuilder
----------------------------
Running Tool: Resources\Tools\platformrebuilder.exe
Working Folder: Build
Arguments:
PLATFORMREBUILDER Copyright (c) 2008-2009 bepe Feb 15 2009 22:53:49
Building for old kernel!
Build: Premium
Locale: 0409
Preparing release structure...
... done!
Collecting standard packages and initializing hives...
XIP: 3 packages
IMG: 77 packages
... done!
Processing standard packages...
MSXIPKernelLTK
MSXIPKernel
OEMXipKernel
SIM_TKit
SMIME
AlarmSounds
Bth_A2DP
LockscreenEA
DRM
CHome
MediaOS
RUNTIMES
GPSID
SMS_Providers
ppgprov
OneNote
ConfettiCore
BaseAppsFiles
MediaOSFiles
Riched20
IMPORTANT - NetCF3.7.8345.00
Entertainment
Lockscreen
SQLCE
BronzeEA
OSFiles
SYSTEM_DEFAULT_FONTS
bronze
BROWSINGCORE
CommonEA
Metadata
OS
BROWSING
BaseApps
Webview
Office
browsingie
Shell
PhoneRedist
Redist
Phone
Bluetooth
INTERNETSHARING
Base
Base_Lang_0409
Bluetooth_Lang_0409
browsingie_Lang_0409
BROWSING_Lang_0409
Bth_A2DP_Lang_0409
CHome_Lang_0409
Entertainment_Lang_0409
MediaOS_Lang_0409
Office_Lang_0409
OneNote_Lang_0409
PhoneRedist_Lang_0409
Phone_Lang_0409
Redist_Lang_0409
16 - QuickGPS
16 - QuickGPS_Lang
3 - Comm Manager
3 - Comm Manager_Lang
5 - Ringtone Plugin
5 - Ringtone Plugin_Lang
IMPORTANT - Camera + Driver
IMPORTANT - Camera + Driver_Lang
IMPORTANT - Control Panel
IMPORTANT - Control Panel_Lang
IMPORTANT - DRMMiddleware
IMPORTANT - DShow
IMPORTANT - FILES FROM SYS 28005
IMPORTANT - mHubVO
IMPORTANT - Mortscript
IMPORTANT - OEMAPPS
IMPORTANT - OEMDrivers
IMPORTANT - OEMVersion
IMPORTANT - Optimizations
IMPORTANT - Psshutxp
OEM_Lang_0409
OPTIONAL - WLAN + Driver
OPTIONAL - WLAN + Driver_Lang
... done!
Processing extended packages...
zzz_feROM_version_pkg
Failed to parse value name HKEY_CURRENT_USER\ControlPanel\Sounds!!!
... done!
Finalizing and optimizing ROM structure...
... done!
Memory Map...
SLOT 0: 0x02000000 - 0x01df0000 (END: 0x00060000, 0 MODULES)
0x02000000 - 0x01fd0000 - ROM 0
0x01f90000 - 0x01df0000 - ROM 1
SLOT 1: 0x04000000 - 0x026e0000 (END: 0x02020000, 222 MODULES)
RAM IMAGE: 0x8c100000 - 0x8c3ac460
RAM: 0x8c3ad000 - 0x8c3da000 - Used for kernel modules
0x8c3da000 - 0x8fc00000 - 56 MB free
... done!
Donations...
If you're using one of my tools and feel a need to support development
and/or buy me a drink, please feel free to send a donation through PayPal.
Created link in the root of the kitchen.
Updating all files with add2*:
Inserting XIP
----------------------------
Running Tool: Resources\Tools\implantxip.exe
Working Folder: Build\Temp
Arguments: -xip "xip.bin" -payload "OS.nb.payload" -PP 8 -uldr -imgstart 0 -nocert
Implantxip & Payload Resizer v. 1.1 by ervius!!!
BLOCKS SIZE IS: 00010000
ROM Block is : 00000200
ImgStart MIN Val. SETTED - ULDR Reducing...
....................................
Preparing payload to be resized.....
....................................
Before:
XIP.BIN Target Size: 002AC460 AdjSize: 002AFFFF
ULRD in Payload Start:00000400 Size:0030FBFF End:0030FFFF
XIP in Payload Start:00310000 Size:0030FFFF End:0061FFFF
IMG in Payload Start:00620000 Size:0000FFFF End:0062FFFF
FAT in Payload Start:00000000 Size:00000000 End:00000000
After:
XIP.BIN Target Size: 002AC460 AdjSize: 002AFFFF
ULRD in Payload Start:00000400 Size:0000FBFF End:0000FFFF
XIP in Payload Start:00010000 Size:002AFFFF End:002BFFFF
IMG in Payload Start:002C0000 Size:0000FFFF End:002CFFFF
FAT in Payload Start:00000000 Size:00000000 End:00000000
XIP PagePool Sign not Found!
XIP PagePool from: 4 Mb, to: 8 Mb
XIP Successfully Nocert patched!
Payload resized and XIP Inserted into: OS.nb.payload
Getting IMGFS from NB
----------------------------
Running Tool: Resources\Tools\imgfsfromnb.exe
Working Folder: Build\Temp
Arguments: "os.nb.payload" "imgfs.bin"
ImgfsFromNb 2.1rc2
Sector size is 0x200 bytes
ImgFs partition starts at 0x002c0000 and ends at 0x002d0000
Dumping IMGFS at offset 0x002c0000 (size 0x00010000)
Done!
Generating IMGFS from DUMP
----------------------------
Running Tool: Resources\Tools\imgfsfromdump.exe
Working Folder: Build\Temp
Arguments: "imgfs.bin" "imgfs-new.bin"
ImgfsFromDump 2.1rc2 TURBO with Prefetching and MultiThreading Support, by Plax
Using compression type 'LZX'!
Sector size is 0x200
Total Sectors: 0x119d2
And putting IMGFS into NB
----------------------------
Running Tool: Resources\Tools\ImgfsToNb.exe
Working Folder: Build\Temp
Arguments: "imgfs-new.bin" "OS.nb.payload" "OS-new.nb.payload" -bigstoragemove
ImgfsToNb 2.1rc2
Using bigstorage mode
Sector size is 0x200 bytes
Writing imgfs to offset byte 0x2c0000, sector 0x1600
Padding ImgFs from 0x233a400 bytes (0x119d2 sectors)
to 0x2340000 bytes (0x11a00 sectors)
Not conservative mode. Changing imgfsEnd from 0x2d0000 to 0x2600000
Partition entry before:
File System: 0x25
Start Sector: 0x00001600
Total Sectors: 0x00000080
Boot indicator: 0x00
First Head: 0x00
First Sector: 0x01
First Track: 0x2c
Last Head: 0x7f
Last Sector: 0x01
Last Track: 0x2c
Partition entry after:
File System: 0x25
Start Sector: 0x00001600
Total Sectors: 0x00011a00
Boot indicator: 0x00
First Head: 0x00
First Sector: 0x01
First Track: 0x2c
Last Head: 0x7f
Last Sector: 0x01
Last Track: 0x25f
ImgFs Flash Region log blocks was 0x1, now is 0x234
No Storage Flash Region found!
Done!
Generating NBH
----------------------------
Running Tool: Resources\Tools\htcrt.exe
Working Folder:
Arguments: /buildrom ".\FLASH\ferom.htcrtproj" ".\FLASH\RUU_Signed_0409.nbh"
Cleaning again
DONE! Now you can flash!
======================================
Hey Ondraster... im currently a user of EVK and want to change to OSKitchen... so i did everything to set the SYS as the doctor said... my oem and EXT pkgs too... but i set UPX compression in EXT pkgs... and the log screen turns white like it stops working.. and it says "Doing UPX on EXT directory" and it stucks i leave it almost 30 minues and nothing.... still stucks. I did something wrong?
Thanks for all your pacience man!
Part of my log
Code:
Copied the directory "Sources\EXT\Must_to_Add\AdvancedConfigurationToolv33" to "Build\EXT\Must_to_Add\AdvancedConfigurationToolv33"
Copied the directory "Sources\EXT\Must_to_Add\AdvancedNetwork_1_0_9_0" to "Build\EXT\Must_to_Add\AdvancedNetwork_1_0_9_0"
Copied the directory "Sources\EXT\Must_to_Add\Apps Flashlight" to "Build\EXT\Must_to_Add\Apps Flashlight"
Copied the directory "Sources\EXT\Must_to_Add\Arcsoft_MMS_5_0_31_98R5" to "Build\EXT\Must_to_Add\Arcsoft_MMS_5_0_31_98R5"
Copied the directory "Sources\EXT\Must_to_Add\BOOTLAUNCHER_1_0_35330_2_IOLITE" to "Build\EXT\Must_to_Add\BOOTLAUNCHER_1_0_35330_2_IOLITE"
Copied the directory "Sources\EXT\Must_to_Add\BT_FTP 1_2_29968_1" to "Build\EXT\Must_to_Add\BT_FTP 1_2_29968_1"
Copied the directory "Sources\EXT\Must_to_Add\ConnectionSetupDB" to "Build\EXT\Must_to_Add\ConnectionSetupDB"
Copied the directory "Sources\EXT\Must_to_Add\ConnectionSetup_HTC_ALL" to "Build\EXT\Must_to_Add\ConnectionSetup_HTC_ALL"
Copied the directory "Sources\EXT\Must_to_Add\DeviceUpdate_v2.1.1.1" to "Build\EXT\Must_to_Add\DeviceUpdate_v2.1.1.1"
Copied the directory "Sources\EXT\Must_to_Add\Dopod_SIM_Manager_1436365_by_NiTroGen" to "Build\EXT\Must_to_Add\Dopod_SIM_Manager_1436365_by_NiTroGen"
Copied the directory "Sources\EXT\Must_to_Add\Dummy_FWUPDATE" to "Build\EXT\Must_to_Add\Dummy_FWUPDATE"
Copied the directory "Sources\EXT\Must_to_Add\IP_fix" to "Build\EXT\Must_to_Add\IP_fix"
Copied the directory "Sources\EXT\Must_to_Add\RingtonePlugin_1_00_080624_2" to "Build\EXT\Must_to_Add\RingtonePlugin_1_00_080624_2"
Copied the directory "Sources\EXT\Must_to_Add\USBTOPC" to "Build\EXT\Must_to_Add\USBTOPC"
Copied the directory "Sources\EXT\OLD_OEM\Files" to "Build\EXT\OLD_OEM\Files"
Copied the directory "Sources\EXT\SIP\Keyboard Layout 8525" to "Build\EXT\SIP\Keyboard Layout 8525"
Copied the directory "Sources\EXT\SIP\MSIMAR_96dpi_BlkSkin" to "Build\EXT\SIP\MSIMAR_96dpi_BlkSkin"
Copied the directory "Sources\EXT\SIP\SIPChange" to "Build\EXT\SIP\SIPChange"
Copied the directory "Sources\EXT\Skins_Themes_UI\Z_UI change for 6.5.1" to "Build\EXT\Skins_Themes_UI\Z_UI change for 6.5.1"
Copied the directory "Sources\EXT\Titanium_plugins\lpaso TitaniumCustomizer" to "Build\EXT\Titanium_plugins\lpaso TitaniumCustomizer"
Copied the directory "Sources\EXT\Tweaks\App Buttons" to "Build\EXT\Tweaks\App Buttons"
Copied the directory "Sources\EXT\Tweaks\Performance" to "Build\EXT\Tweaks\Performance"
Copied the directory "Sources\EXT\Tweaks\Registry - Enable Auto TimeZone from Network and icons" to "Build\EXT\Tweaks\Registry - Enable Auto TimeZone from Network and icons"
Copied the directory "Sources\EXT\Tweaks\touchresponse_azharsunny_enchanced sensitivity" to "Build\EXT\Tweaks\touchresponse_azharsunny_enchanced sensitivity"
Copied the directory "Sources\EXT\Tweaks\zz_MODS 651" to "Build\EXT\Tweaks\zz_MODS 651"
Copied the directory "Sources\EXT\Tweaks\z_Icons" to "Build\EXT\Tweaks\z_Icons"
Copied the directory "Sources\EXT\Utilities\xTask" to "Build\EXT\Utilities\xTask"
Doing UPX on EXT directory
Device Hermes, Native 6.5, build 21644.
EDIT:
well i disabled the upx on ext pkgs and now i have platform rebuilder error crash... some XIP or OEM modules incompatibility
BTW as i have native kernel it must say native 6.5 kernel : true... but mine says false :s
am i doing something wrong?
You should not have any others foders in your oemapps, maybe this is the problem. It was in my case
you can not test roms on emulator, only on your phone
sory for my english
somebody helps me Kane159 the define.txt was wrong set...
Nobody xplains how to use it...