Editing CXML Containers
Jump to navigation
Jump to search
The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then publish the changes below to finish undoing the edit.
| Latest revision | Your text | ||
| Line 229: | Line 229: | ||
|- | |- | ||
! Attribute type<BR />(offset 0x4, length 0x4) | ! Attribute type<BR />(offset 0x4, length 0x4) | ||
! 1<BR />(Single Integer) !! 2<BR />(Single Float) !! 3<BR />(String) !! 4<BR />(Integer Array) !! 5<BR />(Float Array) !! 6<BR />(File) !! 7<BR />(ID with loopback ref) !! 8<BR />(ID | ! 1<BR />(Single Integer) !! 2<BR />(Single Float) !! 3<BR />(String) !! 4<BR />(Integer Array) !! 5<BR />(Float Array) !! 6<BR />(File) !! 7<BR />(ID with loopback ref) !! 8<BR />(ID without loopback ref) | ||
|- | |- | ||
! variable 1<BR />(offset 0x8, length 0x4) | ! variable 1<BR />(offset 0x8, length 0x4) | ||
| Integer value || Float value || ''' | | Integer value || Float value || '''Strings Table''' offset || '''Integer Arrays Table''' offset || '''Float Arrays Table''' offset || '''Files Table''' offset || '''Strings Table''' offset || '''Strings Table''' offset | ||
|- | |- | ||
! variable 2<BR />(offset 0xC, length 0x4) | ! variable 2<BR />(offset 0xC, length 0x4) | ||
| Line 240: | Line 240: | ||
*Attribute types 1 and 2 are the only ones that stores the value inside the attribute definition itself (from the point of view of code are defined and given a value at the same time, from the point of view of the structure can be said the value is "embedded" in the cxml TOC). Attributes 1 and 2 doesnt accesses any of the other tables of the cxml structure | *Attribute types 1 and 2 are the only ones that stores the value inside the attribute definition itself (from the point of view of code are defined and given a value at the same time, from the point of view of the structure can be said the value is "embedded" in the cxml TOC). Attributes 1 and 2 doesnt accesses any of the other tables of the cxml structure | ||
*Attribute types 4 and 5 stores an array of values inside an "arrays table". This tables stores multiples arrays in a special format intended to reduce the final size of the table by avoiding duplicated arrays inside it. To recover a single array from a "arrays table" attributes types 4 and 5 specifyes a position in the table by the "offset" (where the array we are trying to recover begins) and the "number" (of consecutive items that composes the array we are trying to recover) <!-- this arrays table access needs a better explain anyway, because is important to know how works the build process that removes duplicated values --> | *Attribute types 4 and 5 stores an array of values inside an "arrays table". This tables stores multiples arrays in a special format intended to reduce the final size of the table by avoiding duplicated arrays inside it. To recover a single array from a "arrays table" attributes types 4 and 5 specifyes a position in the table by the "offset" (where the array we are trying to recover begins) and the "number" (of consecutive items that composes the array we are trying to recover) <!-- this arrays table access needs a better explain anyway, because is important to know how works the build process that removes duplicated values --> | ||
*Attribute type 7 stores a string in the "strings table", in this table next to the string we are trying to recover is defined an offset that "loops back" to the element that owns the attribute. | *Attribute type 7 stores a string in the same table than attribute type 3 (a "strings table"), in this table next to the string we are trying to recover is defined an offset that "loops back" to the element that owns the attribute. | ||
**Example: in [http://www.psdevwiki.com/ps3/CXML_Containers#QRC_2 rhm.qrc v1.00] the first "file" element of the TOC (at offset 0x78) has 2 attributes: "src" (at offset 0x94) is a type 6 attribute, and "id" (at offset 0xA4) is a type 7 attribute, "src" points to the "file table" and defines the file by his offset and size (this way you know where is the file, no mistery with attribute type 6). And "id" points to the string table at offset 0x1B0. If you look in the string table, the first string has an offset that points back to the "file" element in the TOC at offset 0x78 (in the runtime we have readed the definition of the element "file" in the TOC----> then the attribute id with the string---> then back to the TOC to the same "file" element so we can read all his attributes again). | **Example: in [http://www.psdevwiki.com/ps3/CXML_Containers#QRC_2 rhm.qrc v1.00] the first "file" element of the TOC (at offset 0x78) has 2 attributes: "src" (at offset 0x94) is a type 6 attribute, and "id" (at offset 0xA4) is a type 7 attribute, "src" points to the "file table" and defines the file by his offset and size (this way you know where is the file, no mistery with attribute type 6). And "id" points to the string table at offset 0x1B0. If you look in the string table, the first string has an offset that points back to the "file" element in the TOC at offset 0x78 (in the runtime we have readed the definition of the element "file" in the TOC----> then the attribute id with the string---> then back to the TOC to the same "file" element so we can read all his attributes again). | ||
**The purpose of this pointer that returns back from the strings table ---> to the TOC is a bit unknown, seems to be related with the other attributes owned by the same element and the way how the structure is accessed in the runtime (maybe the system does a loop to access every one of the attributes that belongs to an element). This special feature is what has been named in the wiki page before as "ID with ref" (and "ID without ref" for att type 8 | **The purpose of this pointer that returns back from the strings table ---> to the TOC is a bit unknown, seems to be related with the other attributes owned by the same element and the way how the structure is accessed in the runtime (maybe the system does a loop to access every one of the attributes that belongs to an element). This special feature is what has been named in the wiki page before as "ID with ref" (and "ID without ref" for att type 8 because doesnt have it), now we are using too the concept of loops or loopback but in my oppinion the names of attibute types 7 and 8 cant be decided yet because we dont know exactly what is the purpose of it (is hard to give it a name before knowing exactly how it works) | ||
==Container Common Structure== | ==Container Common Structure== | ||
| Line 425: | Line 425: | ||
*For QRCF_2 and P3T themes only is stored the "id" (the file "src" is missing so there is no way to read the original path or file extension from the container) | *For QRCF_2 and P3T themes only is stored the "id" (the file "src" is missing so there is no way to read the original path or file extension from the container) | ||
{{Boxcode| | {{Boxcode|content=<syntaxhighlight lang="xml"> | ||
<?xml version="1.0" encoding="UTF-8"?> | <?xml version="1.0" encoding="UTF-8"?> | ||
<qrc> | <qrc> | ||
| Line 1,119: | Line 1,119: | ||
*Example of the string table from frame_01.cxml | *Example of the string table from frame_01.cxml | ||
{{Boxcode| | {{Boxcode|content=<syntaxhighlight lang="xml"> | ||
root.version.1.00.jacket.id.normal.base.position.rotation.scale.name.frame_01.filetable.num.file.src.frame_01_lo.gim.lod.frame_01_parallax_512.gtf.shadow_square_1.gtf.shadow_square_2.gtf.shadow_3.gtf.frame_01_mi.gim.frame_01_hi.gim.vptable.vp.vp_jacket_ezprim.vpo.optional.vp_jacket_normalmap.vpo.vp_jacket_ezshadow.vpo.fptable.fp.fp_jacket_ezprim_rgb.fpo.fp_jacket_normalmap_specL.fpo.fp_jacket_ezshadow.fpo.modeltable.model.color.uv.vertex.tangent.binormal.layertable.layer.priority.type.image-frame-static.order.duration.image-thumbnail-auto | root.version.1.00.jacket.id.normal.base.position.rotation.scale.name.frame_01.filetable.num.file.src.frame_01_lo.gim.lod.frame_01_parallax_512.gtf.shadow_square_1.gtf.shadow_square_2.gtf.shadow_3.gtf.frame_01_mi.gim.frame_01_hi.gim.vptable.vp.vp_jacket_ezprim.vpo.optional.vp_jacket_normalmap.vpo.vp_jacket_ezshadow.vpo.fptable.fp.fp_jacket_ezprim_rgb.fpo.fp_jacket_normalmap_specL.fpo.fp_jacket_ezshadow.fpo.modeltable.model.color.uv.vertex.tangent.binormal.layertable.layer.priority.type.image-frame-static.order.duration.image-thumbnail-auto | ||
</syntaxhighlight>}} | </syntaxhighlight>}} | ||
| Line 1,140: | Line 1,140: | ||
In other words... the only difference between QRCF_1 and QRCF_2 is the added attribute '''size''' in every file entry that makes entries 16 bytes longer in QRCF_2... there is no need for an example of QRCF_2 because are almost the same | In other words... the only difference between QRCF_1 and QRCF_2 is the added attribute '''size''' in every file entry that makes entries 16 bytes longer in QRCF_2... there is no need for an example of QRCF_2 because are almost the same | ||
This .xml represents structure of rhm.qrc 1.00, can be used to build it, and can be generated by reading the "TOC" | |||
{{Boxcode|content=<syntaxhighlight lang="xml"> | |||
{{Boxcode| | |||
<?xml version="1.0" encoding="UTF-8"?> | <?xml version="1.0" encoding="UTF-8"?> | ||
<qrc> | <qrc> | ||
<file-table> | <file-table> | ||
<file src=" | <file src="lib/rhm/Clear.fpo" id="lib/rhm/Clear.fpo"/> | ||
<file src=" | <file src="lib/rhm/Clear.vpo" id="lib/rhm/Clear.vpo"/> | ||
<file src=" | <file src="lib/rhm/Copy.fpo" id="lib/rhm/Copy.fpo"/> | ||
<file src=" | <file src="lib/rhm/default.fpo" id="lib/rhm/default.fpo"/> | ||
<file src=" | <file src="lib/rhm/default.vpo" id="lib/rhm/default.vpo"/> | ||
</file-table> | </file-table> | ||
</qrc> | </qrc> | ||
</syntaxhighlight>}} | </syntaxhighlight>}} | ||
When the | When the qrc container is build all the info from he .xml structure is stored in the '''TOC''' (at offset 0x40) after the CXML header, and the files are stored in a "Files table" placed at bottom of the container (at offset 0x250) | ||
The | The '''TOC''' (colored in red tones in the example) starts with the elements: '''qrc''', and his only children '''file-table''' that has 5 '''file''' childrens (all them are elements). Every '''file''' element has 2 attributes '''src''' and '''id'''. | ||
<div style="height:450px; overflow:auto"> | |||
{| class="wikitable" style="font-size:x-small;" | {| class="wikitable" style="font-size:x-small;" | ||
|+[[rhm.qrc]] (v1.00) | |+[[rhm.qrc]] (v1.00) | ||
| Line 1,176: | Line 1,174: | ||
| 0x0C || 0x04 || 00000164 || TOC size | | 0x0C || 0x04 || 00000164 || TOC size | ||
|- bgcolor="#88ff88" | |- bgcolor="#88ff88" | ||
| 0x10 || 0x04 || 000001B0 || Strings table absolute offset || rowspan="2" | Stores the '''VALUES''' of XML attributes type 7 and 8 | | 0x10 || 0x04 || 000001B0 || Strings table absolute offset || rowspan="2" | Stores the '''VALUES''' of XML attributes type 3 ('''string'''), and 7<!-- path/file.ext literal -->, and 8 ? <!-- i never found an 8--> | ||
|- bgcolor="#88ff88" | |- bgcolor="#88ff88" | ||
| 0x14 || 0x04 || 00000071 || Strings table size | | 0x14 || 0x04 || 00000071 || Strings table size | ||
|- bgcolor="#8888ff" | |- bgcolor="#8888ff" | ||
| 0x18 || 0x04 || 00000230 || TAGS table absolute offset || rowspan="2" | Stores the '''NAMES''' of XML [http://www.w3schools.com/xml/xml_syntax.asp '''elements''' and '''attributes''']] | | 0x18 || 0x04 || 00000230 || TAGS table absolute offset || rowspan="2" | Stores the '''NAMES''' of XML [http://www.w3schools.com/xml/xml_syntax.asp '''elements''' and '''attributes''']] | ||
|- bgcolor="#8888ff" | |- bgcolor="#8888ff" | ||
| 0x1C || 0x04 || 0000001B || TAGS table size | | 0x1C || 0x04 || 0000001B || TAGS table size | ||
| Line 1,187: | Line 1,185: | ||
|- bgcolor="#ffeebb" | |- bgcolor="#ffeebb" | ||
| 0x24 || 0x04 || 00000000 || Integer arrays table size | | 0x24 || 0x04 || 00000000 || Integer arrays table size | ||
|- bgcolor="# | |- bgcolor="#ffee99" | ||
| 0x28 || 0x04 || 00000250 || Float arrays table absolute offset || rowspan="2" | Stores the '''VALUES''' of XML attributes type 5 ('''float array''') | | 0x28 || 0x04 || 00000250 || Float arrays table absolute offset || rowspan="2" | Stores the '''VALUES''' of XML attributes type 5 ('''float array''') | ||
|- bgcolor="# | |- bgcolor="#ffee99" | ||
| 0x2C || 0x04 || 00000000 || Float arrays table size | | 0x2C || 0x04 || 00000000 || Float arrays table size | ||
|- bgcolor="#ffaa00" | |- bgcolor="#ffaa00" | ||
| Line 1,195: | Line 1,193: | ||
|- bgcolor="#ffaa00" | |- bgcolor="#ffaa00" | ||
| 0x34 || 0x04 || 00000DD0 || Files table size | | 0x34 || 0x04 || 00000DD0 || Files table size | ||
|- | |- {{cellcolors|lightgrey}} | ||
| 0x38 || 0x04 || 00000000 || | | 0x38 || 0x04 || 00000000 || Reserved area absolute offset || rowspan="2" | | ||
|- | |- {{cellcolors|lightgrey}} | ||
| 0x3C || 0x04 || 00000000 || | | 0x3C || 0x04 || 00000000 || Reserved area size | ||
|- | |- | ||
! rowspan= | ! rowspan=95 | TOC | ||
|- {{cellcolors|#ff7777}} | |- {{cellcolors|#ff7777}} | ||
| 0x40 || 0x04 || 00000000 || element name relative offset || '''qrc''' | | 0x40 || 0x04 || 00000000 || element name relative offset || '''qrc''' | ||
| Line 1,217: | Line 1,213: | ||
|- {{cellcolors|#ff7777}} | |- {{cellcolors|#ff7777}} | ||
| 0x58 || 0x04 || 0000001C || last child element relative offset || 0x40 + 0x1C = '''last child at 0x5C absolute''' | | 0x58 || 0x04 || 0000001C || last child element relative offset || 0x40 + 0x1C = '''last child at 0x5C absolute''' | ||
|- {{cellcolors|#ff8888}} | |- {{cellcolors|#ff8888}} | ||
| 0x5C || 0x04 || 00000004 || element name relative offset || '''file-table''' | | 0x5C || 0x04 || 00000004 || element name relative offset || '''file-table''' | ||
| Line 1,394: | Line 1,388: | ||
| 0x1A0 || 0x04 || 00000000 || attribute variable 2 || | | 0x1A0 || 0x04 || 00000000 || attribute variable 2 || | ||
|- {{cellcolors|lightgrey}} | |- {{cellcolors|lightgrey}} | ||
! rowspan=1 | Padding | |||
| 0x1A4 || 0x0C || 000000000000000000000000 || padding || aligned to a multiply of 0x10 | | 0x1A4 || 0x0C || 000000000000000000000000 || padding || aligned to a multiply of 0x10 | ||
|- | |- | ||
! rowspan= | ! rowspan=11 | Strings Table | ||
|- bgcolor="#88ff88" | |- bgcolor="#88ff88" | ||
| 0x1B0 || 0x04 || 00000038 || 1st element relative offset || 0x40 + 0x38 = '''element at offset 0x78 absolute''' | | 0x1B0 || 0x04 || 00000038 || 1st element relative offset || 0x40 + 0x38 = '''element at offset 0x78 absolute''' | ||
| Line 1,419: | Line 1,414: | ||
|- bgcolor="#cccccc" | |- bgcolor="#cccccc" | ||
|- {{cellcolors|lightgrey}} | |- {{cellcolors|lightgrey}} | ||
! rowspan=1 | Padding | |||
| 0x221 || 0xF || 00000000000000000000000000000000 || padding || aligned to a multiply of 0x10 | | 0x221 || 0xF || 00000000000000000000000000000000 || padding || aligned to a multiply of 0x10 | ||
|- | |- | ||
! rowspan= | ! rowspan=2 | TAGS table | ||
|- bgcolor="#8888ff" | |- bgcolor="#8888ff" | ||
| 0x230 || 0x1B || qrc file-table file src id || Container identifyer (first tag) + other tags from MAKE.XML | | | 0x230 || 0x1B || qrc file-table file src id || Container identifyer (first tag) + other tags from MAKE.XML || <syntaxhighlight lang="xml"> <qrc> <file-table> <file src= id=/></syntaxhighlight> | ||
|- bgcolor="#cccccc" | |- bgcolor="#cccccc" | ||
|- {{cellcolors|lightgrey}} | |- {{cellcolors|lightgrey}} | ||
! rowspan=1 | Padding | |||
| 0x24B || 0x5 || 0000000000 || padding || aligned to a multiply of 0x10 | | 0x24B || 0x5 || 0000000000 || padding || aligned to a multiply of 0x10 | ||
|- | |- | ||
! rowspan="6" | Files Table | ! rowspan="6" | Files Table | ||
| Line 1,447: | Line 1,436: | ||
|- bgcolor="#ffaa00" | |- bgcolor="#ffaa00" | ||
| 0x790 || 0x890 || 00001B5B... || 5th file || | | 0x790 || 0x890 || 00001B5B... || 5th file || | ||
|- | |- | ||
|} | |} | ||
</div> | |||
'''rhm.qrc (from firmware 1.00)''' | '''rhm.qrc (from firmware 1.00)''' | ||
< | <div style="height:450px; overflow:auto"> | ||
Offset(h) 00 04 08 0C | Offset(h) 00 04 08 0C | ||
| Line 1,714: | Line 1,702: | ||
00001000 401F9C6C 01D0100D 8086C0C3 60409F80 @.œl.Ð..€†ÀÃ`@Ÿ€ | 00001000 401F9C6C 01D0100D 8086C0C3 60409F80 @.œl.Ð..€†ÀÃ`@Ÿ€ | ||
00001010 401F9C6C 01D0000D 8086C0C3 60411F81 @.œl.Ð..€†ÀÃ`A.. | 00001010 401F9C6C 01D0000D 8086C0C3 60411F81 @.œl.Ð..€†ÀÃ`A.. | ||
</ | </div> | ||
==P3T== | ==P3T== | ||
| Line 1,950: | Line 1,938: | ||
*Attribute 8 means there is another element in the xml (a different element than the one who owns the attribute) with the same name (so the loopback of his string points to a different element). In this case coldboot.raf uses the name '''mtrl_logo''' 2 times (but is only stored one time in the string table): | *Attribute 8 means there is another element in the xml (a different element than the one who owns the attribute) with the same name (so the loopback of his string points to a different element). In this case coldboot.raf uses the name '''mtrl_logo''' 2 times (but is only stored one time in the string table): | ||
{{Boxcode| | {{Boxcode|content=<syntaxhighlight lang="xml"> | ||
<raf> <!-- this is a reduced version of coldboot.raf.cxml --> | <raf> <!-- this is a reduced version of coldboot.raf.cxml --> | ||
<scene> | <scene> | ||
| Line 1,964: | Line 1,952: | ||
==CXML decompiler== | ==CXML decompiler== | ||
Experimental | Experimental tool coded by flatz for documenting purposes, three versions of the tool was released publically in a IRC channel the same day (first and second versions had bugs which were identified and fixed) | ||
The tool | The tool can extract the files contained inside a CXML container (CXML, QRCF, P3TF, RAFO, etc...), and generates an .xml that represents the original CXML structure | ||
*Usage: | *Usage: | ||
** | **Note the original files [[lines.qrc]] and [[coldboot.raf]] are compressed with zlib (a compression layer that affects the whole file and makes it imposible to read the cxml structure). The command line examples below considers the files was decompressed in a previous step (using a [[Qt_Resource_Container_%28QRC%29#ZLIB_archivers|zlib archiver]]), in this previous step the file-extension was renamed to "qrcf" and "rafo" (which are the decompressed versions where the cxml structure is fully readable and the tool can process it) | ||
{{Keyboard|content= | {{Keyboard|content= | ||
{{Keyboard|content= | '''C:\Portables\cxml decompiler v3 alpha>decompiler.exe lines.qrcf lines.xml''' | ||
}} | |||
{{Keyboard|content= | |||
'''C:\Portables\cxml decompiler v3 alpha>decompiler.exe coldboot.rafo coldboot.xml''' | |||
}} | |||
*CXML decompiler v3 alpha, download links: http://multiupload.biz/dnsipf0dkssz/cxml_decompiler_v3_alpha_MultiUpload.biz.7z.html | |||
'''Changelog''' | '''Changelog''' | ||
'''---------''' | '''---------''' | ||
| Line 1,981: | Line 1,973: | ||
v3 alpha - Fixed offset/length displacements in the function that locates float values (xml tag attributes related with x,y,z axis in 3D space now looks correct) | v3 alpha - Fixed offset/length displacements in the function that locates float values (xml tag attributes related with x,y,z axis in 3D space now looks correct) | ||
v4 alpha - Floats precission increased | v4 alpha - Floats precission increased | ||
{{Boxcode| | *CXML decompiler v4 alpha, source code: | ||
<div style="height:600px; overflow:auto"> | |||
{{Boxcode|content=<syntaxhighlight lang="python"> | |||
#!python2 | #!python2 | ||
| Line 2,258: | Line 2,249: | ||
write_raw(f, '\"{0}\"'.format(attribute.get_int())) | write_raw(f, '\"{0}\"'.format(attribute.get_int())) | ||
elif attribute.type == Attribute.TYPE_FLOAT: | elif attribute.type == Attribute.TYPE_FLOAT: | ||
write_raw(f, '\"{0 | write_raw(f, '\"{0}\"'.format(attribute.get_float())) | ||
elif attribute.type == Attribute.TYPE_STRING: | elif attribute.type == Attribute.TYPE_STRING: | ||
write_raw(f, '\"{0}\"'.format(attribute.get_string())) | write_raw(f, '\"{0}\"'.format(attribute.get_string())) | ||
| Line 2,266: | Line 2,257: | ||
array_length = len(array) | array_length = len(array) | ||
for j in range(array_length): | for j in range(array_length): | ||
write_raw(f, '{0 | write_raw(f, '{0}'.format(array[j])) | ||
if j + 1 < array_length: | if j + 1 < array_length: | ||
write_raw(f, ',') | write_raw(f, ',') | ||
| Line 2,280: | Line 2,271: | ||
write_raw(f, '\"') | write_raw(f, '\"') | ||
elif attribute.type == Attribute.TYPE_FILE: | elif attribute.type == Attribute.TYPE_FILE: | ||
file_name = ' | file_name = '{0}_0x{1:08X}.bin'.format(self.document.file_prefix, attribute.offset) | ||
file_data = attribute.get_file() | file_data = attribute.get_file() | ||
with open(file_name, 'wb') as of: | with open(file_name, 'wb') as of: | ||
| Line 2,445: | Line 2,436: | ||
document.dump(f) | document.dump(f) | ||
</syntaxhighlight>}} | </syntaxhighlight>}} | ||
</div> | |||
*Problems/bugs | |||
<strike>In v3 floats conversion from hex is not correct (is rounded up, causing an innacuracy). Example: | |||
In coldboot.raf the "rotation" value for the "actors" stored in hexadecimal is 3FC90FD8, this value is converted/extracted by "cxml decompiler v3" tool as: 1.5708 (the value has been rounded up) | |||
The accurate conversion should be 1.570796 (no rounded up, 6 decimals precission)</strike> Fixed in v4 | |||
It's easy to fix that, just replace "{0:3.6}" (without quotes) with "{0}". | |||
--[[User:Flatz|Flatz]] ([[User talk:Flatz|talk]]) 07:47, 16 November 2014 (EST) | |||
{{File Formats}} | {{File Formats}} | ||
<noinclude>[[Category:Main]]</noinclude> | <noinclude>[[Category:Main]]</noinclude> | ||