Editing CXML Containers

Jump to navigation Jump to search
Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then publish the changes below to finish undoing the edit.

Latest revision Your text
Line 229: Line 229:
|-
|-
! Attribute type<BR />(offset 0x4, length 0x4)
! Attribute type<BR />(offset 0x4, length 0x4)
! 1<BR />(Single Integer) !! 2<BR />(Single Float) !! 3<BR />(String) !! 4<BR />(Integer Array) !! 5<BR />(Float Array) !! 6<BR />(File) !! 7<BR />(ID with loopback ref) !! 8<BR />(ID with crossed loopback ref)
! 1<BR />(Single Integer) !! 2<BR />(Single Float) !! 3<BR />(String) !! 4<BR />(Integer Array) !! 5<BR />(Float Array) !! 6<BR />(File) !! 7<BR />(ID with loopback ref) !! 8<BR />(ID without loopback ref)
|-
|-
! variable 1<BR />(offset 0x8, length 0x4)
! variable 1<BR />(offset 0x8, length 0x4)
| Integer value || Float value || '''TAGS Table''' offset || '''Integer Arrays Table''' offset || '''Float Arrays Table''' offset || '''Files Table''' offset || '''Strings Table''' offset || '''Strings Table''' offset
| Integer value || Float value || '''Strings Table''' offset || '''Integer Arrays Table''' offset || '''Float Arrays Table''' offset || '''Files Table''' offset || '''Strings Table''' offset || '''Strings Table''' offset
|-
|-
! variable 2<BR />(offset 0xC, length 0x4)
! variable 2<BR />(offset 0xC, length 0x4)
Line 240: Line 240:
*Attribute types 1 and 2 are the only ones that stores the value inside the attribute definition itself (from the point of view of code are defined and given a value at the same time, from the point of view of the structure can be said the value is "embedded" in the cxml TOC). Attributes 1 and 2 doesnt accesses any of the other tables of the cxml structure
*Attribute types 1 and 2 are the only ones that stores the value inside the attribute definition itself (from the point of view of code are defined and given a value at the same time, from the point of view of the structure can be said the value is "embedded" in the cxml TOC). Attributes 1 and 2 doesnt accesses any of the other tables of the cxml structure
*Attribute types 4 and 5 stores an array of values inside an "arrays table". This tables stores multiples arrays in a special format intended to reduce the final size of the table by avoiding duplicated arrays inside it. To recover a single array from a "arrays table" attributes types 4 and 5 specifyes a position in the table by the "offset" (where the array we are trying to recover begins) and the "number" (of consecutive items that composes the array we are trying to recover) <!-- this arrays table access needs a better explain anyway, because is important to know how works the build process that removes duplicated values -->
*Attribute types 4 and 5 stores an array of values inside an "arrays table". This tables stores multiples arrays in a special format intended to reduce the final size of the table by avoiding duplicated arrays inside it. To recover a single array from a "arrays table" attributes types 4 and 5 specifyes a position in the table by the "offset" (where the array we are trying to recover begins) and the "number" (of consecutive items that composes the array we are trying to recover) <!-- this arrays table access needs a better explain anyway, because is important to know how works the build process that removes duplicated values -->
*Attribute type 7 stores a string in the "strings table", in this table next to the string we are trying to recover is defined an offset that "loops back" to the element that owns the attribute.
*Attribute type 7 stores a string in the same table than attribute type 3 (a "strings table"), in this table next to the string we are trying to recover is defined an offset that "loops back" to the element that owns the attribute.
**Example: in [http://www.psdevwiki.com/ps3/CXML_Containers#QRC_2 rhm.qrc v1.00] the first "file" element of the TOC (at offset 0x78) has 2 attributes: "src" (at offset 0x94) is a type 6 attribute, and "id" (at offset 0xA4) is a type 7 attribute, "src" points to the "file table" and defines the file by his offset and size (this way you know where is the file, no mistery with attribute type 6). And "id" points to the string table at offset 0x1B0. If you look in the string table, the first string has an offset that points back to the "file" element in the TOC at offset 0x78 (in the runtime we have readed the definition of the element "file" in the TOC----> then the attribute id with the string---> then back to the TOC to the same "file" element so we can read all his attributes again).
**Example: in [http://www.psdevwiki.com/ps3/CXML_Containers#QRC_2 rhm.qrc v1.00] the first "file" element of the TOC (at offset 0x78) has 2 attributes: "src" (at offset 0x94) is a type 6 attribute, and "id" (at offset 0xA4) is a type 7 attribute, "src" points to the "file table" and defines the file by his offset and size (this way you know where is the file, no mistery with attribute type 6). And "id" points to the string table at offset 0x1B0. If you look in the string table, the first string has an offset that points back to the "file" element in the TOC at offset 0x78 (in the runtime we have readed the definition of the element "file" in the TOC----> then the attribute id with the string---> then back to the TOC to the same "file" element so we can read all his attributes again).
**The purpose of this pointer that returns back from the strings table ---> to the TOC is a bit unknown, seems to be related with the other attributes owned by the same element and the way how the structure is accessed in the runtime (maybe the system does a loop to access every one of the attributes that belongs to an element). This special feature is what has been named in the wiki page before as "ID with ref" (and "ID without ref" for att type 8 <strike>because doesnt have it</strike>), now we are using too the concept of loops, loopback, and crossed loopbacks, but in my oppinion the names of attibute types 7 and 8 cant be decided yet because we dont know exactly what is the purpose of it (is hard to give it a name before knowing exactly how it works)
**The purpose of this pointer that returns back from the strings table ---> to the TOC is a bit unknown, seems to be related with the other "brother" attributes and the way how the structure is accessed in the runtime (maybe the system does a loop to access every one of the attributes that belongs to an element). This special feature is what has been named in the wiki page before as "ID with ref" (and "ID without ref" for att type 8 because doesnt have it), now we are using too the concept of loops or loopback but in my oppinion the names of attibute types 7 and 8 cant be decided yet because we dont know exactly what is the purpose of it (is hard to give it a name before knowing exactly how it works)


==Container Common Structure==
==Container Common Structure==
Line 370: Line 370:
| 0x30 || 0x04 || 00 00 00 38 || First Entry
| 0x30 || 0x04 || 00 00 00 38 || First Entry
|-
|-
| 0x34 || 0x04 || 00 00 01 28 || Last Entry
| 0x34 || 0x04 || xx xx xx xx || Last Entry
|-
|-
| 0x38 || 0x04 || 00 00 00 0F || String Pointer (Offset within '''String Table''', in this case it will be a Element <file>)  
| 0x38 || 0x04 || 00 00 00 0F || String Pointer (Offset within '''String Table''', in this case it will be a Element <file>)  
Line 400: Line 400:
| 0x6C || 0x04 || 00 00 00 00 || Offset of filename within the ID Table
| 0x6C || 0x04 || 00 00 00 00 || Offset of filename within the ID Table
|-
|-
| 0x70 || 0x04 || 00 00 00 00 || not used
| 0x70 || 0x04 || 00 00 00 00 || on icontex here are additional 16 bytes
|-
| 0x74 || 0x04 || 00 00 00 0F || String Pointer
|-
| 0x78 || 0x04 || 00 00 00 02 || Attribute Counter
|-
| 0x7C || 0x04 || 00 00 00 1C || Parent Element
|-
| 0x80 || 0x04 || 00 00 00 38 || Previous Element
|-
| 0x84 || 0x04 || 00 00 00 B0 || Next Element
|-
|-
|}
|}


On icontex.qrc there are additional 16 bytes added on offset 0x70 within the '''Tree Table''' because it has an additional attribute '''size'''
On icontex.qrc there are additional 16 bytes added on offset 0x70 within the '''Tree Table'''
{| class="wikitable"
{| class="wikitable"
! Offset !! Size !! Example !! Remark
! Offset !! Size !! Example !! Remark
|-
| 0x70 || 0x04 || 00 00 00 00 || on icontex.qrc
|-
|-
| 0x74 || 0x04 || 00 00 00 1B || String Pointer (Offset within '''String Table''', in this case it will be a Attribute <size>)
| 0x74 || 0x04 || 00 00 00 1B || String Pointer (Offset within '''String Table''', in this case it will be a Attribute <size>)
Line 414: Line 426:
| 0x7C || 0x04 || 00 01 55 D4 || on icontex.qrc (size of .dds uncompressed [Dec = 87508])
| 0x7C || 0x04 || 00 01 55 D4 || on icontex.qrc (size of .dds uncompressed [Dec = 87508])
|-
|-
| 0x80 || 0x04 || 00 00 00 00 || not used
| 0x80 || 0x04 || 00 00 00 00 || similar all qrc files
|-
| 0x84 || 0x04 || 00 00 00 0F || String Pointer
|-
| 0x88 || 0x04 || 00 00 00 02 || Attribute Count
|-
| 0x8C || 0x04 || 00 00 00 1C || Parent Element
|-
| 0x90 || 0x04 || 00 00 00 38 || Previous Element
|-
| 0x94 || 0x04 || 00 00 00 D0 || Next Element
|-
|-
|}
|}
Line 425: Line 447:
*For QRCF_2 and P3T themes only is stored the "id" (the file "src" is missing so there is no way to read the original path or file extension from the container)
*For QRCF_2 and P3T themes only is stored the "id" (the file "src" is missing so there is no way to read the original path or file extension from the container)


{{Boxcode|code=<syntaxhighlight lang="xml">
{{Boxcode|content=<syntaxhighlight lang="xml">
<?xml version="1.0" encoding="UTF-8"?>
<?xml version="1.0" encoding="UTF-8"?>
<qrc>
<qrc>
Line 1,119: Line 1,141:
*Example of the string table from frame_01.cxml
*Example of the string table from frame_01.cxml


{{Boxcode|code=<syntaxhighlight lang="xml">
{{Boxcode|content=<syntaxhighlight lang="xml">
root.version.1.00.jacket.id.normal.base.position.rotation.scale.name.frame_01.filetable.num.file.src.frame_01_lo.gim.lod.frame_01_parallax_512.gtf.shadow_square_1.gtf.shadow_square_2.gtf.shadow_3.gtf.frame_01_mi.gim.frame_01_hi.gim.vptable.vp.vp_jacket_ezprim.vpo.optional.vp_jacket_normalmap.vpo.vp_jacket_ezshadow.vpo.fptable.fp.fp_jacket_ezprim_rgb.fpo.fp_jacket_normalmap_specL.fpo.fp_jacket_ezshadow.fpo.modeltable.model.color.uv.vertex.tangent.binormal.layertable.layer.priority.type.image-frame-static.order.duration.image-thumbnail-auto
root.version.1.00.jacket.id.normal.base.position.rotation.scale.name.frame_01.filetable.num.file.src.frame_01_lo.gim.lod.frame_01_parallax_512.gtf.shadow_square_1.gtf.shadow_square_2.gtf.shadow_3.gtf.frame_01_mi.gim.frame_01_hi.gim.vptable.vp.vp_jacket_ezprim.vpo.optional.vp_jacket_normalmap.vpo.vp_jacket_ezshadow.vpo.fptable.fp.fp_jacket_ezprim_rgb.fpo.fp_jacket_normalmap_specL.fpo.fp_jacket_ezshadow.fpo.modeltable.model.color.uv.vertex.tangent.binormal.layertable.layer.priority.type.image-frame-static.order.duration.image-thumbnail-auto
</syntaxhighlight>}}
</syntaxhighlight>}}
Line 1,140: Line 1,162:
In other words... the only difference between QRCF_1 and QRCF_2 is the added attribute '''size''' in every file entry that makes entries 16 bytes longer in QRCF_2... there is no need for an example of QRCF_2 because are almost the same
In other words... the only difference between QRCF_1 and QRCF_2 is the added attribute '''size''' in every file entry that makes entries 16 bytes longer in QRCF_2... there is no need for an example of QRCF_2 because are almost the same


This .xml represents structure of rhm.qrc 1.00, can be used to build it, and can be generated by reading the "TOC"


 
{{Boxcode|content=<syntaxhighlight lang="xml">
The .xml below represents the structure of [[rhm.qrc]] used in firmwares from 1.00 up to 1.32 ready to be compiled (compiled hash should be MD5: D1E7281246EC8A46EB6D368BD4B332BD)
{{Boxcode|code=<syntaxhighlight lang="xml">
<?xml version="1.0" encoding="UTF-8"?>
<?xml version="1.0" encoding="UTF-8"?>
<qrc>
<qrc>
<file-table>
<file-table>
<file src="C:/workdir/Clear_old.fpo" id="lib/rhm/Clear.fpo"/>
<file src="lib/rhm/Clear.fpo" id="lib/rhm/Clear.fpo"/>
<file src="C:/workdir/Clear_test12.vpo" id="lib/rhm/Clear.vpo"/>
<file src="lib/rhm/Clear.vpo" id="lib/rhm/Clear.vpo"/>
<file src="X:/DontCopyMe.fap" id="lib/rhm/Copy.fpo"/>
<file src="lib/rhm/Copy.fpo" id="lib/rhm/Copy.fpo"/>
<file src="D:/anotherworkdir/default_v12.fpo" id="lib/rhm/default.fpo"/>
<file src="lib/rhm/default.fpo" id="lib/rhm/default.fpo"/>
<file src="D:/anotherworkdir/default_v34.vpo" id="lib/rhm/default.vpo"/>
<file src="lib/rhm/default.vpo" id="lib/rhm/default.vpo"/>
</file-table>
</file-table>
</qrc>
</qrc>
</syntaxhighlight>}}
</syntaxhighlight>}}


When the file is compiled the info from the .xml is stored in the '''TOC''' (at offset 0x40, colored in red in the example below). It starts with the main element '''qrc''' and his only children '''file-table''' that has his own 5 childrens '''file''' (all them are elements). Every '''file''' element has 2 attributes '''src''' and '''id'''.
When the qrc container is build all the info from he .xml structure is stored in the '''TOC''' (at offset 0x40) after the CXML header, and the files are stored in a "Files table" placed at bottom of the container (at offset 0x250)


The attribute '''src''' stores the information needed to locate the file, either in PC (before compiled) or locally inside the CXML container (after compiled). In the .xml example above the value of '''src''' are PC paths (used by the compiler to locate the file inside the PC), but when compiled the value is lost, the file is stored inside the container '''Files table''' (at offset 0x250, colored in orange in the example below), and is indexed in the container '''TOC''' by using an entry with data type 6 composed of '''file offset''' + '''file size''' (used by PS3 firmware to locate the file inside the container '''Files table''')
The '''TOC''' (colored in red tones in the example) starts with the elements: '''qrc''', and his only children '''file-table''' that has 5 '''file''' childrens (all them are elements). Every '''file''' element has 2 attributes '''src''' and '''id'''.
 
The value of attribute '''id''' is stored inside the container '''Strings table''' (at offset 0x0x1B0, colored in green in the example below), and is indexed in the container '''TOC''' by using an entry with data type 7 composed by '''filename offset''' + '''unknown''' (used by the PS3 to locate the text string inside the container '''Strings table''')


<div style="height:450px; overflow:auto">
{| class="wikitable" style="font-size:x-small;"
{| class="wikitable" style="font-size:x-small;"
|+[[rhm.qrc]] (v1.00)
|+[[rhm.qrc]] (v1.00)
Line 1,176: Line 1,196:
| 0x0C || 0x04 || 00000164 || TOC size  
| 0x0C || 0x04 || 00000164 || TOC size  
|- bgcolor="#88ff88"
|- bgcolor="#88ff88"
| 0x10 || 0x04 || 000001B0 || Strings table absolute offset || rowspan="2" | Stores the '''VALUES''' of XML attributes type 7 and 8
| 0x10 || 0x04 || 000001B0 || Strings table absolute offset || rowspan="2" | Stores the '''VALUES''' of XML attributes type 3 ('''string'''), and 7<!-- path/file.ext literal -->, and 8 ? <!-- i never found an 8-->
|- bgcolor="#88ff88"
|- bgcolor="#88ff88"
| 0x14 || 0x04 || 00000071 || Strings table size  
| 0x14 || 0x04 || 00000071 || Strings table size  
|- bgcolor="#8888ff"
|- bgcolor="#8888ff"
| 0x18 || 0x04 || 00000230 || TAGS table absolute offset || rowspan="2" | Stores the '''NAMES''' of XML [http://www.w3schools.com/xml/xml_syntax.asp '''elements''' and '''attributes''']]<br/ >and the '''VALUES''' of attribute types 3 ('''string''')
| 0x18 || 0x04 || 00000230 || TAGS table absolute offset || rowspan="2" | Stores the '''NAMES''' of XML [http://www.w3schools.com/xml/xml_syntax.asp '''elements''' and '''attributes''']]
|- bgcolor="#8888ff"
|- bgcolor="#8888ff"
| 0x1C || 0x04 || 0000001B || TAGS table size  
| 0x1C || 0x04 || 0000001B || TAGS table size  
Line 1,187: Line 1,207:
|- bgcolor="#ffeebb"
|- bgcolor="#ffeebb"
| 0x24 || 0x04 || 00000000 || Integer arrays table size  
| 0x24 || 0x04 || 00000000 || Integer arrays table size  
|- bgcolor="#ffee66"
|- bgcolor="#ffee99"
| 0x28 || 0x04 || 00000250 || Float arrays table absolute offset || rowspan="2" | Stores the '''VALUES''' of XML attributes type 5 ('''float array''')
| 0x28 || 0x04 || 00000250 || Float arrays table absolute offset || rowspan="2" | Stores the '''VALUES''' of XML attributes type 5 ('''float array''')
|- bgcolor="#ffee66"
|- bgcolor="#ffee99"
| 0x2C || 0x04 || 00000000 || Float arrays table size  
| 0x2C || 0x04 || 00000000 || Float arrays table size  
|- bgcolor="#ffaa00"
|- bgcolor="#ffaa00"
Line 1,195: Line 1,215:
|- bgcolor="#ffaa00"
|- bgcolor="#ffaa00"
| 0x34 || 0x04 || 00000DD0 || Files table size  
| 0x34 || 0x04 || 00000DD0 || Files table size  
|- bgcolor="#888888"
|- {{cellcolors|lightgrey}}
| 0x38 || 0x04 || 00000000 || Unknown area absolute offset || rowspan="2" | Unknown
| 0x38 || 0x04 || 00000000 || Reserved area absolute offset || rowspan="2" |  
|- bgcolor="#888888"
|- {{cellcolors|lightgrey}}
| 0x3C || 0x04 || 00000000 || Unknown area size  
| 0x3C || 0x04 || 00000000 || Reserved area size  
|-
|-
! rowspan=98 | TOC
! rowspan=95 | TOC
|- bgcolor="#cccccc"
| colspan="5" |
|- {{cellcolors|#ff7777}}
|- {{cellcolors|#ff7777}}
| 0x40 || 0x04 || 00000000 || element name relative offset || '''qrc'''
| 0x40 || 0x04 || 00000000 || element name relative offset || '''qrc'''
Line 1,217: Line 1,235:
|- {{cellcolors|#ff7777}}
|- {{cellcolors|#ff7777}}
| 0x58 || 0x04 || 0000001C || last child element relative offset || 0x40 + 0x1C = '''last child at 0x5C absolute'''
| 0x58 || 0x04 || 0000001C || last child element relative offset || 0x40 + 0x1C = '''last child at 0x5C absolute'''
|- bgcolor="#cccccc"
| colspan="5" |
|- {{cellcolors|#ff8888}}
|- {{cellcolors|#ff8888}}
| 0x5C || 0x04 || 00000004 || element name relative offset || '''file-table'''
| 0x5C || 0x04 || 00000004 || element name relative offset || '''file-table'''
Line 1,394: Line 1,410:
| 0x1A0 || 0x04 || 00000000 || attribute variable 2 ||  
| 0x1A0 || 0x04 || 00000000 || attribute variable 2 ||  
|- {{cellcolors|lightgrey}}
|- {{cellcolors|lightgrey}}
! rowspan=1 | Padding
| 0x1A4 || 0x0C || 000000000000000000000000 || padding || aligned to a multiply of 0x10
| 0x1A4 || 0x0C || 000000000000000000000000 || padding || aligned to a multiply of 0x10
|-
|-
! rowspan=12 | Strings Table
! rowspan=11 | Strings Table
|- bgcolor="#88ff88"
|- bgcolor="#88ff88"
| 0x1B0 || 0x04 || 00000038 || 1st element relative offset || 0x40 + 0x38 = '''element at offset 0x78 absolute'''
| 0x1B0 || 0x04 || 00000038 || 1st element relative offset || 0x40 + 0x38 = '''element at offset 0x78 absolute'''
Line 1,419: Line 1,436:
|- bgcolor="#cccccc"
|- bgcolor="#cccccc"
|- {{cellcolors|lightgrey}}
|- {{cellcolors|lightgrey}}
! rowspan=1 | Padding
| 0x221 || 0xF || 00000000000000000000000000000000 || padding || aligned to a multiply of 0x10
| 0x221 || 0xF || 00000000000000000000000000000000 || padding || aligned to a multiply of 0x10
|-
|-
! rowspan=3 | TAGS table
! rowspan=2 | TAGS table
|- bgcolor="#8888ff"
|- bgcolor="#8888ff"
| 0x230 || 0x1B || qrc file-table file src id || Container identifyer (first tag) + other tags from MAKE.XML || style="padding:0px;" | <pre style="margin:0px; padding:3px; background:#8888ff; color:#ffffff; border:none;"> <qrc>
| 0x230 || 0x1B || qrc file-table file src id || Container identifyer (first tag) + other tags from MAKE.XML || <syntaxhighlight lang="xml"> <qrc> <file-table> <file src= id=/></syntaxhighlight>
<file-table>
<file src=fileaccess id=filename/></pre>
|- bgcolor="#cccccc"
|- bgcolor="#cccccc"
|- {{cellcolors|lightgrey}}
|- {{cellcolors|lightgrey}}
! rowspan=1 | Padding
| 0x24B || 0x5 || 0000000000 || padding || aligned to a multiply of 0x10
| 0x24B || 0x5 || 0000000000 || padding || aligned to a multiply of 0x10
|- bgcolor="#ffeebb"
! Integer arrays table
| 0x250 || 0x00 ||  ||  || Empty
|- bgcolor="#ffee66"
! Float arrays table
| 0x250 || 0x00 ||  ||  || Empty
|-
|-
! rowspan="6" | Files Table
! rowspan="6" | Files Table
Line 1,447: Line 1,458:
|- bgcolor="#ffaa00"
|- bgcolor="#ffaa00"
| 0x790 || 0x890 || 00001B5B... || 5th file ||  
| 0x790 || 0x890 || 00001B5B... || 5th file ||  
|- bgcolor="#888888"
|-
! Unknown area
| 0x1020 || 0x00 ||  ||  || Empty
|}
|}
</div>


'''rhm.qrc (from firmware 1.00)'''
'''rhm.qrc (from firmware 1.00)'''
<pre style="height:450px;">
<div style="height:450px; overflow:auto">
  Offset(h) 00      04      08      0C
  Offset(h) 00      04      08      0C
   
   
Line 1,714: Line 1,724:
  00001000  401F9C6C 01D0100D 8086C0C3 60409F80  @.œl.Ð..€†ÀÃ`@Ÿ€
  00001000  401F9C6C 01D0100D 8086C0C3 60409F80  @.œl.Ð..€†ÀÃ`@Ÿ€
  00001010  401F9C6C 01D0000D 8086C0C3 60411F81  @.œl.Ð..€†ÀÃ`A..  
  00001010  401F9C6C 01D0000D 8086C0C3 60411F81  @.œl.Ð..€†ÀÃ`A..  
</pre>
</div>


==P3T==
==P3T==
Line 1,790: Line 1,800:
| 0x60 || 0x04 || 00000002 || xml attribute TAG type || 2
| 0x60 || 0x04 || 00000002 || xml attribute TAG type || 2
|- {{cellcolors|#ff8888}}
|- {{cellcolors|#ff8888}}
| 0x64 || 0x04 || 3F800000 || float || 3F800000 = '''1.0'''
| 0x64 || 0x04 || 3F800000 || float ! || 3F800000
|- {{cellcolors|#ff8888}}
|- {{cellcolors|#ff8888}}
| 0x68 || 0x04 || 00000000 || Not used || 0
| 0x68 || 0x04 || 00000000 || || 0
|- bgcolor="#cccccc"
|- bgcolor="#cccccc"
| colspan="5" | element start "scene" (is the 1st children of "raf" element)
| colspan="5" | element start "scene" (is the 1st children of "raf" element)
Line 1,818: Line 1,828:
| 0x90 || 0x04 || 00000001 || integer || 1
| 0x90 || 0x04 || 00000001 || integer || 1
|- {{cellcolors|#ff9999}}
|- {{cellcolors|#ff9999}}
| 0x94 || 0x04 || 00000000 || Not used || 0
| 0x94 || 0x04 || 00000000 || || 0
|- bgcolor="#cccccc"
|- bgcolor="#cccccc"
| colspan="5" | attribute start "light_num" (is an attribute of "scene" element)
| colspan="5" | attribute start "light_num" (is an attribute of "scene" element)
Line 1,828: Line 1,838:
| 0xA0 || 0x04 || 00000002 || integer || 2
| 0xA0 || 0x04 || 00000002 || integer || 2
|- {{cellcolors|#ff9999}}
|- {{cellcolors|#ff9999}}
| 0xA4 || 0x04 || 00000000 || Not used || 0
| 0xA4 || 0x04 || 00000000 || || 0
|- bgcolor="#cccccc"
|- bgcolor="#cccccc"
| colspan="5" | attribute start "script_num" (is an attribute of "scene" element)
| colspan="5" | attribute start "script_num" (is an attribute of "scene" element)
Line 1,838: Line 1,848:
| 0xB0 || 0x04 || 00000001 || integer || 1
| 0xB0 || 0x04 || 00000001 || integer || 1
|- {{cellcolors|#ff9999}}
|- {{cellcolors|#ff9999}}
| 0xB4 || 0x04 || 00000000 || Not used || 0
| 0xB4 || 0x04 || 00000000 || || 0
|- bgcolor="#cccccc"
|- bgcolor="#cccccc"
| colspan="5" | attribute start "actor_num" (is an attribute of "scene" element)
| colspan="5" | attribute start "actor_num" (is an attribute of "scene" element)
Line 1,848: Line 1,858:
| 0xC0 || 0x04 || 00000006 || integer || 6
| 0xC0 || 0x04 || 00000006 || integer || 6
|- {{cellcolors|#ff9999}}
|- {{cellcolors|#ff9999}}
| 0xC4 || 0x04 || 00000000 || Not used || 0
| 0xC4 || 0x04 || 00000000 || || 0
|- bgcolor="#cccccc"
|- bgcolor="#cccccc"
| colspan="5" | element start "actor" (is the children of "scene" element)  
| colspan="5" | element start "actor" (is the children of "scene" element)  
Line 1,872: Line 1,882:
| 0xE8 || 0x04 || 00000005 || xml attribute TAG type || 5
| 0xE8 || 0x04 || 00000005 || xml attribute TAG type || 5
|- {{cellcolors|#ff9999}}
|- {{cellcolors|#ff9999}}
| 0xEC || 0x04 || 00000000 || Float Arrays Table offset || 0
| 0xEC || 0x04 || 00000000 || || 0
|- {{cellcolors|#ff9999}}
|- {{cellcolors|#ff9999}}
| 0xF0 || 0x04 || 00000004 || Array size || 4 = '''1.0,1.0,1.0,1.0'''
| 0xF0 || 0x04 || 00000004 || || 4
|- bgcolor="#cccccc"
|- bgcolor="#cccccc"
| colspan="5" | attribute start "scale" (is an attribute of "actor" element)
| colspan="5" | attribute start "scale" (is an attribute of "actor" element)
Line 1,882: Line 1,892:
| 0xF8 || 0x04 || 00000005 || xml attribute TAG type || 5
| 0xF8 || 0x04 || 00000005 || xml attribute TAG type || 5
|- {{cellcolors|#ff9999}}
|- {{cellcolors|#ff9999}}
| 0xFC || 0x04 || 00000004 || Float Arrays Table offset || 4
| 0xFC || 0x04 || 00000004 || || 4
|- {{cellcolors|#ff9999}}
|- {{cellcolors|#ff9999}}
| 0x100 || 0x04 || 00000003 || Array size || 3 = '''1.28,0.64,0.64'''
| 0x100 || 0x04 || 00000003 || || 3
|- bgcolor="#cccccc"
|- bgcolor="#cccccc"
| colspan="5" | attribute start "uv_offset" (is an attribute of "actor" element)
| colspan="5" | attribute start "uv_offset" (is an attribute of "actor" element)
Line 1,892: Line 1,902:
| 0x108 || 0x04 || 00000005 || xml attribute TAG type || 5
| 0x108 || 0x04 || 00000005 || xml attribute TAG type || 5
|- {{cellcolors|#ff9999}}
|- {{cellcolors|#ff9999}}
| 0x10C || 0x04 || 00000007 || Float Arrays Table offset || 7
| 0x10C || 0x04 || 00000007 || || 7
|- {{cellcolors|#ff9999}}
|- {{cellcolors|#ff9999}}
| 0x110 || 0x04 || 00000002 || Array size || 2 = '''0.0,0.0'''
| 0x110 || 0x04 || 00000002 || || 2
|- bgcolor="#cccccc"
|- bgcolor="#cccccc"
| colspan="5" | attribute start "color" (is an attribute of "actor" element)
| colspan="5" | attribute start "color" (is an attribute of "actor" element)
Line 1,902: Line 1,912:
| 0x118 || 0x04 || 00000005 || xml attribute TAG type || 5  
| 0x118 || 0x04 || 00000005 || xml attribute TAG type || 5  
|- {{cellcolors|#ff9999}}
|- {{cellcolors|#ff9999}}
| 0x11C || 0x04 || 00000000 || Float Arrays Table offset || 0  
| 0x11C || 0x04 || 00000000 || || 0  
|- {{cellcolors|#ff9999}}
| 0x120 || 0x04 || 00000004 || Array size || 4 = '''1.0,1.0,1.0,1.0'''
|- bgcolor="#cccccc"
| colspan="5" | attribute start "anim_weight" (is an attribute of "actor" element)
|- {{cellcolors|#ff9999}}
|- {{cellcolors|#ff9999}}
| 0x124 || 0x04 || 00000063 || xml attribute TAG name relative offset || 63 = '''anim_weight'''
| 0x120 || 0x04 || 00000004 || || 4  
|- {{cellcolors|#ff9999}}
| 0x128 || 0x04 || 00000005 || xml attribute TAG type || 5
|- {{cellcolors|#ff9999}}
| 0x12C || 0x04 || 00000009 || Float Arrays Table offset || 9
|- {{cellcolors|#ff9999}}
| 0x130 || 0x04 || 00000004 || Array size || 4 = '''1.0,0.0,0.0,0.0'''
|- bgcolor="#cccccc"
| colspan="5" | attribute start "material" (is an attribute of "actor" element)
|- {{cellcolors|#ff9999}}
| 0x134 || 0x04 || 0000006F || xml attribute TAG name relative offset || 6F = '''material'''
|- {{cellcolors|#ff9999}}
| 0x138 || 0x04 || 00000008 || xml attribute TAG type || 8
|- {{cellcolors|#ff9999}}
| 0x13C || 0x04 || 00000000 || strings table offset || 0 = '''mtrl_logo'''
|- {{cellcolors|#ff9999}}
| 0x140 || 0x04 || 00000000 || Not used || 0
|- bgcolor="#cccccc"
| colspan="5" | attribute start "uv_scale" (is an attribute of "actor" element)
|- {{cellcolors|#ff9999}}
| 0x144 || 0x04 || 00000078 || xml attribute TAG name relative offset || 78 = '''uv_scale'''
|- {{cellcolors|#ff9999}}
| 0x148 || 0x04 || 00000005 || xml attribute TAG type || 5
|- {{cellcolors|#ff9999}}
| 0x14C || 0x04 || 0000000D || Float Arrays Table offset || D
|- {{cellcolors|#ff9999}}
| 0x150 || 0x04 || 00000002 || Array size || 2 = '''1.0,-1.0'''
|- bgcolor="#cccccc"
| colspan="5" | attribute start "id" (is an attribute of "actor" element)
|- {{cellcolors|#ff9999}}
| 0x154 || 0x04 || 00000081 || xml attribute TAG name relative offset || 6F = '''id'''
|- {{cellcolors|#ff9999}}
| 0x158 || 0x04 || 00000007 || xml attribute TAG type || 7
|- {{cellcolors|#ff9999}}
| 0x15C || 0x04 || 0000000E || strings table offset || E = '''logo'''
|- {{cellcolors|#ff9999}}
| 0x160 || 0x04 || 00000000 || Not used || 0
|-
|-
|}
|}
You can compare this table with the xml representation of colboot.raf contents in [http://www.psdevwiki.com/ps3/Talk:Coldboot.raf#Coldboot.sxml coldboot.raf talk page]
*Attribute 8 means there is another element in the xml (a different element than the one who owns the attribute) with the same name (so the loopback of his string points to a different element). In this case coldboot.raf uses the name '''mtrl_logo''' 2 times (but is only stored one time in the string table):
{{Boxcode|code=<syntaxhighlight lang="xml">
<raf> <!-- this is a reduced version of coldboot.raf.cxml -->
<scene>
<actor material="mtrl_logo"/>
</scene>
<material-table>
<material id="mtrl_logo">
</material-table>
</raf>
</syntaxhighlight>}}


=Tools=
=Tools=


==CXML decompiler==
==CXML decompiler==
Experimental python script made by flatz for research purposes, there was several versions of it where some minor changes where made, to read about that changes check the history of the page
Experimental tool coded by flatz for documenting purposes, three versions of the tool was released publically in a IRC channel the same day (first and second versions had bugs which were identified and fixed)


The tool extracts the files inside a CXML container (CXML, QRCF, P3TF, RAFO, etc...), the extracted filenames contains the hexcodes for fileoffset and filelenght, additionally it generates an .xml that represents the original CXML structure where can be seen the order of the files and all the other data inside it
The tool can extract the files contained inside a CXML container (CXML, QRCF, P3TF, RAFO, etc...), and generates an .xml that represents the original CXML structure


*Usage:
*Usage:
**You can copypaste the python code below and save it in your PC as cxmdecompiler.py then run it in command line without arguments to verify that it works fine (you need python v2.7 installed)
**Note the original files [[lines.qrc]] and [[coldboot.raf]] are compressed with zlib (a compression layer that affects the whole file and makes it imposible to read the cxml structure). The command line examples below considers the files was decompressed in a previous step (using a [[Qt_Resource_Container_%28QRC%29#ZLIB_archivers|zlib archiver]]), in this previous step the file-extension was renamed to "qrcf" and "rafo" (which are the decompressed versions where the cxml structure is fully readable and the tool can process it)
**The original files [[lines.qrc]] and [[coldboot.raf]] are zlib compressed (and the CXML decompiler cant read zlib compressed files). The command line examples below considers the files has been decompressed in a previous step (using a [[Qt_Resource_Container_(QRC)#QRC_related_tools|zlib tool]]), in that previous step the file-extension was renamed to "qrcf" and "rafo" (which are the decompressed versions where the cxml structure is fully readable and the tool can process it)


{{Keyboard|content=<syntaxhighlight lang="bash">C:\>cxmldecompiler.py lines.qrcf lines.xml</syntaxhighlight>}}
{{Keyboard|content=
{{Keyboard|content=<syntaxhighlight lang="bash">C:\>cxmldecompiler.py coldboot.rafo coldboot.xml</syntaxhighlight>}}
'''C:\Portables\cxml decompiler v3 alpha>decompiler.exe lines.qrcf lines.xml'''
}}
{{Keyboard|content=
'''C:\Portables\cxml decompiler v3 alpha>decompiler.exe coldboot.rafo coldboot.xml'''
}}


*CXML decompiler v3 alpha, download links: http://multiupload.biz/dnsipf0dkssz/cxml_decompiler_v3_alpha_MultiUpload.biz.7z.html
  '''Changelog'''
  '''Changelog'''
  '''---------'''
  '''---------'''
Line 1,981: Line 1,942:
  v3 alpha - Fixed offset/length displacements in the function that locates float values (xml tag attributes related with x,y,z axis in 3D space now looks correct)
  v3 alpha - Fixed offset/length displacements in the function that locates float values (xml tag attributes related with x,y,z axis in 3D space now looks correct)
  v4 alpha - Floats precission increased
  v4 alpha - Floats precission increased
v5 alpha - Floats precission adjusted (previous versions had errors in the output of coldboot.raf)
v6 alpha - Now the extracted filenames contains the hexcodes for fileoffset and filelength, both values are needed for the SEARCH/REPLACE patterns to remap them in the internal .QRC structure, as can be seen in the experiments made [https://www.psx-place.com/threads/research-modifying-the-coldboot-gameboot-sequence-custom_render_plugin-sprx-rco.25952/page-19#post-214240 here] and [https://www.psx-place.com/threads/research-modifying-gaia-visualization-custom_render_plugin-earth-qrc.27756/ here]
v7 alpha - Filenames simplifyed, is better to have the filenames standarized for automated/mass comparison purposes in between official and custom files


{{Boxcode|height=600px|title=CXML decompiler v7 alpha (python script)|code=<syntaxhighlight lang="python">
*CXML decompiler v4 alpha, source code:
<div style="height:600px; overflow:auto">
{{Boxcode|content=<syntaxhighlight lang="python">
#!python2
#!python2


Line 2,258: Line 2,218:
write_raw(f, '\"{0}\"'.format(attribute.get_int()))
write_raw(f, '\"{0}\"'.format(attribute.get_int()))
elif attribute.type == Attribute.TYPE_FLOAT:
elif attribute.type == Attribute.TYPE_FLOAT:
write_raw(f, '\"{0:3.7}\"'.format(attribute.get_float()))
write_raw(f, '\"{0}\"'.format(attribute.get_float()))
elif attribute.type == Attribute.TYPE_STRING:
elif attribute.type == Attribute.TYPE_STRING:
write_raw(f, '\"{0}\"'.format(attribute.get_string()))
write_raw(f, '\"{0}\"'.format(attribute.get_string()))
Line 2,266: Line 2,226:
array_length = len(array)
array_length = len(array)
for j in range(array_length):
for j in range(array_length):
write_raw(f, '{0:3.7}'.format(array[j]))
write_raw(f, '{0}'.format(array[j]))
if j + 1 < array_length:
if j + 1 < array_length:
write_raw(f, ',')
write_raw(f, ',')
Line 2,280: Line 2,240:
write_raw(f, '\"')
write_raw(f, '\"')
elif attribute.type == Attribute.TYPE_FILE:
elif attribute.type == Attribute.TYPE_FILE:
file_name = 'Offset=0x{0:08X}, Length=0x{1:08X}.bin'.format(attribute.offset, attribute.length)
file_name = '{0}_0x{1:08X}.bin'.format(self.document.file_prefix, attribute.offset)
file_data = attribute.get_file()
file_data = attribute.get_file()
with open(file_name, 'wb') as of:
with open(file_name, 'wb') as of:
Line 2,445: Line 2,405:
document.dump(f)
document.dump(f)
</syntaxhighlight>}}
</syntaxhighlight>}}
</div>


===CXML decompiler (exe for windows)===
*Problems/bugs
There are tools to pack a python script together with some python libraries into an executable .exe to run it in windows. Some versions of the CXML decompiler script was distributed in this format but most probably the download inks are dead or are old versions, if someone wants to upload some of them post the links here
<strike>In v3 floats conversion from hex is not correct (is rounded up, causing an innacuracy). Example:
 
In coldboot.raf the "rotation" value for the "actors" stored in hexadecimal is 3FC90FD8, this value is converted/extracted by "cxml decompiler v3" tool as: 1.5708 (the value has been rounded up)
*CXML decompiler v3 alpha, download links: http://multiupload.biz/dnsipf0dkssz/cxml_decompiler_v3_alpha_MultiUpload.biz.7z.html
The accurate conversion should be 1.570796 (no rounded up, 6 decimals precission)</strike> Fixed in v4
 
 
It's easy to fix that, just replace "{0:3.6}" (without quotes) with "{0}".
--[[User:Flatz|Flatz]] ([[User talk:Flatz|talk]]) 07:47, 16 November 2014 (EST)


{{File Formats}}
{{File Formats}}
<noinclude>[[Category:Main]]</noinclude>
<noinclude>[[Category:Main]]</noinclude>
Please note that all contributions to PS3 Developer wiki are considered to be released under the GNU Free Documentation License 1.2 (see PS3 Developer wiki:Copyrights for details). If you do not want your writing to be edited mercilessly and redistributed at will, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource. Do not submit copyrighted work without permission!

To protect the wiki against automated edit spam, we kindly ask you to solve the following hCaptcha:

Cancel Editing help (opens in new window)

Templates used on this page:

Retrieved from "http://www.psdevwiki.com/ps3/CXML_Containers"