Editing CXML Containers

Jump to navigation Jump to search
Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then publish the changes below to finish undoing the edit.

Latest revision Your text
Line 425: Line 425:
*For QRCF_2 and P3T themes only is stored the "id" (the file "src" is missing so there is no way to read the original path or file extension from the container)
*For QRCF_2 and P3T themes only is stored the "id" (the file "src" is missing so there is no way to read the original path or file extension from the container)


{{Boxcode|code=<syntaxhighlight lang="xml">
{{Boxcode|content=<syntaxhighlight lang="xml">
<?xml version="1.0" encoding="UTF-8"?>
<?xml version="1.0" encoding="UTF-8"?>
<qrc>
<qrc>
Line 1,119: Line 1,119:
*Example of the string table from frame_01.cxml
*Example of the string table from frame_01.cxml


{{Boxcode|code=<syntaxhighlight lang="xml">
{{Boxcode|content=<syntaxhighlight lang="xml">
root.version.1.00.jacket.id.normal.base.position.rotation.scale.name.frame_01.filetable.num.file.src.frame_01_lo.gim.lod.frame_01_parallax_512.gtf.shadow_square_1.gtf.shadow_square_2.gtf.shadow_3.gtf.frame_01_mi.gim.frame_01_hi.gim.vptable.vp.vp_jacket_ezprim.vpo.optional.vp_jacket_normalmap.vpo.vp_jacket_ezshadow.vpo.fptable.fp.fp_jacket_ezprim_rgb.fpo.fp_jacket_normalmap_specL.fpo.fp_jacket_ezshadow.fpo.modeltable.model.color.uv.vertex.tangent.binormal.layertable.layer.priority.type.image-frame-static.order.duration.image-thumbnail-auto
root.version.1.00.jacket.id.normal.base.position.rotation.scale.name.frame_01.filetable.num.file.src.frame_01_lo.gim.lod.frame_01_parallax_512.gtf.shadow_square_1.gtf.shadow_square_2.gtf.shadow_3.gtf.frame_01_mi.gim.frame_01_hi.gim.vptable.vp.vp_jacket_ezprim.vpo.optional.vp_jacket_normalmap.vpo.vp_jacket_ezshadow.vpo.fptable.fp.fp_jacket_ezprim_rgb.fpo.fp_jacket_normalmap_specL.fpo.fp_jacket_ezshadow.fpo.modeltable.model.color.uv.vertex.tangent.binormal.layertable.layer.priority.type.image-frame-static.order.duration.image-thumbnail-auto
</syntaxhighlight>}}
</syntaxhighlight>}}
Line 1,142: Line 1,142:




The .xml below represents the structure of [[rhm.qrc]] used in firmwares from 1.00 up to 1.32 ready to be compiled (compiled hash should be MD5: D1E7281246EC8A46EB6D368BD4B332BD)
The .xml below represents the structure of [[rhm.qrc]] from firmware 1.00 ready to be compiled
{{Boxcode|code=<syntaxhighlight lang="xml">
 
{{Boxcode|content=<syntaxhighlight lang="xml">
<?xml version="1.0" encoding="UTF-8"?>
<?xml version="1.0" encoding="UTF-8"?>
<qrc>
<qrc>
Line 1,156: Line 1,157:
</syntaxhighlight>}}
</syntaxhighlight>}}


When the file is compiled the info from the .xml is stored in the '''TOC''' (at offset 0x40, colored in red in the example below). It starts with the main element '''qrc''' and his only children '''file-table''' that has his own 5 childrens '''file''' (all them are elements). Every '''file''' element has 2 attributes '''src''' and '''id'''.
When the file is compiled the info from he .xml is stored in the '''TOC''' (at offset 0x40, colored in red in the example below). It starts with the main element '''qrc''' and his only children '''file-table''' that has his own 5 childrens '''file''' (all them are elements). Every '''file''' element has 2 attributes '''src''' and '''id'''.


The attribute '''src''' stores the information needed to locate the file, either in PC (before compiled) or locally inside the CXML container (after compiled). In the .xml example above the value of '''src''' are PC paths (used by the compiler to locate the file inside the PC), but when compiled the value is lost, the file is stored inside the container '''Files table''' (at offset 0x250, colored in orange in the example below), and is indexed in the container '''TOC''' by using an entry with data type 6 composed of '''file offset''' + '''file size''' (used by PS3 firmware to locate the file inside the container '''Files table''')
The attribute '''src''' stores the information needed to locate the file, either in PC (before compiled) or locally inside the CXML container (after compiled). In the .xml example above the value of '''src''' are PC paths (used by the compiler to locate the file inside the PC), but when compiled the value is lost, the file is stored inside the container '''Files table''' (at offset 0x250, colored in orange in the example below), and is indexed in the container '''TOC''' by using an entry with data type 6 composed of '''file offset''' + '''file size''' (used by PS3 firmware to locate the file inside the container '''Files table''')
Line 1,187: Line 1,188:
|- bgcolor="#ffeebb"
|- bgcolor="#ffeebb"
| 0x24 || 0x04 || 00000000 || Integer arrays table size  
| 0x24 || 0x04 || 00000000 || Integer arrays table size  
|- bgcolor="#ffee66"
|- bgcolor="#ffee99"
| 0x28 || 0x04 || 00000250 || Float arrays table absolute offset || rowspan="2" | Stores the '''VALUES''' of XML attributes type 5 ('''float array''')
| 0x28 || 0x04 || 00000250 || Float arrays table absolute offset || rowspan="2" | Stores the '''VALUES''' of XML attributes type 5 ('''float array''')
|- bgcolor="#ffee66"
|- bgcolor="#ffee99"
| 0x2C || 0x04 || 00000000 || Float arrays table size  
| 0x2C || 0x04 || 00000000 || Float arrays table size  
|- bgcolor="#ffaa00"
|- bgcolor="#ffaa00"
Line 1,195: Line 1,196:
|- bgcolor="#ffaa00"
|- bgcolor="#ffaa00"
| 0x34 || 0x04 || 00000DD0 || Files table size  
| 0x34 || 0x04 || 00000DD0 || Files table size  
|- bgcolor="#888888"
|- {{cellcolors|lightgrey}}
| 0x38 || 0x04 || 00000000 || Unknown area absolute offset || rowspan="2" | Unknown
| 0x38 || 0x04 || 00000000 || Reserved area absolute offset || rowspan="2" |  
|- bgcolor="#888888"
|- {{cellcolors|lightgrey}}
| 0x3C || 0x04 || 00000000 || Unknown area size  
| 0x3C || 0x04 || 00000000 || Reserved area size  
|-
|-
! rowspan=98 | TOC
! rowspan=96 | TOC
|- bgcolor="#cccccc"
| colspan="5" |
|- {{cellcolors|#ff7777}}
|- {{cellcolors|#ff7777}}
| 0x40 || 0x04 || 00000000 || element name relative offset || '''qrc'''
| 0x40 || 0x04 || 00000000 || element name relative offset || '''qrc'''
Line 1,217: Line 1,216:
|- {{cellcolors|#ff7777}}
|- {{cellcolors|#ff7777}}
| 0x58 || 0x04 || 0000001C || last child element relative offset || 0x40 + 0x1C = '''last child at 0x5C absolute'''
| 0x58 || 0x04 || 0000001C || last child element relative offset || 0x40 + 0x1C = '''last child at 0x5C absolute'''
|- bgcolor="#cccccc"
| colspan="5" |
|- {{cellcolors|#ff8888}}
|- {{cellcolors|#ff8888}}
| 0x5C || 0x04 || 00000004 || element name relative offset || '''file-table'''
| 0x5C || 0x04 || 00000004 || element name relative offset || '''file-table'''
Line 1,423: Line 1,420:
! rowspan=3 | TAGS table
! rowspan=3 | TAGS table
|- bgcolor="#8888ff"
|- bgcolor="#8888ff"
| 0x230 || 0x1B || qrc file-table file src id || Container identifyer (first tag) + other tags from MAKE.XML || style="padding:0px;" | <pre style="margin:0px; padding:3px; background:#8888ff; color:#ffffff; border:none;"> <qrc>
| 0x230 || 0x1B || qrc file-table file src id || Container identifyer (first tag) + other tags from MAKE.XML || <syntaxhighlight lang="xml"> <qrc>
<file-table>
<file-table>
<file src=fileaccess id=filename/></pre>
<file src=fileaccess id=filename/></syntaxhighlight>
|- bgcolor="#cccccc"
|- bgcolor="#cccccc"
|- {{cellcolors|lightgrey}}
|- {{cellcolors|lightgrey}}
| 0x24B || 0x5 || 0000000000 || padding || aligned to a multiply of 0x10
| 0x24B || 0x5 || 0000000000 || padding || aligned to a multiply of 0x10
|- bgcolor="#ffeebb"
! Integer arrays table
| 0x250 || 0x00 ||  ||  || Empty
|- bgcolor="#ffee66"
! Float arrays table
| 0x250 || 0x00 ||  ||  || Empty
|-
|-
! rowspan="6" | Files Table
! rowspan="6" | Files Table
Line 1,447: Line 1,438:
|- bgcolor="#ffaa00"
|- bgcolor="#ffaa00"
| 0x790 || 0x890 || 00001B5B... || 5th file ||  
| 0x790 || 0x890 || 00001B5B... || 5th file ||  
|- bgcolor="#888888"
|-
! Unknown area
| 0x1020 || 0x00 ||  ||  || Empty
|}
|}


'''rhm.qrc (from firmware 1.00)'''
'''rhm.qrc (from firmware 1.00)'''
<pre style="height:450px;">
<div style="height:450px; overflow:auto">
  Offset(h) 00      04      08      0C
  Offset(h) 00      04      08      0C
   
   
Line 1,714: Line 1,703:
  00001000  401F9C6C 01D0100D 8086C0C3 60409F80  @.œl.Ð..€†ÀÃ`@Ÿ€
  00001000  401F9C6C 01D0100D 8086C0C3 60409F80  @.œl.Ð..€†ÀÃ`@Ÿ€
  00001010  401F9C6C 01D0000D 8086C0C3 60411F81  @.œl.Ð..€†ÀÃ`A..  
  00001010  401F9C6C 01D0000D 8086C0C3 60411F81  @.œl.Ð..€†ÀÃ`A..  
</pre>
</div>


==P3T==
==P3T==
Line 1,950: Line 1,939:


*Attribute 8 means there is another element in the xml (a different element than the one who owns the attribute) with the same name (so the loopback of his string points to a different element). In this case coldboot.raf uses the name '''mtrl_logo''' 2 times (but is only stored one time in the string table):
*Attribute 8 means there is another element in the xml (a different element than the one who owns the attribute) with the same name (so the loopback of his string points to a different element). In this case coldboot.raf uses the name '''mtrl_logo''' 2 times (but is only stored one time in the string table):
{{Boxcode|code=<syntaxhighlight lang="xml">
{{Boxcode|content=<syntaxhighlight lang="xml">
<raf> <!-- this is a reduced version of coldboot.raf.cxml -->
<raf> <!-- this is a reduced version of coldboot.raf.cxml -->
<scene>
<scene>
Line 1,964: Line 1,953:


==CXML decompiler==
==CXML decompiler==
Experimental python script made by flatz for research purposes, there was several versions of it where some minor changes where made, to read about that changes check the history of the page
Experimental tool coded by flatz for documenting purposes, three versions of the tool was released publically in a IRC channel the same day (first and second versions had bugs which were identified and fixed)


The tool extracts the files inside a CXML container (CXML, QRCF, P3TF, RAFO, etc...), the extracted filenames contains the hexcodes for fileoffset and filelenght, additionally it generates an .xml that represents the original CXML structure where can be seen the order of the files and all the other data inside it
The tool can extract the files contained inside a CXML container (CXML, QRCF, P3TF, RAFO, etc...), and generates an .xml that represents the original CXML structure


*Usage:
*Usage:
**You can copypaste the python code below and save it in your PC as cxmdecompiler.py then run it in command line without arguments to verify that it works fine (you need python v2.7 installed)
**Note the original files [[lines.qrc]] and [[coldboot.raf]] are compressed with zlib (a compression layer that affects the whole file and makes it imposible to read the cxml structure). The command line examples below considers the files was decompressed in a previous step (using a [[Qt_Resource_Container_%28QRC%29#ZLIB_archivers|zlib archiver]]), in this previous step the file-extension was renamed to "qrcf" and "rafo" (which are the decompressed versions where the cxml structure is fully readable and the tool can process it)
**The original files [[lines.qrc]] and [[coldboot.raf]] are zlib compressed (and the CXML decompiler cant read zlib compressed files). The command line examples below considers the files has been decompressed in a previous step (using a [[Qt_Resource_Container_(QRC)#QRC_related_tools|zlib tool]]), in that previous step the file-extension was renamed to "qrcf" and "rafo" (which are the decompressed versions where the cxml structure is fully readable and the tool can process it)


{{Keyboard|content=<syntaxhighlight lang="bash">C:\>cxmldecompiler.py lines.qrcf lines.xml</syntaxhighlight>}}
{{Keyboard|content=
{{Keyboard|content=<syntaxhighlight lang="bash">C:\>cxmldecompiler.py coldboot.rafo coldboot.xml</syntaxhighlight>}}
'''C:\Portables\cxml decompiler v3 alpha>decompiler.exe lines.qrcf lines.xml'''
}}
{{Keyboard|content=
'''C:\Portables\cxml decompiler v3 alpha>decompiler.exe coldboot.rafo coldboot.xml'''
}}


*CXML decompiler v3 alpha, download links: http://multiupload.biz/dnsipf0dkssz/cxml_decompiler_v3_alpha_MultiUpload.biz.7z.html
  '''Changelog'''
  '''Changelog'''
  '''---------'''
  '''---------'''
Line 1,981: Line 1,974:
  v3 alpha - Fixed offset/length displacements in the function that locates float values (xml tag attributes related with x,y,z axis in 3D space now looks correct)
  v3 alpha - Fixed offset/length displacements in the function that locates float values (xml tag attributes related with x,y,z axis in 3D space now looks correct)
  v4 alpha - Floats precission increased
  v4 alpha - Floats precission increased
v5 alpha - Floats precission adjusted (previous versions had errors in the output of coldboot.raf)
v6 alpha - Now the extracted filenames contains the hexcodes for fileoffset and filelength, both values are needed for the SEARCH/REPLACE patterns to remap them in the internal .QRC structure, as can be seen in the experiments made [https://www.psx-place.com/threads/research-modifying-the-coldboot-gameboot-sequence-custom_render_plugin-sprx-rco.25952/page-19#post-214240 here] and [https://www.psx-place.com/threads/research-modifying-gaia-visualization-custom_render_plugin-earth-qrc.27756/ here]
v7 alpha - Filenames simplifyed, is better to have the filenames standarized for automated/mass comparison purposes in between official and custom files


{{Boxcode|height=600px|title=CXML decompiler v7 alpha (python script)|code=<syntaxhighlight lang="python">
*CXML decompiler v4 alpha, source code:
<div style="height:600px; overflow:auto">
{{Boxcode|content=<syntaxhighlight lang="python">
#!python2
#!python2


Line 2,258: Line 2,250:
write_raw(f, '\"{0}\"'.format(attribute.get_int()))
write_raw(f, '\"{0}\"'.format(attribute.get_int()))
elif attribute.type == Attribute.TYPE_FLOAT:
elif attribute.type == Attribute.TYPE_FLOAT:
write_raw(f, '\"{0:3.7}\"'.format(attribute.get_float()))
write_raw(f, '\"{0}\"'.format(attribute.get_float()))
elif attribute.type == Attribute.TYPE_STRING:
elif attribute.type == Attribute.TYPE_STRING:
write_raw(f, '\"{0}\"'.format(attribute.get_string()))
write_raw(f, '\"{0}\"'.format(attribute.get_string()))
Line 2,266: Line 2,258:
array_length = len(array)
array_length = len(array)
for j in range(array_length):
for j in range(array_length):
write_raw(f, '{0:3.7}'.format(array[j]))
write_raw(f, '{0}'.format(array[j]))
if j + 1 < array_length:
if j + 1 < array_length:
write_raw(f, ',')
write_raw(f, ',')
Line 2,280: Line 2,272:
write_raw(f, '\"')
write_raw(f, '\"')
elif attribute.type == Attribute.TYPE_FILE:
elif attribute.type == Attribute.TYPE_FILE:
file_name = 'Offset=0x{0:08X}, Length=0x{1:08X}.bin'.format(attribute.offset, attribute.length)
file_name = '{0}_0x{1:08X}.bin'.format(self.document.file_prefix, attribute.offset)
file_data = attribute.get_file()
file_data = attribute.get_file()
with open(file_name, 'wb') as of:
with open(file_name, 'wb') as of:
Line 2,445: Line 2,437:
document.dump(f)
document.dump(f)
</syntaxhighlight>}}
</syntaxhighlight>}}
</div>


===CXML decompiler (exe for windows)===
*Problems/bugs
There are tools to pack a python script together with some python libraries into an executable .exe to run it in windows. Some versions of the CXML decompiler script was distributed in this format but most probably the download inks are dead or are old versions, if someone wants to upload some of them post the links here
<strike>In v3 floats conversion from hex is not correct (is rounded up, causing an innacuracy). Example:
 
In coldboot.raf the "rotation" value for the "actors" stored in hexadecimal is 3FC90FD8, this value is converted/extracted by "cxml decompiler v3" tool as: 1.5708 (the value has been rounded up)
*CXML decompiler v3 alpha, download links: http://multiupload.biz/dnsipf0dkssz/cxml_decompiler_v3_alpha_MultiUpload.biz.7z.html
The accurate conversion should be 1.570796 (no rounded up, 6 decimals precission)</strike> Fixed in v4
 
 
It's easy to fix that, just replace "{0:3.6}" (without quotes) with "{0}".
--[[User:Flatz|Flatz]] ([[User talk:Flatz|talk]]) 07:47, 16 November 2014 (EST)


{{File Formats}}
{{File Formats}}
<noinclude>[[Category:Main]]</noinclude>
<noinclude>[[Category:Main]]</noinclude>
Please note that all contributions to PS3 Developer wiki are considered to be released under the GNU Free Documentation License 1.2 (see PS3 Developer wiki:Copyrights for details). If you do not want your writing to be edited mercilessly and redistributed at will, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource. Do not submit copyrighted work without permission!

To protect the wiki against automated edit spam, we kindly ask you to solve the following hCaptcha:

Cancel Editing help (opens in new window)

Templates used on this page:

Retrieved from "http://www.psdevwiki.com/ps3/CXML_Containers"