PlayStation archive (PSARC)

From PS3 Developer wiki
Revision as of 18:32, 14 July 2014 by Sandungas (talk | contribs) (→‎dtd)
Jump to navigation Jump to search

Description

PlayStation ARChive is a container that allows subfolders and per-file compression using Zlib or lzma with 9 compression levels

  • Notes:
    • lzma with compression level 9 has a compression ratio of around 50% of the size for general files
    • some files doesnt admits much compression like .BIK videos (max compression ratio for .BIK doesnt reachs 1% or 2%). For this reason is better to dont use compression with videos because the size reduction is ridiculous

Structure

See Discussion page

Header

PSARC header example: 

Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000000  50 53 41 52 00 01 00 04 7A 6C 69 62 00 01 23 BA  PSAR....zlib..#º
00000010  00 00 00 1E 00 00 09 16 00 01 00 00 00 00 00 02  ................
Offset Size Name Example Value (conversion) Notes
PSARC Header
0x00 0x04 magic 50 53 41 52 PSAR PlayStation Archive
0x04 0x04 version 00 01 00 04 v1.4 First 2 bytes is the main version, next 2 bytes is the subversion and are processed separatedlly, so 0001.0004 is "translated" to 1.4 by the system
0x08 0x04 compression type 7A 6C 69 62 zlib zlib (default) or lzma
0x0C 0x04 table of content length 00 01 23 BA 0x123BA
0x10 0x04 table of content entries size 00 00 00 1E 30 Bytes Default is 30 bytes
0x14 0x04 number of entries 00 00 09 16 1+2325 files The filecount includes the "file path/names definitions" as an additionall file always placed at first position in the list and without an assigned ID
0x18 0x04 block size 00 01 00 00 65536 Bytes Default is 65536 Bytes
0x1C 0x04 archive flags 00 00 00 02 Absolute paths Default is 0. See table below

PSARC Archive Flags are stored in the header as a big endian number, and defines the characteristics and structure of the archive when it was created

PSARC Archive Flags
Hexadecimal
(big endian)		 Decimal Binary
(flags)		 Flag Name Notes
00 00 00 00 0 00000000 00000000 00000000 00000000 relative archive paths (no slash character at the start of every line) default value when using command line or xml
00 00 00 01 1 00000000 00000000 00000000 00000001 ignorecase archive paths (all characters in capitals) same than --ignorecase in command line and ignorecase="true" in the xml
00 00 00 02 2 00000000 00000000 00000000 00000010 absolute archive paths (slash character added at the start of every line) same than --absolute in command line and absolute="true" in xml

TOC Table

Blocks info table

File table

The compression is applyed "per-block", is posible to know the compression level by looking at the zlib header

Zlib Header (more info here and here)
[edit]
byte index bit 7 bit 6 bit 5 bit 4 bit 3 bit 2 bit 1 bit 0
[0] CMF Compression info
  • 7 = indicates a 32K window size
 Compression method
  • 8 = denotes the "deflate" compression
[1] FLG FLEVEL
  • 0 - compressor used fastest algorithm
  • 1 - compressor used fast algorithm
  • 2 - compressor used default algorithm
  • 3 - compressor used maximum compression, slowest algorithm
 FDICT

(Preset dictionary) 

If set, a DICT dictionary identifier is present
immediately after the FLG byte. The dictionary is a sequence of
bytes which are initially fed to the compressor without
producing any compressed output. DICT is the Adler-32 checksum
of this sequence of bytes (see the definition of ADLER32
below).  The decompressor can use this identifier to determine
which dictionary has been used by the compressor.
FCHECK 
value must be such that CMF and FLG, when viewed as
a 16-bit unsigned integer stored in MSB order (CMF *256 + FLG),
is a multiple of 31.

PSARC Tools

There are several versions of PSARC related tools, features has been implemented along the different versions and is hard to keep a record of them, please help filling the info in the page

GUI versions are derivated from the command line versions

Command line versions

Open PSARC PS3 Extractor

By Matthieu Milan. It allows you to extract PSARC archive data in linux

Source code: https://www.ferb.fr/ps3/PSARC/

PSARC tool

Port of Matthieu Milan's open PSARC PS3 extractor to Windows. Tool written by Matthieu Milan (@usineur). Ported to VS2012 by AlexAltea.

source code: https://github.com/AlexAltea/psarc-tool 

Usage:
 psarc [option] filename

Options:
 -l                Create a text file that lists the file id, size, and name of every file in the archive.
 -x                Extracts all files.
 -e START END      Extracts files with the file id specified by the range between START and END (inclusive).

PSARC Extractor

By Insomniac @ xentax forums, download link: http://www.ps3hax.net/downloads.php?do=file&id=452 and http://www.modcontrol.com/Board/pc-tools/28869-psarc-ps3-extractor.html 

Usage:
 PSARC [flag] <fileName>

Flags:
 -l             Creates a text file that lists the File ID, Size, and Name of every file in the archive.
 -e START END   Extracts files with the File ID specified by the range between START and END (inclusive). 
 <blank>        Extracts all files.
  • Examples:
    • Extract all files from data.psarc : PSARC data.psarc
    • Create list of files from data.psarc : PSARC -l data.psarc
    • Extract file with File ID 5 : PSARC -e 5 5 data.psarc
    • Extract files with File ID 7 through 12 : PSARC -e 7 12 data.psarc


The flag -l creates a .txt file with a list of all the contents of the archive, the info included is: File ID, File original size, and File path inside psarc archive

Example of a .txt file created with the -l flag: 

ID       Size         Name
1      | 7,39 KB    | /my/files/dummies/dummy1.bin
2      | 8,80 KB    | /my/files/dummies/dummy2.bin
3      | 3,30 KB    | /my/files/dummies/dummy3.bin

PSARC

1.60.008 SDK : x
1.92.001 SDK : x
3.40.001 SDK : psarc-138.002 (C++), built on Feb 15 2010 at 16:17:41
3.41.001 SDK : psarc-138.002 (C++), built on Feb 15 2010 at 16:17:41
3.60.001 SDK : psarc-138.002 (C++), built on Feb 15 2010 at 16:17:41
3.70.001 SDK : psarc-138.002 (C++), built on Feb 15 2010 at 16:17:41
4.00.001 SDK : psarc-1310.001 (C++), built on Nov 16 2011 at 03:31:36
4.50.001 SDK : psarc-1310.001 (C++), built on Nov 16 2011 at 03:31:36

usage:
 psarc verb [options] [file...]
 psarc --xml=XMLFILE

verbs:
 create                                  Create an archive. (default)
 extract                                 Extract contents of an existing archive.
 list                                    List contents of an existing archive.
 dump                                    Print detailed info about archive for unit testing.
 verify                                  Verify an existing archive's structural integrity and ensure that all files can be decompressed.
 dtd                                     Display a DTD describing psarc's XML input syntax.

general options:
 -h, --help                              Show this help message and exit.
     --version                           Display version and exit.
 -d, --debug                             Debug: Show debug messages.
 -v, --verbose                           Verbose: Show additional progress. (default)
 -q, --quiet                             Quiet: Don't show any progress.
 -y, --overwrite                         Overwrite existing files when creating/extracting.
 --xml=XMLFILE                           XML list of actions to perform.

create options:
 -oFILE, --output=FILE                   Archive filename to create.
 -IINPUTFILE, --inputfile=INPUTFILE      Inputfile listing files to archive. Consider --xml instead, which gives you more control.
 -m, --mself                             Create a PSARC-MSELF hybrid file for PS3, which can contain encrypted special files like SDATA, SPRX, and SPU SELF.
 --zlib                                  Use Zlib when compressing. (default)
 --lzma                                  Use LZMA when compressing. (usually for PS3 PSN)
 --level=N                               Compression quality. 1 is fastest, 9 is best. (default = 9)
 -N, --nocompress                        Store all files uncompressed in the archive.
 -bBS, --blocksize=BS                    Use blocks of size BS. (default = 64KiB)
 -jJOBS, --jobs=JOBS                     Compression threads to run at a time. (default = number of CPUs)
 -sREGEX, --strip=REGEX                  Perl-compatible regex specifying a prefix to strip from the pathnames stored in the archive. This may be specified more than once. (default = current dir)
 -S, --strip-all                         Strip all paths from files stored in the archive.
 -a, --absolute                          Make the paths within the archive absolute.
 -R, --relative                          Make the paths within the archive relative. (default)
 -i, --ignorecase                        Make the archive directory case-insensitive.
 --exclude=WILDCARD                      Wildcard specifying files to exclude.
 --skip-missing-files                    If set then ignore it when a file cannot be found.
 --mergedups                             Compare the content of all files, and merge identical files so that only one copy of the data is included.

extract options:
 --input=FILE                            Archive to extract files from. (default is first file argument)
 --to=DIRECTORY                          Directory to write extracted files to. (default is current directory)

--blocksize is in bytes.

verify

Type This
C:\Portables\psarc>psarc.exe verify c:\psarctests\custom_folders.psarc
Verifying c:\psarctests\custom_folders.psarc
Validating 1: Folder1/Image1.bmp (6220854 bytes)
Validating 2: Folder2/Folder21/Image21.png (6142 bytes)
Validating 3: Folder3/Folder31/Folder311/Image311.jpg (12524 bytes)
Archive OK
verify output per-file
Validating <file ID/number>: file path inside psarc archive (file original size in bytes)

list

Type This
C:\Portables\psarc>psarc.exe list c:\psarctests\custom_folders.psarc
Listing c:\psarctests\custom_folders.psarc
Folder1/Image1.bmp (6220854/6220854 100%)
Folder2/Folder21/Image21.png (157/6142 2%)
Folder3/Folder31/Folder311/Image311.jpg (209/12524 1%)
list output per-file
file path inside psarc archive (file compressed size file original size compression percentage)
  • Notes:
    • In the compression percentage: 100% is no reduction, 0% is a huge reduction

dump

Type This
C:\Portables\psarc>psarc.exe dump c:\psarctests\custom_folders.psarc
Dumping c:\psarctests\custom_folders.psarc
Folder1/Image1.bmp  os=6220854 cs=6220854 fb=2
Folder2/Folder21/Image21.png  os=6142 cs=157 fb=97
Folder3/Folder31/Folder311/Image311.jpg  os=12524 cs=209 fb=98
dump output per-file
file path inside psarc archive file original size file compressed size file block start offset ?

dtd

The output of this command is a layout that defines the structure of an .xml document. See: http://www.w3schools.com/DTD/

Type This
C:\Portables\psarc>psarc.exe dtd
<!ELEMENT psarc (create|extract)+>

<!ELEMENT create (compression|strip|exclude|file|directory|filelist)+>
<!ATTLIST create archive CDATA #IMPLIED
                 absolute (true|false) "false"
                 ignorecase (true|false) "false"
                 mergedups (true|false) "false"
                 stripall (true|false) "false"
                 blocksize CDATA "65536"
                 jobs CDATA #IMPLIED
                 skipmissingfiles (true|false) "false"
                 format (psarc|psarc-mself) "psarc"
                 overwrite (true|false) "false">

<!ELEMENT compression EMPTY>
<!ATTLIST compression type (zlib|lzma) "zlib"
                      level (1|2|3|4|5|6|7|8|9) "9"
                      enabled (true|false) "true">

<!ELEMENT strip EMPTY>
<!ATTLIST strip regex CDATA #REQUIRED>

<!ELEMENT exclude EMPTY>
<!ATTLIST exclude regex CDATA #IMPLIED
                  wildcard CDATA #IMPLIED>

<!ELEMENT file (CDATA)>
<!ATTLIST file wildcard CDATA #IMPLIED
               regex CDATA #IMPLIED
               path CDATA #IMPLIED
               archivepath CDATA #IMPLIED
               fromarchive CDATA #IMPLIED
               compressed (true|false) #IMPLIED
               skipifmissing (true|false) #IMPLIED>

<!ELEMENT filelist EMPTY>
<!ATTLIST filelist path CDATA #REQUIRED>

<!ELEMENT directory EMPTY>
<!ATTLIST directory path CDATA #REQUIRED
                    archivepath CDATA #IMPLIED
                    fromarchive CDATA #IMPLIED
                    compressed (true|false) #IMPLIED
                    skipifmissing (true|false) #IMPLIED>

<!ELEMENT extract (file)+>
<!ATTLIST extract archive CDATA #IMPLIED
                  input CDATA #IMPLIED
                  to CDATA #IMPLIED
                  stripall (true|false) "false"
                  skipmissingfiles (true|false) "false"
                  overwrite (true|false) "false">
  • DTD Notes:
    • Elements are defined in the DTD as: <!ELEMENT element-name (element-content)>
    • Attributes are defined in the DTD as: <!ATTLIST element-name attribute-name attribute-type attribute-value>
    • Elements that contains childrens elements that can be repeated several times are defined with a +
    • Character strings attribute data types are defined with: CDATA
    • Required attributes are defined with: #REQUIRED, and optional attributes with: #IMPLIED

Based in the DTD layout is posible to create an .xml layout that automatizes all the actions of the tool, this example is a generic layout containing all the elements and attributes availables together 

Code Sample
  • Notes:
    • The purpose of this XML is to use it as a base layout containing all, but is not posible to use it like that, is needed to delete the tags that are not needed for a specific case, the ones that are incompatibles with other tags, and is needed to add paths, etc...
    • Some attributes appears as common in the DTD for "create" and "extract" but are not common, some of them has been deleted to avoid mistakes... but probably is needed to delete some more
    • Is posible to generate this kind of .xml by using as the source the output of the "list" command after applying over it some character strings replacing using regular expressions with a tool like http://findandreplace.codeplex.com/#

Total Commander plugin

http://www.totalcmd.net/plugring/PSARC.html

GUI versions

PS3 PSARC GUI

It's a graphical user interface to view, extract or create psarc archives. GUI written by aldo, unknown psarc tool used internally

Download link: http://aldostools.org/ps3_psarc_gui.rar

Games or Apps using PSARC files

External information PSARC Header info Notes
Platform TITLE_ID TITLE Version
(offset 0x04)		 Compression
(offset 0x08)		 TOC entry size
(offset 0x10)		 Block Size
(offset 0x18)		 Flags
(offset 0x1C)
PS3 icon Ape Escape 00010004 Zlib 0000001E 00010000 00000002 uses full paths (created with --absolute)
PS3 icon Atelier Rorona Plus 00010004 zlib 0000001E 00010000 00000002
PS3 icon God of War. Ascension 00010004 zlib 0000001E 00010000 00000002
PS3 icon Killzone 3 00010004 zlib 0000001E 00010000 00000001
PS3 icon Mass Effect 3 00010004 lzma 0000001E 00010000 00000003 AUDIO.PSARC created with: --lzma --nocompress --absolute --ignorecase
MAIN.PSARC created with: --lzma --absolute --ignorecase
PS3 icon Motorstorm. Pacific Rift
PS3 icon Ni No Kuni .psarc file extensions renamed to .adat
PS3 icon The Last Of Us 00010003 zlib 0000001E 00010000 00000000 the game can load files from decompressed psarcs (because the original psarcs doesnt uses flags)
PS3 icon Uncharted 2 00010003 zlib 0000001E 00010000 00000000 the game can load files from decompressed psarcs (because the original psarcs doesnt uses flags)
PS3 icon Warhawk
PS4 icon VidZone 00010004 zlib 0000001E 00010000 00000000
  • Other games that probably uses PSARC's
    • Uncharted 1?, and 3?
    • Killzone colletion ?
    • God of War. III ?
    • God of War. Collection ?
    • other Motorstorms ?
  • Rebuild notes:
    • Mass effect 3 MAIN.PSARC contains a folder with generic files (compressed with level 9) and another folder with .BIK videos (not compressed at all). In the creation options suggested above in the table the compression level is not defined and this means all them will use the default compression level 9 (so the .BIK files will have an 1% or 2% of compression ratio). This is not how the original file was created (but works), it seems there is a way to define specific compression levels "per file" or "per folder" (by using a DTD.XML filelist?)
SCE File Types
Certified File
NPDRM
NPDRM · App Types · ACT.DAT · RIF
PKG
EDAT
RCOXML
CXML
XMBML
XMB Config
DB Config
Databases
System Data
XMB Preload
Licences
 ·
Games/Apps
PS3
Emulators
Web
Unsorted
Retrieved from "http://www.psdevwiki.com/ps3/index.php?title=PlayStation_archive_(PSARC)&oldid=29091"