Editing Talk:EDAT files


===EDAT Stuff===
My WIP on decryting EDATs. I've found 4 likely keys used in EDAT processing inside appldr (included in unedat.c).
[http://pastie.org/2765146 Mostly non-working unedat]

====JuanNadie====
Actually an EDAT has several keys. One is in the SELF, others are at the rif and others on firmware. That an EDAT works on lower firmware only proves that the firmware has those keys.

BTW I have tricked the PS3 into decrypt an EDAT (Buzz). The procedure involves knowing how the NPD element is used (most of this is already published).
<pre>
       struct {
         uint32_t magic;
         uint32_t unknown2;
         uint32_t license; /* 1 network, 2 local, 3 free */
         uint32_t type; /* 1 exec, 0x21 update */
         uint8_t content_id[48];
         uint8_t digest[16];
         uint8_t titlefilenameHash[16];
         uint8_t headerHash[16];
         uint64_t unknown3;
         uint64_t unknown4;
       } npdrm;
</pre>
http://www.ps3hax.net/showthread.php?p=275238#post275238

====EXE.trim.ALL====
Actually as I know:
<pre>
uint32_t unknown2;
/* is */
edat_version[4]; /* 1, 2 or 3 (as I know, 3.15 has version 1 & 2, >=3.40 have 1&2&3 */
/* and some more info for: */
uint32_t license; /* 0 = debug */
uint32_t type; /* 0 - just file */
</pre>
and uint64_t unknown4 is:
<pre>
uint8_t finalize[1]; /* 0x00 - finalized edat, 0x01 - finalized sdat, 0x80 - nonfinalized edat, 0x81 nonfinalized sdat */
uint8_t data_type[3]; /* 0x01 - compressed, 0x02 - plain txt, 0x03 - compressed plain txt, 0x05 -  compressed, 0x06 - plain txt, 0x07 - compressed plain txt, ..., [B]0x0c - edat[/B], 0x0d - compressed edat, 0x3c - sdat? */
uint8_t block_size[4]; /* default is 16 kbytes = 0x4000 */
uint8_t data_size[8]; /*decoded data size */
</pre>
[http://www.ps3hax.net/showpost.php?p=280975&postcount=74 74]


====JuanNadie====
I haven´t fully reversed the EDAT algorithm (I’m missing bytes from 0xB0 to 0xFF) but there is enough info for a post. The explanation is for those with flag (offset 0x80-0x83) equals to 0xC which is a normal EDAT. For others values minor changes should be made. Let’s begin.

'''USER SPACE:'''
(Probably incomplete (enough for decryption), most of it is guessed)
On previous post I mentioned the command useds by developers to open/read an EDAT. That command has 3 parts:

*Verify: This parts verifies the NPD element by checking that user is authorized to open the file (using the devklic). In addition it calls LV2 to create an entry in the memory (probably using syscall471) so for further steps it will use the devklic (free) or the decrypted rif key (paid) (see SELF algorithm). I’ll call this rifkey on the rest of the document.
*Open the file: This will call kernel space. Performs several checks and creates an entry on another table.
*Read: It also calls LV2, which will decrypt and send data back.

'''<KERNEL SPACE:'''

*When an open request is received:
**Read the first 0x100 bytes of the file.
**Validates first 0xA0 bytes on EDAT. This uses appldr with the following execution (see below):
***Single execution
***Encrypted arguments
***CMAC. Key: rifkey, expectedHash: data at 0xA0-0xAF
***No Encryption erk:null,riv:null
**Validates metadatasections. Again uses appldr:
***Multi execution. Blocks upto 0xFC00.
***Encrypted arguments
***CMAC. Key: rifkey, expectedHash:data at 0x90-0x9F
***No Encryption: erk:null, riv:null

Once the NPD is validated (We have validated header upto 0xAF and the metadatasections( which validate the data itself)) an entry on a memory table is created, the entry created by syscall471 is erased and returns. I’ll call this the EDATEncryptionDataTable. This table has 10 entries (remember the limitation of simultaneous files opened??...). Each entry has the following structure:
<pre>
typedef struct {
    uint64_t fileDescriptor;
    uint64_t offset;
    uint64_t fileLen;
    uint32_t flag;
    uint32_t blockSize;
    uint32_tnumBlocks;
    uint8_t blockBase[0x0C];
    uint8_t devklic[0x10];
    uint8_t digest[0x10];	
}EDATEncryptionData ;
</pre>
*fileDescriptor: The file descriptor
*offset: An additional offset to calculate the position of the metadataSection. Origin unknown. Values seen 0
*File len: Length of the uncompressed data. Copied from EDAT offset 0x88-0x8F
*flag: Flags required to process the data. Copied from EDAT offset 0x80-0x83. Some flags are:
**0x80000000: Debug
**0x01000000: SDAT… for those rifkey is calculated differently. Algorithm is not tested yet.
**0x00000004: Arguments for appldr are encrypted
**0x00000001: Data compressed. MetadataSection length will be 0x20 bytes.
*blockSize: The data is stored on blocks of that length. Required for decryption. Copied from EDAT offset 0x84-0x87
*numBlocks: Number of blocks on file. Calculated using: numBlocks = (fileLen + blockSize - 1)/blockSize
*blockBase: Common part of the key used on decryption. First 0xC bytes of headerHash. Copied from EDAT offset 0x60-0x6B.
*devklic: Our rif key. Copied from the entry created by syscall471. FOR SDAT it is calculated using other algorithm.
*padding: the digest from NPD. Will be the IV. Copied from EDAT offset 0x40-0x4F.
*When an read request is received:
**The system determines the blocks required (E.g; read from 0x3FFF to 0x8001 will read blocks 0,1 and 2).
**For each of the blocks:
***Calculate the blockKey as blockBase concat blockNumber.
***Calculate block offset: Start of NPD + offset + 0x100 + metadataSectionLen* numBlocks + blockSize*blockNumber. metadataSectionLen is 0x10 for flag 0xC (if compressed or with special hash would be 0x20).
***Calculate erk: AESECB128Encrypt, key:rifKey, data:blockKey
***Calculate hashKey: For 0xC is erk. (Others have an additional encryption)
***Execute appLdr with the following parameters:
****Single execution
****Encrypted arguments
****CMAC. Key: hashKey, expectedHash: metadataSection[blockIndex].hash
****EncryptionCBC erk:erk, riv:npd.digest

'''APPLDR:'''

As you know the appldr is used for decrypting SELFs (that part is documented on the wiki). However there is a second mode that is used for EDAT. To use that mode spu_arg.field35 is 5. For values 0 to 4 it will process a SELF and for others will return error. Graf_chokolo described the structure of spu_args for the SELF. However this changes for EDATs…

<pre>typedef struct
{
	uint64_t unk1;
	uint32_t hashFlags;
	uint32_t encryptionFlags;      
	uint64_t unk2;
	uint64_t len;
	uint64_t unk3;
	uint64_t stateAddress;       
	uint32_t field30;
	uint8_t  field34;       //Must be 0,1,2
	uint8_t  field35;       //Must be 0,1,2,3
	uint16_t  field36;      //Must be 0,1,2,3,4,5
	uint64_t field38;
	uint8_t hashKey[0x10];
	uint8_t riv[0x10];
	uint8_t erk[0x10];
}spu_args;
</pre>

Above I indicated several parameters for the appldr. Those parameters are controlled by hashFlag and encryptionFlag.

*HashFlag: The hash is performed on input data not decrypted data. So if hash fails nothing is decrypted

<pre>MSB indicates how to use parameter hashKey.

        0x10000000: Indicates that hashKey is encrypted. Decrypt it with EDATKEY and RIVKEY
        0x20000000: Ignore hashKEY. USE EDATDEFAULTHASHKEY
        0x00000000: hashKey is not encrypted. Use it directly
</pre>
<pre>LSB indicates the type of hash.

        0x00000001: HMACWithSHA1. Hash Len 0x14 bytes
        0x00000002: CMAC128. Hash len 0x10 (obvious)
        0x00000004: HMACWithSHA1. Hash Len 0x10 bytes (Hash is trunked)
</pre>

*EncryptionFlag:

<pre>MSB indicates how to use parameters riv and erk

        0x10000000: Indicated that erk is encrypted. Decrypt it with EDATKEY and RIVKEY. riv is copied
        0x20000000: Ignore riv and erk. Use EDATKEY and RIVKEY
        0x00000000: Use erk and riv directly
</pre>

<pre>LSB indicates which encoding is used.

        0x00000001: No encryption (algorithm is memcpy)
        0x00000002: AESCBC128Decrypt is used.
</pre>

When a key must be decrypted the algorithm is AESCBC128Decrypt. 

Keys SHA1:<pre>EDATKEY: 84E9FC3574EAA11A9462FFA53D5EA46B4D0003BF. Already in wiki
RIVKEY: E129F27C5103BC5CC44BCDF0A15E160D445066FF. This is top secret ;D
EDATDEFAULTHASHKEY: 8A721A06ABC7BB9BF398C5EF5D6F1FD997BC0A56</pre>

The multiple execution /single execution refers to the ability of appLdr to store an encrypted (and hashed) data on main memory to resume the hashing/decryption operations on next execution. That data is stored at stateAddress. Basically the functioning has several submodes. One for initialization (that decrypts erk, riv and hkey) and copies the state to main memory, another for processing data, which retrieves the state, process the data and updates the state, and finally another that receives the expected hash and validates. Another submode does all these ops on a single execution (calls the three submodes).

Summing up (for 0xC) for each block:
*Calculate data offset: 0x100 + offset(0) + metadataSectionLen*numberOfBlocks + currentBlockNumber*blockSize
*Generate block key: concat first 0xC byte of headerHash with current block number
*Encrypt block key with rifKey using AESECB128Encrypt. This would be erk and hkey
*Calculate riv. It is NPD digest field
*Decrypt erk using AESCBC128Decrypt with EDATKEY and RIVKEY -> call result decryptionKey.
*Using AESCBC128Decrypt decrypt data from offset to offset + blockSize( if last block calculate remaining bytes rounded up to 0x10 multiple). Key is decryptionKey iv is riv
*Copy data to output file

Source: http://www.ps3hax.net/showthread.php?p=289101#post289101

====JuanNadie====

After searching for those missing bytes I did not found any reference on code. So I zeroed them and feed the file to the PS3... The file was properly loaded and decrypted so:
-Those bits are a random padding or
-They are not checked ,nor required for creating an EDAT.

That means that you are able to "free" your EDATs using the method above.
Remember that you're going to need the devklic and the rifkey

@Octopus For p3T you don't require the devklic cause for paid decryption you only need the key from rif... so decrypt it and use it directly like any other theme

@EXE.trim.ALL You're right on version. LV2 uses that value to check which flags are valid. However unk4 (and unk3) which are at 0x70 are zeroes. Your info is for the next 16 bytes located at 0x80. Also you separate finalize and type. From the assembly I can say that the original source code considers it a signed integer (int32_t) (that or they have a really weird compiler cause it always load that field as a word (PPC can read a single byte)).

BTW the text on flowers says something about "werewolves" and the devklic
is written as an ASCII text (not binary). It was the first non free content that I decrypted on PC.

<!--// snip //-->
It's a pity that I would never enter on efnet (nor other sites) as long as they don't allow proxy and TOR.
:notes:
:: actually wiki accepts proxy (and tor?)
:: in case you need tor for ef.net, irc.paraphysics.net accepts tor connections

Finally a bit of drama: Hotz8611, I reversed your reactPSN to see the your method and discovered that you just used the info on this post. I haven't check the vsh.self mode but I suppose that youre nullifyng the curve check and generate RAP by encrypting the rifkey (An AES and another crypto algorithm of 5 rounds with shuffle, acummulation and xoring.). I thank you for making people provide me all those RAP than I'm able to transform to rifkey but next time give proper credits (I don't know if your works is based on mallory or mine but obviously is based on info in this topic). 

Source: http://www.ps3hax.net/showthread.php?p=289719#post289719

====JuanNadie====
First of all, here is an implementation of the algorithm. It is not fully tested (for example with ISO.BIN.EDAT it validates but fails when decrypting) and is missing the decompression algorithm (I tried deflate but is does not work if someone identifies the algorithm please post it. Blocks start with 0x05). Also keys have been eliminated. On previous port you have the SHA1.

http://pastebin.com/SuAukd8B

About compression: Instead of having metadatasections of 0x10 byes the new section is 0x20 bytes long.

Code:

struct compressMetadataSection {
	uint8_t hash[0x10];
	uint64_t fileOffset;
	uint32_t len;
	uint32_t isEndOfCompression;   
}

The obtain bytes 0x10-01F xor of data is used

@jester You probably forgot to decrypt the key. The result of syscall471 must be decrypted using EDATKEY and RIVKEY.

@Octopus to make free files.
- Create a memory image of the file.
- Decrypt the data, so you have a memory copy of the file decrypted (for compressed no need to decompress)
- Modify NPD to make if free (0x03 instead of 0x02). Recalculate hashes using devlikc (that why we need it).
- Recrypt each block using the devklic as base to calculate blocks keys (wich will then be transformed again as does the appldr for that type).
- Once recrypted, recalculate the hashes of the sections so they matched the new value for the encrypted data.
-If compressed recalculate the offset and len on the metadatasection
-Using the new metadatasection recalculate data 0x90-0x9F.
-Using the new NPD header + new metadatasectionHash calculate hash and place it at 0xA0-0xAF.
-Write file to disk (Remember, that file is function of filename, you must overwrite (not recommend while testing) or remember to rename it properly).

Done 

Source: http://www.ps3hax.net/showpost.php?p=299753&postcount=83

====JuanNadie====
The code I posted already implements SDAT. You just need the SDATKEY.
The SHA1 is ED2A015EEB1BD0CE06D0447F1A22AF4C1C401E4A

However you won't find it by bruteforce as it is coded as a series of inmediate values. If you check routine sub_5529C at graf_chokolo's dump_lv2 you'll see the routine that checks the edat/sdat header. Few lines below at address 5543C you'll see those inmediate values that are xored with the hash of the NPD to create the fake rifkey (the first 0x100 bytes of the npd are loaded starting at sp + 0x110)

About compression: We don´t know the algorithm but if we decrypt the data and modify the NPD so it looks like a debug you can use make_edata_npdrm to decompress it 

Source: http://www.ps3hax.net/showpost.php?p=300371&postcount=89

====JuanNadie====
First of all a new revision of the algorithm:

http://pastebin.com/E9ghYKj0  (backup: http://pastie.org/private/zanb7y2hjbk9yel71jhpq)

CHANGELOG:
*Added partial support for debug files
*Added NPD version check (between 0 and3)
*Added per version flags check
*Added support for FLAG_0x20. (Untested)
*Added support for versions 0 and 1 (ISO.BIN.EDAT). See below

When version is 0 or 1 instead of using the digest and the hash of the NPD for calculating the block key and the apploader's IV a zeroed byte array is used as base

I hope this helps your project Snowy.

@dsadsadsa I don't think I could help Kakaroto cause most of the checks are not on current code (They probably added further checking on later firmware version) and my expertise is reading assembly which I don't have. However if I can help then in any form they only need to contact me. In fact the only think that I know that is not already public is that 0x20 bytes are copied from the appldr to main memory just before setting mailbox to 7. Those bytes are at 0x890 and probably is the hash for the whitelist.

@EXE.trim.ALL There is no hash for SDAT. Those values are unknow and not checked(not confirmed). The only function as you have seen is to be xored to generate the key. To get the exacts values I'll need to check the SDK as those are generated there.

@Octopus I haven't look for the second part of the act.dat. My hyphotesis is that the second part of the file is used when debug is enabled and is common for all the consoles (will explain the COD trick). About the digest as I said there is no checked (is a hash of the original data which is unknown until the whole file is read so it can not be used as check). Geohot zeroed it on his code, An analysis of make package will obtain then (or wait until Kakaroto has the SELF fix and hope that it included the algorithm) 

Source: http://www.ps3hax.net/showpost.php?p=300903&postcount=93

====Wikify test====

====='''Header'''=====
{| class="wikitable" style="text-align: center;"
|-
! style="background-color:#123EDA; color:#FFFFFF;" |Offset  || style="background-color:#123EDA; color:#FFFFFF;" |Description || style="background-color:#123EDA; color:#FFFFFF;" |Size || style="background-color:#123EDA; color:#FFFFFF;" |Type || style="background-color:#123EDA; color:#FFFFFF;" |Note 
|-
| 0x00   || magic       || 4       || uint32 ||"NPD" (4E 50 44 00)
|-
| 0x04   || version     || 4       || uint32 ||
*01
*02
*03
*04
|-
| 0x08   || [[License_Types|License Types]]      || 4       || uint32 || 
*00 Debug / 
*01 Network License
*02 Local License
*03 Free
|-
| 0x10   || [[PARAM.SFO#CONTENT_ID|CONTENT ID]]    || 48 (0x30) || uint8 || 
|-
| 0x40   || Digest         || 16 (0x10) || uint8 || 
|-
| 0x50   || TitlefilenameHash     || 16 (0x10) || uint8 ||
|-
| 0x60   || HeaderHash     || 16 (0x10) || uint8 ||
|-
| 0x70   || 00             || 16 (0x10) || uint8 || filled with 00
|-
| 0x80   || NPD Flag       || 1  || uint8 || (Separated from NPD type for wiki format)
*00 Finalized EDAT
*01 Finalized SDAT
*80 Finalized EDAT
*81 Non Finalized SDAT
|-
| 0x83   || NPD type      || 3 || uint8 ||
*01 Compressed?
*02 Plain text?
*03 Compressed plain text?
*05 Compressed?
*06 Plain text?
*07 Compressed plain text?
*0D Compressed data? 
*3C data/misc? 
|-
| 0x84   || Block size     || 4 || uint8 || default is 16 kbytes = 0x4000
|-
| 0x88   || Data size     || 8  || uint8 || decoded data size
|-
|}

====='''Footer'''=====
{| class="wikitable" style="text-align: center;"
|-
! style="background-color:#123EDA; color:#FFFFFF;" |Description || style="background-color:#123EDA; color:#FFFFFF;" |Size || style="background-color:#123EDA; color:#FFFFFF;" |Note 
|-
| Name    || 5 || EDATA / SDATA
|-
| Space   || 1 || 0x20
|-
| Revision Version? || 1 ||
*Version 2: 2.4.0.L 
*Version 3: 3.3.0.W
*Version 4: 4.4.0.W
|-
| End of the file || 3 || zeroed
|}