Introduction

The following information was reverse engineered from LV1, Storage Manager in LPAR1, sb_iso_spu_module.self and sv_iso_spu_module.self.

Linux Driver ps3encdec

HDD sectors arrive in LV1 with tweak values already XORed.
But VFLASH sectors are first encrypted/decrypted with ENCDEC keys and the fact is that VFLASH sectors are NOT already XORed with tweak values. LV1 does pre- and post-XORing with tweak values.
LV1 allocates a DMA region where it stores the sector number for each requested sector and ENCDEC encryptes these tweak values.
After that, LV1 XORes encrypted tweak values.
See encdec_device_enqueue_decsec_request and EdecXTS_XorWithMask in LV1.
I patched my LV1 for testing and killed XORing with encrypted tweak values. After that VFLASH sectors were encrypted/decrypted without tweak values, only with XTS data key.

First host and ENCDEC device exchange random numbers.
From the exchanged random numbers host and ENCDEC device compute the session key.
ENCDEC commands, e.g. to set ATA keys, are encrypted with the session key and AES-CBC-192.
Before a secure communication channel is established, host and ENCDEC device use static AES-CBC-192 keys to encrypt communication data. The static keys can be found e.g. in sb_iso_spu_module.self or sv_iso_spu_module.self.
Static ENCDEC keys depend on SB bus version. To get your SB bus version, read v2 of repository node SB bus id.
During the communication, host and ENCDEC device use random IVs which are sent unencrypted together with encrypted payload.
The ENCDEC commands, which are encrypted with the session key, contain magic 24 bytes which are checked by ENCDEC device and if some bits are not correct then the command is denied. The magic bytes can be found in sb_iso_spu_module.self too.
The format of ENCDEC command to set ATA keys is slightly different from the ENCDEC command to set ENCDEC keys.

Used by host to send host random to ENCDEC device and receive ENCDEC random from ENCDEC device.
Sent data is encrypted with host static key and received data is encrypted with ENCDEC static key.

The command resets ENCDEC device state.
LV1 calls this command "EdecKgenFlash", probably correct name is "EdecKgenFlush" :)

No clue what it does but it is sent by Storage Manager after ATA keys are cleared.