Introduction

The following information was reverse engineered from LV1, Storage Manager in LPAR1, sb_iso_spu_module.self and sv_iso_spu_module.self.

Linux Driver ps3encdec

HDD sectors arrive in LV1 with tweak values already XORed.
But VFLASH sectors are first encrypted/decrypted with ENCDEC keys and the fact is that VFLASH sectors are NOT already XORed with tweak values. LV1 does pre- and post-XORing with tweak values.
LV1 allocated a DMA region where it stores sectors numbers for each request sector and ENCDEC encryptes these tweak values.
After that, LV1 XORes encrypted tweak values.
See EdecXTS_XorWithMask in LV1.

First host and ENCDEC device exchange random numbers.
From the exchanged random numbers host and ENCDEC device compute the session key.
ENCDEC commands, e.g. to set ATA keys, are encrypted with the session key and AES-CBC-192.
Before a secure communication channel is established, host and ENCDEC device use static AES-CBC-192 keys to encrypt communication data. The static keys can be found e.g. in sb_iso_spu_module.self or sv_iso_spu_module.self.
Static ENCDEC keys depend on SB bus version. To get your SB bus version, read v2 of repository node SB bus id.
During the communication, host and ENCDEC device use random IVs which are sent unencrypted together with encrypted payload.
The ENCDEC commands, which are encrypted with the session key, contain magic 24 bytes which are checked by ENCDEC device and if some bits are not correct then the command is denied.
The format of ENCDEC command to set ATA keys is slightly different from the ENCDEC command to set ENCDEC keys.