Basic Bluray disc authentication procedure
This is based on my (mathieulh) post in lan.st: http://lan.st/showthread.php?t=1722 and will need further editing and additions that will happen soon. So far this article is a 1:1 copy of my post. Thank you for your understanding.
Whenever you insert a Blu-Ray disc, the PlayStation 3 drive will look at a special area of the disc called the PIC Zone (the BD ROM Mark is actually used in movie discs but not in game unlike what I first thought).
The PIC Zone (which stands for Permanent Information & Control data Zone) contains information/data related to the disc's authentication which is done by the Blu-Ray drive. This area cannot easily be dumped (you'd pretty much need a Blu-Ray drive with a hacked firmware) and of course that specific area cannot be burned on any kind of discs or with any kind of burners commercially available.
There is no absolute certainty to what data the PIC Zone actually contains. I initially thought that Sony would use a public/private cryptography cipher to authenticate the discs but that is quite unlikely considering the limited resources of the drive controller. There isn't any kind of hard cryptographic layer on the discs as I first expected, so the security on the discs themselves is much less invasive as I initially thought. Yet the fact that the PIC Zone can't be rewitten without any kind of special equipment does its job well when it comes to preventing backups.
The authentication procedure itself is done through the use of a per Blu-Ray drive key pair, one being located on the drive controller itself while the other is stored encrypted in the playstation 3's EID area located on NAND. This key is also used while updating the drive which firmware's will be physically re-encrypted using that very same key and stored that way. As such you cannot swap a drive controller board from one ps3 to another, at least on earlier "fat" models. I have no idea if the drives are still paired with unique keys on the newer "slim" systems, though I do not know why it would be done another way. This also means that physically dumping the drive's firmware would lead nowhere with it being stored in an encrypted form. The only way to get a plain version of it would be to dump the drive controller's ram at runtime. Beside although I am not entirely sure about this, it is very unlikely that a command exists to read the firmware from the drive, should it exist, the dumped binary would still be encrypted, thus connecting the drive to a computer (the ps3 slim bluray drive uses a regular SATA or PATA bus depending on the model) literally leads nowhere..
Finally once the authentication procedure is done, there is another protection which happens to be a per sector software cryptographic layer, which I have the algorithm for (but which I can't share because I wasn't the one that initially reversed it) that cryptographic layer is the very same used on playstation 3 master discs as on retail ones with the exception of the key being used, masterdiscs are identified through their special masterdisc sectors locted at offset 0x7000 (sector 14) on the disc. The encryption itself is done in sector ranges rather than files, where the key for each sector is defined by the address of the said sector in correlation with an initial static key, on retail discs, there is a per disc key located at offset 0x800 on the disc with a header composed of "Playstation3" and the discs' title id such as "PlayStation3 BCES-00141" I assume that this key is in encrypted format and likely decrypted through lv2 by appldr. The software cryptographic layer is done in such way that the disc sectors will be transparently decrypted so long as you are running a game or a playstation 3 application, this explains how easy it is to dump a disc's content whenever you can run playstation3 code on top of lv2.
The EBOOT.BIN as well as other self and sprx binaries themselves aren't tied to the disc's encryption, however their metadata may contain specific flags (in fact the ones on game discs always do) that will prevent them from being loaded from anywhere other than an authenticated disc (masterdiscs != authenticated discs) This would explain why someone on a debug console for instance can't just grab the games' binaries, put them on a masterdisc and hope for the game to just play. Other flags are being involved such as a "no debug" flag that will pretty much prevent you from loading your binary into sony's debugger.
Because the binaries on the discs still have to be signed as they are verified and decrypted by appldr, in the event that you would somewhat trick the drive into thinking that your disc is a genuine playstation 3 disc, you could still not have your own fself ("fake" secure elf, complied with Sony's sdk) in there and get it to run, thus this would never lead to homebrews no matter what some clueless people may claim about it.