User talk:Masterzorag

From PS3 Developer wiki
Revision as of 17:08, 13 October 2014 by Masterzorag (talk | contribs)
Jump to navigation Jump to search

SPU Problems on Linux > 3.2, OpenCL related

As far as I know, I'm the only coding OpenCL on the Cell here, if someone want to test something be warned that due some spufs changes that ppc-kernel-devs are (maybe) trying to fix, latest 3.3/3.4/3.5 branches falls into 'possible circular locking dependency detected' and slowdown runtime.

  • It's stable until 3.2 branch.
  • Even disabling lock debugging it slowdowns without warnings, it happens even with OpenCL samples from IBM.

http://permalink.gmane.org/gmane.linux.ports.ppc.embedded/50547

Latest tested kernels:

  • 3.2.55 works fine
# ./perlin
OpenCL took 22.496168 seconds to compute 1000 frames. Pixel Rate = 46.611316 Mpixels/sec, Frame Rate = 44.452015 frames/sec
Host code took 12.620616 seconds to compute 10 frames. Pixel Rate = 0.830844 Mpixels/sec, Frame Rate = 0.792354 frames/sec
OpenCL provided a 56.101182 speedup
  • 3.3.3/3.4.6/3.5.3 falls into 'possible circular locking dependency detected' and slowdown runtime

Here the slowdown effect: 

# ./perlin
OpenCL took 93.280273 seconds to compute 1000 frames. Pixel Rate = 11.241133 Mpixels/sec, Frame Rate = 10.720380 frames/sec
Host code took 12.948244 seconds to compute 10 frames. Pixel Rate = 0.809821 Mpixels/sec, Frame Rate = 0.772305 frames/sec
OpenCL provided a 13.881010 speedup

In this specific case time spent is 4x to do the same thing!
When program runs something is going weird, e.g. in my program I'm used to query an OpenCL builtin function to tell me how many available SPEs there are, and its reply 8.
Using spu_base.enum_shared=1 parameter it should reply 7, so seems that the issue is OpenCL related.

OtherOS region

OtherOS/OtherOS++ region is on HDD (ps3dd), we have new linux tools (ps3sed) and drivers.
To resize ps3da I've tried new ps3sed (manually), unsuccesfully: GameOS always detect corruption and redo its own things.

I've found a way to force resize on 4.46, no emer_init patch, no downgrading: GameOS respect standards.
I can now resize ps3da at arbitrary size.
Swapping HDD on pc is necessary to me to send a couple to SET MAX ADDRESS ata commands to get the job done: set the limit, left GameOS (partition and) format, then reset size back the same way.
On boot all regions are fine, plus empty space as tail, nice to fit a fouth region.

Here I've forced ps3da to use 1216709344 sectors, this left me about 16G for ps3dd.
After that GameOS do it own things, I've resetted ps3da to its real geometry (1250263728) and booted a new petitboot. 

root@ps3-linux:~# dmesg | grep ps3disk
[    3.220526] ps3disk_init:601: registered block device major 254
[    3.220549] ps3_system_bus_match:369: dev=6.0(sb_04), drv=6.0(ps3disk): match
[    3.220856] ps3disk sb_04: accessible region 0 start 0 size 1250263728
[    3.220952] ps3disk sb_04: accessible region 1 start 32 size 1212515008
[    3.221045] ps3disk sb_04: accessible region 2 start 1212515040 size 4194296
[    3.221051] ps3disk sb_04: ps3stor_probe_access:133: 3 accessible regions found
[    3.227341] ps3disk sb_04: ps3da is a SAMSUNG HM641JI (610480 MiB total, 610480 MiB region)
[    3.229035] ps3disk sb_04: ps3db is a SAMSUNG HM641JI (610480 MiB total, 592048 MiB region)
[    3.230008] ps3disk sb_04: ps3dc is a SAMSUNG HM641JI (610480 MiB total, 2047 MiB region)

root@ps3-linux:~# ps3sed print_region 3
   0                0       1250263728    1
   1               32       1212515008    8
   2       1212515040          4194296    8

root@ps3-linux:~# create_hdd_region.sh
INFO: device id 3
INFO: number of regions 3
INFO: total number of blocks 1250263728
INFO: last region start block 1212515040
INFO: last region number of blocks 4194296
INFO: new region start block 1216709344
INFO: new region number of blocks 33554376
INFO: new region id 3

root@ps3-linux:~# ps3sed print_region 3
   0                0       1250263728    1
   1               32       1212515008    8
   2       1212515040          4194296    8
   3       1216709344         33554376    1

root@ps3-linux:~# reboot && exit

Last number 1 is wrong, it says that last region has only one acl entry, we need to fix it at 8 entries:

  • manually with ps3sed
  • rebooting

Petitboot finally detect a new ps3dd device, the fourth region, of (33554376 * 512 =) 17179840512 bytes.
All of this with a 3.10.26 kernel and new tools: no vflash hacking involved, linux on vflash7 is deprecated.

Sometimes HDD is reported as second device (something buggy in my kernel?): 

root@ps3-linux:~# ps3sed print_device
     flash    1      512           491008        7
     cdrom    3     2048       2147483647        1
      disk    2      512       1250263728        4

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000010  00 00 00 00 0f ac e0 ff  00 00 00 00 de ad fa ce  |................|
00000020  00 00 00 00 00 00 00 03  00 00 00 00 00 00 00 02  |................|
00000030  00 00 00 00 00 00 00 20  00 00 00 00 48 45 82 c0  |....... ....HE..|
00000040  10 70 00 00 02 00 00 01  00 00 00 00 00 00 00 03  |.p..............|
00000050  10 70 00 00 01 00 00 01  00 00 00 00 00 00 00 03  |.p..............|
00000060  10 20 00 00 03 00 00 01  00 00 00 00 00 00 00 03  |. ..............|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000000c0  00 00 00 00 48 45 82 e0  00 00 00 00 00 3f ff f8  |....HE.......?..|
000000d0  10 70 00 00 02 00 00 01  00 00 00 00 00 00 00 03  |.p..............|
000000e0  10 70 00 00 01 00 00 01  00 00 00 00 00 00 00 03  |.p..............|
000000f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000150  00 00 00 00 48 85 82 e0  00 00 00 00 01 ff ff c8  |....H...........|
00000160  10 70 00 00 02 00 00 01  00 00 00 00 00 00 00 03  |.p..............|
00000170  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400

Another test on a different, not hacked PS3, running GameOS 4.11

ST96812AS is a 117210240 sectors of 512 bytes each, original PS3 60Gb HDD: here I've limited to 1/10, but I can force every size.

6Gb
60Gb

GameOS (must) accept it.

ps3vuart-tools

We miss some stuff from old ps3sm-utils, looking to port: temperature, get_fan_policy and set_fan_policy to new ps3vuart-tools.
We need to enable some sort of fan control on petitboot now. 

root@fedora_clone ~]# /home/ps3vuart-tools-2012-09-01/ps3sm/ps3sm get_fan_policy 0
0x01 0x01 0x48 0x00

[root@fedora_clone ~]# /home/ps3vuart-tools-2012-09-01/ps3sm/ps3sm temperature 0
01 00 00 00 3f 49 00 00

Updating the Real Time Clock with hwclock results in error: 

Mar 31 18:09:18 fedora_clone kernel: os_area_queue_work_handler: Could not update FLASH ROM

UPL.xml.pkg

tar -t -f update_files.tar

ls UPL.xml.unpkg/
-rw-r--r-- 1 0 0 2.8K Jun 27 13:40 content
-rw-r--r-- 1 0 0   64 Jun 27 13:40 info0
-rw-r--r-- 1 0 0   64 Jun 27 13:40 info1
...
-rwxr-xr-x 1 0 0  640 Jun 27 15:20 UPL.xml.pkg.spkg_hdr.1

UPL.xml.unpkg/content:                XML document text
UPL.xml.unpkg/info0:                  data
UPL.xml.unpkg/info1:                  data
UPL.xml.unpkg/UPL.xml.pkg.spkg_hdr.1: data

scetool -v -i update_files.untar/UPL.xml.pkg
[*] Using keyset [pkg 0x0000 03.55]
[*] Header decrypted.
[*] Data decrypted.
[*] SCE Header:
Magic           0x53434500 [OK]
Version         0x00000002
Key Revision    0x0000
Header Type     [PKG]
Metadata Offset 0x00000000
Header Length   0x0000000000000280
Data Length     0x0000000000000B9D  // 2973 bytes, content + info0 + info1
[*] Metadata Info:
Key 87 EE 46 44 60 DA DA EA 49 74 58 F9 02 1D 6D 11
IV  F4 9F 43 D8 D0 6A F0 FC 33 AF 5E 6E CF 2F 30 1E
[*] Metadata Header:
Signature Input Length 0x0000000000000250
unknown_0              0x00000001
Section Count          0x00000003
Key Count              0x00000014
Optional Header Size   0x00000000
unknown_1              0x00000000
unknown_2              0x00000000
[*] Metadata Section Headers:
Idx Offset   Size     Type Index Hashed SHA1 Encrypted Key IV Compressed
000 00000280 00000040 01   01    [YES]  00   [NO ]     --  -- [NO ]
001 000002C0 00000040 02   02    [YES]  06   [NO ]     --  -- [NO ]
002 00000300 0000016B 03   03    [YES]  0C   [YES]     12  13 [YES]
[*] SCE File Keys: n 14

hexdump -C UPL.xml.pkg 

00000280  00 00 00 03 00 00 00 04  00 00 00 00 00 00 00 0a  |................|
00000290  20 14 06 19 01 15 45 00  00 00 00 00 00 00 0b 1d  | .....E.........|
000002a0  00 00 00 00 00 00 01 6b  00 00 00 00 00 00 00 00  |.......k........|
000002b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

000002c0  00 00 00 00 00 00 00 03  00 00 00 00 00 00 00 40  |...............@|
000002d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 0b 1d  |................|
000002e0  00 00 00 00 00 00 00 01  00 00 00 00 00 00 00 01  |................|
000002f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

We got info0 + info1, 64bytes each
content is 2845bytes, total is data lenght 2973
info0, info1 are not encrypted

UPL.xml.pkg is decrypted using pkg keyset [0x0000 03.55], UPL.xml.pkg.spkg_hdr.1 is decrypted using spkg keyset for > 3.55: only Metadata Info and Signature changes.

I think that on 3.55 all .spkg_hdr.1 files are not involved in updating, on 3.56 all of them are used as in overlayfs, so all .spkg_hdr.1 files are readed as metadata headers, decrypted by newer spkg keyset to get the same SCE keys to get rest of data decrypted.

.spkg_hdr.1 decryption

Verifing decryption of signed pkg headers with openssl:

  • strip metadata info
# dd if=CORE_OS_PACKAGE.pkg.spkg_hdr.1 of=metainfo.crypt skip=32 count=64 bs=1 
64+0 records in
64+0 records out
64 bytes (64 B) copied, 0.0021009 s, 30.5 kB/s
# hexdump -C metainfo.crypt 
00000000  d7 f9 82 9e 75 0a 3f 20  8b 6f e7 41 b1 bb 52 15  |....u.? .o.A..R.|
00000010  e1 8f d2 86 43 b5 4f 56  4c 42 a0 10 e1 1a 25 38  |....C.OVLB....%8|
00000020  9c 28 c7 fd 38 31 24 3b  1b 2b 9f 3f dc 72 4f c4  |.(..81$;.+.?.rO.|
00000030  95 34 b8 0a af 25 a1 05  b6 8f ce 2c 88 e9 2b 7b  |.4...%.....,..+{|
  • verify metadata info decryption with standard tool
# openssl enc -d -aes-256-cbc -in metainfo.crypt -K erk -iv riv | hexdump -C
00000000  7c f2 9a 4b 96 de 5f 75  a1 32 87 c0 42 ec 8f cf  ||..K.._u.2..B...|   Key
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  6f 85 6a 60 2a 8d b4 3f  2a 81 1b 1a 9c a3 02 f6  |o.j`*..?*.......|   IV
00000030
  • strip rest of crypted metadata
# dd if=CORE_OS_PACKAGE.pkg.spkg_hdr.1 of=metarest.crypt skip=96 bs=1
544+0 records in
544+0 records out
544 bytes (544 B) copied, 0.00626423 s, 86.8 kB/s
  • decrypt rest of metadata
# openssl enc -d -aes-128-ctr -in metarest.crypt -K Key -iv IV | hexdump -C
00000000  00 00 00 00 00 00 02 50  00 00 00 01 00 00 00 03  |.......P........| mh
00000010  00 00 00 14 00 00 00 00  00 00 00 00 00 00 00 00  |................|

00000020  00 00 00 00 00 00 02 80  00 00 00 00 00 00 00 40  |...............@| m_sh0
00000030  00 00 00 01 00 00 00 01  00 00 00 02 00 00 00 00  |................|
00000040  00 00 00 01 ff ff ff ff  ff ff ff ff 00 00 00 01  |................|

00000050  00 00 00 00 00 00 02 c0  00 00 00 00 00 00 00 40  |...............@| m_sh1
00000060  00 00 00 02 00 00 00 02  00 00 00 02 00 00 00 06  |................|
00000070  00 00 00 01 ff ff ff ff  ff ff ff ff 00 00 00 01  |................|

00000080  00 00 00 00 00 00 03 00  00 00 00 00 00 5a 01 74  |.............Z.t| m_sh2
00000090  00 00 00 03 00 00 00 03  00 00 00 02 00 00 00 0c  |................|
000000a0  00 00 00 03 00 00 00 12  00 00 00 13 00 00 00 02  |................|

000000b0  fd 5e fe 3f a0 42 fe 31  d6 61 83 26 98 07 ce e8  |.^.?.B.1.a.&....| h0 0x20, HMAC-SHA1
000000c0  09 a7 65 7a 00 00 00 00  00 00 00 00 00 00 00 00  |..ez............|
000000d0  6a 49 08 b2 87 ee 02 65  8c 73 5a 4c 54 f8 bf 5d  |jI.....e.sZLT..]| k0 0x40, HMAC key
000000e0  b6 3a d4 ef d3 94 74 f5  f3 a3 f2 ad af 1c 45 4e  |.:....t.......EN|
000000f0  08 aa 2b 61 fb 91 52 9d  69 a8 44 b0 a0 dd 39 1c  |..+a..R.i.D...9.|
00000100  bc 4d a0 51 ed f5 30 16  d1 35 8c b1 d1 09 54 d0  |.M.Q..0..5....T.|
00000110  60 32 ff 1c 3d 1a 9d 03  61 28 d8 ad 1f dc 34 57  |`2..=...a(....4W| h1 0x20, HMAC-SHA1
00000120  05 2a 1e f6 00 00 00 00  00 00 00 00 00 00 00 00  |.*..............|
00000130  6a 49 08 b2 87 ee 02 65  8c 73 5a 4c 54 f8 bf 5d  |jI.....e.sZLT..]| k1 0x40, HMAC key
00000140  b6 3a d4 ef d3 94 74 f5  f3 a3 f2 ad af 1c 45 4e  |.:....t.......EN|
00000150  08 aa 2b 61 fb 91 52 9d  69 a8 44 b0 a0 dd 39 1c  |..+a..R.i.D...9.|
00000160  bc 4d a0 51 ed f5 30 16  d1 35 8c b1 d1 09 54 d0  |.M.Q..0..5....T.|
00000170  3c 37 73 07 73 d5 69 69  52 67 97 3b e9 20 70 a9  |<7s.s.iiRg.;. p.| h2 0x20, HMAC-SHA1
00000180  53 a7 11 6d 00 00 00 00  00 00 00 00 00 00 00 00  |S..m............|
00000190  6a 49 08 b2 87 ee 02 65  8c 73 5a 4c 54 f8 bf 5d  |jI.....e.sZLT..]| k2 0x40, HMAC key
000001a0  b6 3a d4 ef d3 94 74 f5  f3 a3 f2 ad af 1c 45 4e  |.:....t.......EN|
000001b0  08 aa 2b 61 fb 91 52 9d  69 a8 44 b0 a0 dd 39 1c  |..+a..R.i.D...9.|
000001c0  bc 4d a0 51 ed f5 30 16  d1 35 8c b1 d1 09 54 d0  |.M.Q..0..5....T.|
000001d0  35 53 b4 b5 44 28 fe 09  5a 04 e5 38 e8 38 f4 a6  |5S..D(..Z..8.8..| Key
000001e0  5d 24 fa b1 e0 55 f4 58  15 22 c2 73 00 00 00 00  |]$...U.X.".s....| IV

000001f0  00 c9 57 08 f1 6f e6 75  39 6f 2a 12 51 48 5f 8c  |..W..o.u9o*.QH_.| r, s
00000200  97 53 e3 e2 a7 00 b7 47  fa c2 0b 01 2b df 5a 34  |.S.....G....+.Z4|
00000210  f7 a8 d0 21 d2 f8 94 4b  6e 30 00 00 00 00 00 00  |...!...Kn0......|

verify ecdsa signatures

  • get signature
  • join sceh + decrypted metadata info + decrypted rest of metadata until signature
# ls -l metadata.bin
-rw-r--r-- 1 root root 592 Jul 11  metadata.bin
  • compute digest of the whole decrypted metadata until signature
# sha1sum metadata.bin 
70942107d35df85091bf4949d7fa46421e27d056  metadata.bin
  • verify signature against computed digest
ecdsa_verify(digest, r, s) == 1

collecting spkg_hdr.1 signatures

I've started collecting into an SQLite database all publicily available ECDSA signatures of spkg.hdr.1 files, to research.

  • get spkg_hdr.tar file from PS3UPDAT.PUP, here I've just added an argument to pupunpack to extract only the needed section:
ps3tools # ./pupunpack ../PS3\ 3.56\ OFW/PS3UPDAT.PUP ../PS3\ 3.56\ OFW/unpacked 7
sections:    9
hdr size:    00000000_00000290
data size:   00000000_0b0f3d3d
header hmac: OK
extracting only section 7
section 7: unpacking spkg_hdr.tar       (00000000_00011800 bytes; hmac: OK)...
  • untar spkg.hdr.1 files
# cd ../PS3\ 3.56\ OFW/unpacked/
unpacked # tar -xf spkg_hdr.tar
  • strip signature data off spkg.hdr.1 files into spkg.hdr.1_sigdata files, here I've added a function to sceverify
unpacked # for file in *hdr.1; do ../../ps3tools/sceverify $file; done
...
hash:   c72f26e67e03b743a8c830a541e82bfb2e9f6c19
r:      0056af2dad5647288c75e76dc92ed875de86876d7e
s:      00d7de25130de0f6dc0602e590b42bc5f21ab2a3e6
Signature: ecdsa_verify(hash, r, s) : OK
Collecting digest + signature for file: UPL.xml.pkg.spkg_hdr.1
exported: UPL.xml.pkg.spkg_hdr.1_sigdata, recheck ecdsa signature: OK

unpacked # for file in *1_sigdata; do ls -h $file; done
BDIT_FIRMWARE_PACKAGE.pkg.spkg_hdr.1_sigdata
BDPT_FIRMWARE_PACKAGE_301R.pkg.spkg_hdr.1_sigdata
BDPT_FIRMWARE_PACKAGE_302R.pkg.spkg_hdr.1_sigdata
...
  • collect sigdata into an sqlite database, I've coded an app to do the job:
unpacked # for file in *1_sigdata; do ../../ps3tools/collector 1 $file 3.56; done
Connection successful
BDIT_FIRMWARE_PACKAGE.pkg.spkg_hdr.1_sigdata
BDIT_FIRMWARE_PACKAGE.pkg.spkg_hdr.1 3.56
dig = 749e7ba6fba9ff4c8ad83d1fd62feec4ea9ad3f2
sig = 00574c57e9ceaac1285b3f6d7b06feaa7daa180edf002b0cc8548f42c523afa39c6984e4b67b9639afbc
...
BDPT_FIRMWARE_PACKAGE_301R.pkg.spkg_hdr.1_sigdata
BDPT_FIRMWARE_PACKAGE_301R.pkg.spkg_hdr.1 3.56
dig = cee76eb85bf899952acc8723ef3e35fdc9b0da23
sig = 0001bd4af80d5e8f190baea58e6613e5672d1c29a8000dcfad27f4edcef9f0b9f000759d89ebb050d871
...
  • collector can also export to STDOUT, we can filter, sort...
# ./ps3tools/collector 2 | head -12
Connection successful
Export to STDOUT
BDIT_FIRMWARE_PACKAGE.pkg.spkg_hdr.1 3.57
dig = fcde1eaf97e24c9d0a5fe2af312e5b1ce1e6ab14
r = 003798ace6c2097c02e2f9214bfc2bbf5c7bd347ae
s = 002b1a77b56e0995f80761fa779db5b09e269ebf96
BDPT_FIRMWARE_PACKAGE_301R.pkg.spkg_hdr.1 3.57
dig = 0d953e1f78195b960a145a0c20753269a2352204
r = 00b2a103abbf09db35eb5a2580554446255fee4504
s = 0061292680dccc79e92373f002e669cf238c520a70

# ./ps3tools/collector 2 | grep "r =" | sort | head
r = 000119f11cc9076d039c1c0a625afe3dc5f9e38af9
r = 000171cfd17643b35e3f7a9fa1d64398413c35a949
r = 0001bd4af80d5e8f190baea58e6613e5672d1c29a8
r = 00023c6b278121e57ae6c836c0a188344dc676f9b7
r = 0003c42810e1b9f3cc4a93e71b101359116c06e68a
r = 000418380d9e2d2f4a11e88bcb42af8f17b042cd31
r = 00049a30c5e2d38ad19765e6c02e94377bf93444f9
r = 0004fa6eb1641706c20a14ede5dc0fe497b01f4dee
r = 00051b73b3eea4904f899ffc10f445ee080f2f3912
r = 0005639f0f3dd5585ed5e64df0421b274e842a2a2f
  • check for the same r (they just failed in this, but watch how is simple!)
# ./ps3tools/collector 2 | grep "r =" | sort | wc -l
298 lines
# ./ps3tools/collector 2 | grep "r =" | sort | uniq | wc -l
298 lines (no same r!)
  • database can be accessed via SQL query too, but it contains binary blobs...
# sqlite3 /tmp/collector_db.sqlite3
SQLite version 3.8.5 2014-06-04 14:06:34
Enter ".help" for usage hints.
sqlite> SELECT * FROM spkg_hdr WHERE name = "SYS_CON_FIRMWARE_01010303.pkg.spkg_hdr.1 3.70";
SYS_CON_FIRMWARE_01010303.pkg.spkg_hdr.1 3.70|rØxY£‰õû“Ùˆ©Zö£Ž|
  • we can just use a browser plugin to deal with database
spkg_hdr SQLite table
Retrieved from "http://www.psdevwiki.com/ps3/index.php?title=User_talk:Masterzorag&oldid=30487"