Hypervisor Reverse Engineering

This is a copy of the page from 22nd of February 2011, right before ps3wiki.lan.st went down.

---

Event Notification

• Event Notfication is used e.g. to notify a LPAR about some event, e.g. device interrupt or notify a LPAR about destruction of another LPAR.
• For example Process 9 is notified through Event Notification when LPAR 2 is destructed.
• During LPAR construction, Process 9 creates an Outlet object with syscall 0x1001A and then passes the outlet ID to the syscall 0x10009 that constructs the LINUX LPAR. In this way Process 9 is notified when LINUX LPAR is destructed.

Outlet class

This is the base Outlet class. There are different types of Outlet and they derive from this base class.

vtable

0x00357DC0 (3.15)

Member variables

offset 0x30 - type (8 bytes)

offset 0x38 - pointer to LPAR that owns this Outlet object

offset 0x48 - outlet id (8 bytes)

offset 0x90 - VIRQ assigned to this Outlet object (4 bytes)

Event Receive Port class

• This type of Outlet is created e.g. in lv1_construct_event_receive_port and in syscall 0x1001A.
• HV calls lv1_connect_irq_plug and lv1_connect_irq_plug_ext assigns a VIRQ to Event Receive Port object.

vtable

0x00357E88

VUART Outlet

• HV supports only one VUART Outlet per LPAR
• lv1_configure_virtual_uart_irq constructs a VUART Outlet object and passes the address of LPAR's VUART IRQ Bitmap to HV

vtable

0x00357DC0

VUART IRQ Bitmap

• At address 0x38(LPAR ptr) + 0x158 is the VUART IRQ Bitmap owned by HV for LPAR (4 * 8 bytes = 256 bits)
• At address 0x38(LPAR ptr) + 0x150 is stored the physical address of LPAR's VUART IRQ Bitmap that was passed to lv1_configure_virtual_uart_irq
• When a VUART interrupt is generated by HV then first the VUART IRQ Bitmap owned by HV is updated and then this bitmap is copied to LPAR's VUART IRQ Bitmap, so VUART IRQ Bitmap is stored twice, once in HV and once in LPAR, just like IRQ State Bitmap.
• VUART IRQ Bitmap is not allowed to cross page boundary of LPAR memory region where it is stored. HV checks it and makes sure that it doesn't happen.
• GameOS 3.41 VUART IRQ bitmap is at address 0x80000000003556E8 and of size 32 bytes (256 bits, each bit corresponds to a VUART port).
• GameOS 3.15 VUART IRQ bitmap is at address 0x8000000000354768.

Logical PPE

• Logical PPE is used for interrupt management of LPAR.
• A Logical PPE object is created in syscall 0x10005. It' used e.g. in Process 9 during LPAR construction.
• syscall 0x10007 activates a Logical PPE object
• 0x67F0(HSPRG0) - pointer to currently active Logical PPE object (in HV dump it points to Linux PPE object naturally because the dump was made on Linux, so Linux LPAR was active at that time)
• E.g. lv1_get_logical_ppe_id, lv1_start_ppe_periodic_tracer and lv1_set_ppe_periodic_tracer_frequency grab the currently active Logical PPE object

vtable

0x00357DF0 (3.15)

Member variables

offset 0x90 - pointer to an object that contains VIRQ-Outlet mapping table for thread 0

offset 0x98 - pointer to an object that contains VIRQ-Outlet mapping table for thread 1

Objects

Here is the list of Logical PPE objects i found in HV 3.15.

Address in HV dump	LPAR id	PPE id
0x0069C7F0	1	1
0x007A8900	2	1

Virtual IRQ - Outlet Mapping

• HV maintains 2 tables per PPE that map a VIRQ to an Outlet object.
• The table has 256 entries and is indexed by VIRQ.
• Each entry is a pointer to Outlet object.
• Each Logical PPE object has 2 tables, one for each thread of Cell CPU.

LPAR 1 PPE 1 Thread 0

0x0069C990 (3.15) - address of VIRQ-Outlet table for LPAR 1 PPE 1 Thread 0 (not empty)

VIRQ	Address of Outlet object in HV dump	Description
58	0x00090D10	-
59	0x006BAC50	-
60	0x006B3ED0	FLASH storage device / Storage device notification for LPAR 1
61	0x00697E70	VUART interrupts
62	0x001C8F20	-

LPAR 1 PPE 1 Thread 1

0x0069D9B0 (3.15) - address of VIRQ-Outlet table for LPAR 1 PPE 1 Thread 1 (empty)

LPAR 2 PPE 1 Thread 0

0x000A06B0 (3.15) - address of VIRQ-Outlet table for LPAR 2 PPE 1 Thread 0 (not empty)

VIRQ	Address of Outlet object in HV dump	Description
20	0x003AA210	-
21	0x003AFEC0	-
22	0x001FC010	-
23	0x003A8E50	-
24	0x001FFED0	SPE 0 Class 0 Interrupt
25	0x003AE160	SPE 0 Class 1 Interrupt
26	0x003AE350	SPE 0 Class 2 Interrupt
27	0x003AB100	SPE 1 Class 0 Interrupt
28	0x003AB2F0	SPE 1 Class 1 Interrupt
29	0x003AB4E0	SPE 1 Class 2 Interrupt
30	0x003AA6A0	SPE 2 Class 0 Interrupt
31	0x003AA890	SPE 2 Class 1 Interrupt
32	0x003AAA80	SPE 2 Class 2 Interrupt
33	0x003B44A0	SPE 3 Class 0 Interrupt
34	0x003B4690	SPE 3 Class 1 Interrupt
35	0x003B4AD0	SPE 3 Class 2 Interrupt
36	0x003B5300	SPE 4 Class 0 Interrupt
37	0x003B54F0	SPE 4 Class 1 Interrupt
38	0x003B56E0	SPE 4 Class 2 Interrupt
39	0x003AE7C0	SPE 5 Class 0 Interrupt
40	0x003AE9B0	SPE 5 Class 1 Interrupt
41	0x003AEBA0	SPE 5 Class 2 Interrupt
42	0x003B2040	Storage device notification for LPAR 2
43	0x003AEE30	VUART interrupts
44	0x001FEAA0	-
45	0x001FEED0	HDD storage device
46	0x003B5E20	-
47	0x003B7040	-
48	0x003B9B40	-
49	0x003B3A40	-
50	0x003BACA0	Gelic device
51	0x003BAE10	UNKNOWN storage device
52	0x003B8350	-

LPAR 2 PPE 1 Thread 1

0x007A89E0 (3.15) - address of VIRQ-Outlet table for LPAR 2 PPE 1 Thread 1 (not empty)

VIRQ	Address of Outlet object in HV dump	Description
16	0x003B2480	-
17	0x003B2590	-
18	0x003B26A0	-
19	0x003B27B0	-

IRQ State Bitmap

• There is one IRQ State Bitmap (256 bits = 32 bytes) per thread of Logical PPE
• HSPRG0 value is per thread, so there are 2 HSPRG0 values in HV dump !!!
• The IRQ State Bitmap of a thread is stored at -0x68E0(HSPRG0)
• When an Event or Interrupt happens then the bitmap at 0x68E0(HSPRG0) is updated
• The physical address of LPAR's IRQ State Bitmap of thread is stored at offset -0x68C0(HSPRG0)
• The address of LPAR's IRQ State Bitmap is passed to Hypervisor through HV call lv1_configure_irq_state_bitmap
• lv1_detect_pending_interrupts returns value of current IRQ State Bitmap.
• The IRQ State Bitmap is updated if an Outlet object is assigned to VIRQ and when Outlet generates an event
• After IRQ State Bitmap update, it's copied to LPAR's IRQ State Bitmap and a hardware interrupt is generated so that LPAR can read it's IRQ State Bitmap and handle interrupts.
• So, IRQ State Bitmap is stored twice, once in HV and once in LPAR, just like VUART IRQ Bitmap.
• GameOS IRQ state bitmap is stored at address SPRG0 + 0x1C0 and of size 64 bytes (256 bits state + 256 bits mask) per thread of Cell CPU. So there are 2 IRQ state bitmaps.

0x8941FC0 - physical address of LPAR's IRQ State Bitmap for Thread 0 of LINUX LPAR

0x8948FC0 - physical address of LPAR's IRQ State Bitmap for Thread 1 of LINUX LPAR

System Controller (SC or SYSCON)

• Data received from SC is sent to a VUART
• lv1_get_rtc and syscall 0x10036 communicate with SC VUART 4.

VUART Table

• Address of SC VUART Table - 0x00610410 (3.15).
• There are 5 VUARTs for SC in HV 3.15

Here is the SC VUART table from HV 3.15:

Index	Address of VUART object in HV dump	Description
0	0x0060FD20	This VUART is connected with the VUART 0 (/dev/sc0) of LPAR 1
1	0x0060FE20	This VUART is connected with the VUART 1 (/dev/sc1) of LPAR 1
2	0x0060FF20	This VUART is not connected to some peer VUART but i guess that it should be connected to VUART 2 (/dev/sc2) of LPAR1
3	0x006124E0	This VUART is connected with the VUART 3 (/dev/sc3) of LPAR 1
4	0x00612DF0	lv1_get_rtc and syscall 0x10036 communicate with this VUART.

Interrupt Handling

spider_sc_interrupt_handler - 0x0020A68C (3.15)

Methods

sc_vuart_4_get_peer_vuart - 0x002ED384 (3.15)

sc_send - 0x0020A908 (3.15)

lv1_get_rtc

• lv1_get_rtc communicates with SC VUART 4.
• 20 bytes are written to the peer VUART of SC VUART 4.
• After a request is sent to SC VUART 4, lv1_get_rtc busy waits until SC VUART 4 receive data buffer is not empty.
• When SC VUART 4 receive data buffer is not empty, lv1_get_rtc reads 24 bytes from the VUART.