[Feature Request] Move PCIe / VGA init before DRAM training

[madscientist159]

Many of these systems take some minutes to finish IPL, during which time any discrete GPUs attached to the system remain blank / dark. Moving PCIe initialization before DRAM training would allow early init of the discrete GPUs in the same manner as x86 machines.

[madscientist159]

The thought is that once the PCIe devices are online, with VGA initialized from the x86 oprom via emulation, that a process on the host can copy frames from the BMC startup screen framebuffer to the discrete GPU framebuffer every x milliseconds.

[dcrowell77]

There is currently zero support (or even domain knowledge) for PCI device configuration in the Hostboot code. The PCI master logic isn't even configured completely until istep 14.3 proc_pcie_config either. There is a placeholder istep 6.4 host_init_bmc_pcie where one could theoretically add code to access the VGA through the BMC. Note that at this point any work in this area would need to come from external contributions.

[madscientist159]

@dcrowell77 Understood. This feature request issue is here simply to track when/if this can be accomplished.

Is there a technical barrier to moving proc_pcie_config and associated logic earlier in the IPL process, e.g. to istep 10 or earlier? The goal isn't so much to talk to the BMC VGA as it is to get basic early framebuffer support on an attached DGPU -- even PCIe 1 would work for that goal.

This is something Raptor may want to work on over time, especially if P10 ends up using a similar IPL process. Main concern is whether there is a technical barrier preventing the concept from working in the first place (e.g. PCIe setup requiring DRAM trained first, etc.).

[dcrowell77]

I'm pretty sure that we would have trouble moving anything before the SMP is built in 10.1 because of the secureboot concerns. I suppose one could change things around to only initialize the master proc's logic to avoid that. Obviously that would mean that you could only use VGA off that proc. Step 8.12 might be the next barrier since we're setting some processor voltages there. I'm not aware of any specific issues beyond that but I'll need to go poke some other people. The biggest hurdle might just be adding duplicate paths for master/slave which adds to the testing matrix.

[madscientist159]

On x86 / legacy BIOS machines, the master processor (core 0 on an SMP system) is the only thing drawing to early framebuffer in the first place, so that seems reasonable enough. If istep 8,4 is the earliest we can get the PECs online, that seems reasonable as well.

[ghost]

Timothy Pearson <notifications@github.com> writes:

The thought is that once the PCIe devices are online, with VGA initialized from the x86 oprom via emulation, that a process on the host can copy frames from the BMC startup screen framebuffer to the discrete GPU framebuffer every x milliseconds.

This is basically how it's done on (some) ARM systems... https://www.youtube.com/watch?v=uxvAH1Q4Mx0 http://events17.linuxfoundation.org/sites/events/files/slides/QEMU%20in%20UEFI.pdf Which is all a bit crazy, but pretty much what we'd have to do to satisfy the "insert random graphics card, have it be basically functional until a Linux is booted with appropriate drivers" brief.

…

-- Stewart Smith OPAL Architect, IBM.

[madscientist159]

Long term I still want to have the option to have JITable firmware loaded into some standard location on each card. This doesn't exist yet, but the idea would be to expose some standard set of basic, minimal opcodes that can be easily interpreted by any platform regardless of architecture, then have the card firmware compiled to / written in that scripting language. Kind of like ATOMBIOS in some ways.

I also want to see the linear framebuffer passed up through skiboot into Linux via the device tree...had some initial ideas about cards that implement the aforementioned JIT-type firmware being able to advertise to hostboot/skiboot what the card is, what address ranges it has, etc. and that gets dynamically added to the DTB passed on to the petitboot and guest kernels. It might be worth building a sort of technical demo infrastructure for both POWER and coreboot , in order to show card vendors they can target one (virtual) architecture instead of forcing everyone non-x86 into a suboptimal experience with their hardware.

I wonder if OPF might be the right place to work on such a standard.