Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Booting into linux on simulation #12

Open
troibe opened this issue Jun 8, 2021 · 10 comments
Open

Booting into linux on simulation #12

troibe opened this issue Jun 8, 2021 · 10 comments

Comments

@troibe
Copy link
Contributor

troibe commented Jun 8, 2021

At the moment, I'm not aware of a good way to tell the firmware/BIOS to simply jump to the first RAM address and start executing the side-loaded boot.bin which is now located there. Although that should be an easy fix.

For linux-on-litex-blackparrot we are setting the ROM_BOOT_ADDRESS in litex_sim.py to 0x80000000.
However even with that change I didn't manage to boot into linux with rocket.
Do you have exact steps that you followed to accomplish that?
Using a method other than side loading would be fine as well of course.

I just want to make sure that I'm having a reproducible boot into linux in simulation before I further attempt it on my ulx3s and arty7.
For the ulx3s so far I'm reliably getting into the bios. On the arty7 I encounter some memory errors with newer versions of litex.

@gsomlo
Copy link
Collaborator

gsomlo commented Jun 8, 2021

Last time I tried this was a very long time ago. Make sure you have libevent-devel and json-c-devel installed. The command line where you load boot.bin over tftp should be:

litex/litex/tools/litex_sim.py \
  --threads 8 --opt-level Ofast \
  --with-ethernet --with-sdram \
  --cpu-type rocket --cpu-variant linux

However, simulating 64-bit rocket running the full tftp download sequence is about an order of magnitude less entertaining than watching paint dry. So, I remember replacing --with-sdram with --ram-init boot.bin, to bypass tftp. Hope it all still works...

@troibe
Copy link
Contributor Author

troibe commented Jun 8, 2021

Hm I'm still not getting it to start unfortunately.
This is the csr.csv that Litex generates for me:

#--------------------------------------------------------------------------------
# Auto-generated by Migen (3ffd64c) & LiteX (bd146351) on 2021-06-08 21:07:56
#--------------------------------------------------------------------------------
csr_base,ctrl,0x12000000,,
csr_base,ethmac,0x12000800,,
csr_base,ethphy,0x12001000,,
csr_base,identifier_mem,0x12001800,,
csr_base,timer0,0x12002000,,
csr_base,uart,0x12002800,,
csr_register,ctrl_reset,0x12000000,1,rw
csr_register,ctrl_scratch,0x12000004,1,rw
csr_register,ctrl_bus_errors,0x12000008,1,ro
csr_register,ethmac_sram_writer_slot,0x12000800,1,ro
csr_register,ethmac_sram_writer_length,0x12000804,1,ro
csr_register,ethmac_sram_writer_errors,0x12000808,1,ro
csr_register,ethmac_sram_writer_ev_status,0x1200080c,1,ro
csr_register,ethmac_sram_writer_ev_pending,0x12000810,1,ro
csr_register,ethmac_sram_writer_ev_enable,0x12000814,1,rw
csr_register,ethmac_sram_reader_start,0x12000818,1,rw
csr_register,ethmac_sram_reader_ready,0x1200081c,1,ro
csr_register,ethmac_sram_reader_level,0x12000820,1,ro
csr_register,ethmac_sram_reader_slot,0x12000824,1,rw
csr_register,ethmac_sram_reader_length,0x12000828,1,rw
csr_register,ethmac_sram_reader_ev_status,0x1200082c,1,ro
csr_register,ethmac_sram_reader_ev_pending,0x12000830,1,ro
csr_register,ethmac_sram_reader_ev_enable,0x12000834,1,rw
csr_register,ethmac_preamble_crc,0x12000838,1,ro
csr_register,ethphy_crg_reset,0x12001000,1,rw
csr_register,timer0_load,0x12002000,1,rw
csr_register,timer0_reload,0x12002004,1,rw
csr_register,timer0_en,0x12002008,1,rw
csr_register,timer0_update_value,0x1200200c,1,rw
csr_register,timer0_value,0x12002010,1,ro
csr_register,timer0_ev_status,0x12002014,1,ro
csr_register,timer0_ev_pending,0x12002018,1,ro
csr_register,timer0_ev_enable,0x1200201c,1,rw
csr_register,uart_rxtx,0x12002800,1,rw
csr_register,uart_txfull,0x12002804,1,ro
csr_register,uart_rxempty,0x12002808,1,ro
csr_register,uart_ev_status,0x1200280c,1,ro
csr_register,uart_ev_pending,0x12002810,1,ro
csr_register,uart_ev_enable,0x12002814,1,rw
csr_register,uart_txempty,0x12002818,1,ro
csr_register,uart_rxfull,0x1200281c,1,ro
constant,config_clock_frequency,1000000,,
constant,config_cpu_has_interrupt,None,,
constant,config_cpu_reset_addr,268435456,,
constant,config_cpu_type_rocket,None,,
constant,config_cpu_variant_linux,None,,
constant,config_cpu_human_name,rocketrv64[imac],,
constant,config_cpu_nop,nop,,
constant,config_with_build_time,None,,
constant,rom_boot_address,2147483648,,
constant,localip1,192,,
constant,localip2,168,,
constant,localip3,1,,
constant,localip4,50,,
constant,remoteip1,192,,
constant,remoteip2,168,,
constant,remoteip3,1,,
constant,remoteip4,100,,
constant,config_csr_data_width,32,,
constant,config_csr_alignment,32,,
constant,config_bus_standard,wishbone,,
constant,config_bus_data_width,32,,
constant,config_bus_address_width,32,,
constant,config_cpu_has_dma_bus,None,,
constant,ethmac_rx_slots,2,,
constant,ethmac_tx_slots,2,,
constant,ethmac_slot_size,2048,,
constant,ethmac_interrupt,2,,
constant,timer0_interrupt,1,,
constant,uart_interrupt,0,,
memory_region,rom,0x10000000,131072,cached
memory_region,sram,0x11000000,8192,cached
memory_region,main_ram,0x80000000,268435456,cached
memory_region,ethmac,0x30000000,8192,io
memory_region,csr,0x12000000,65536,io

This is the sim.dts that I'm using right now:

/dts-v1/;

/ {
	#address-cells = <1>;
	#size-cells = <1>;
	compatible = "freechips,rocketchip-unknown-dev";
	model = "freechips,rocketchip-unknown";
	chosen {
		bootargs = "earlycon=sbi console=liteuart swiotlb=noforce";
	};
	L13: cpus {
		#address-cells = <1>;
		#size-cells = <0>;
		timebase-frequency = <1000000>;
		L6: cpu@0 {
			clock-frequency = <50000000>;
			compatible = "sifive,rocket0", "riscv";
			d-cache-block-size = <64>;
			d-cache-sets = <64>;
			d-cache-size = <4096>;
			d-tlb-sets = <1>;
			d-tlb-size = <4>;
			device_type = "cpu";
			hardware-exec-breakpoint-count = <1>;
			i-cache-block-size = <64>;
			i-cache-sets = <64>;
			i-cache-size = <4096>;
			i-tlb-sets = <1>;
			i-tlb-size = <4>;
			mmu-type = "riscv,sv39";
			next-level-cache = <&L8>;
			reg = <0x0>;
			/* FIXME: blatant lie! BBL should s/imac/imafdc/;
			   https://github.com/riscv/riscv-pk/issues/166 */
			riscv,isa = "rv64imafdc";
			riscv,pmpregions = <8>;
			status = "okay";
			tlb-split;
			L4: interrupt-controller {
				#interrupt-cells = <1>;
				compatible = "riscv,cpu-intc";
				interrupt-controller;
			};
		};
	};
	L8: memory@80000000 {
		device_type = "memory";
		reg = <0x80000000 0x04000000>; /* 64MB (simulation default) */
	};
	L12: soc {
		#address-cells = <1>;
		#size-cells = <1>;
		compatible = "freechips,rocketchip-unknown-soc", "simple-bus";
		ranges;
		L2: clint@2000000 {
			compatible = "riscv,clint0";
			interrupts-extended = <&L4 3 &L4 7>;
			reg = <0x2000000 0x10000>;
			reg-names = "control";
		};
		L3: debug-controller@0 {
			compatible = "sifive,debug-013", "riscv,debug-013";
			interrupts-extended = <&L4 0x10>;
			reg = <0x0 0x1000>;
			reg-names = "control";
		};
		L0: error-device@3000 {
			compatible = "sifive,error0";
			reg = <0x3000 0x1000>;
		};
		L7: external-interrupts {
			interrupt-parent = <&L1>;
			interrupts = <1 2 3 4>;
		};
		L1: interrupt-controller@c000000 {
			#interrupt-cells = <1>;
			compatible = "riscv,plic0";
			interrupt-controller;
			interrupts-extended = <&L4 11 &L4 9>;
			reg = <0xc000000 0x4000000>;
			reg-names = "control";
			riscv,max-priority = <7>;
			riscv,ndev = <4>;
		};
		L10: rom@10000 {
			compatible = "sifive,rom0";
			reg = <0x10000 0x10000>;
			reg-names = "mem";
		};
		soc_ctrl0: soc_controller@12000000 {
			compatible = "litex,soc-controller";
			reg = <12000000 0xc>;
		};
		liteuart0: serial@12002800 {
			compatible = "litex,liteuart";
			reg = <0x12002800 0x100>;
			interrupt-parent = <&L1>;
			interrupts = <1>;
		};
		mac0: mac@12000800 {
			compatible = "litex,liteeth";
			reg = <0x12000800 0x7c>,
				<0x12001000 0x0a>,
				<0x30000000 0x2000>;
			tx-fifo-depth = <2>;
			rx-fifo-depth = <2>;
			interrupt-parent = <&L1>;
			interrupts = <3>;
		};
	};
};

Console output before things get stuck:


        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2021 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs

 BIOS built on Jun  8 2021 21:07:54
 BIOS CRC passed (e6fbe0f2)

 Migen git sha1: 3ffd64c
 LiteX git sha1: bd146351

--=============== SoC ==================--
CPU:            RocketRV64[imac] @ 1MHz
BUS:            WISHBONE 32-bit @ 4GiB
CSR:            32-bit data
ROM:            128KiB
SRAM:           8KiB
MAIN-RAM:       262144KiB 

--========== Initialization ============--
Ethernet init...

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
Executing booted program at 0x80000000

--============= Liftoff! ===============--

Maybe I'm not waiting long enough but for blackparrot and vexriscv at least the BBL loader line gets shown pretty much instantly.

@troibe
Copy link
Contributor Author

troibe commented Jun 8, 2021

The vmlinux file is the same that I'm using for the arty where I'm now getting some console output.

@troibe
Copy link
Contributor Author

troibe commented Jun 24, 2021

I revisited this issue and made some progress. Surprisingly I ran into a kernel panic.

lxsim --with-ethernet --with-sdram --cpu-type rocket --cpu-variant linux --csr-csv "csr.csv" --sdram-init boot-sim.bin --threads 8
        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2021 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs

 BIOS built on Jun 23 2021 21:32:11
 BIOS CRC passed (16f2dc15)

 Migen git sha1: 3ffd64c
 LiteX git sha1: adb71bde

--=============== SoC ==================--
CPU:            RocketRV64[imac] @ 1MHz
BUS:            WISHBONE 32-bit @ 4GiB
CSR:            32-bit data
ROM:            128KiB
SRAM:           8KiB
SDRAM:          65536KiB 32-bit @ 1MT/s (CL-2 CWL-2)

--========== Initialization ============--
Ethernet init...
Initializing SDRAM @0x80000000...
Switching SDRAM to software control.
Switching SDRAM to hardware control.

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Cancelled

--============= Console ================--

litex> romboot
Executing booted program at 0x80000000

--============= Liftoff! ===============--
bbl loader
              vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
                  vvvvvvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrr       vvvvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrr      vvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrrrr    vvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrrrr    vvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrrrr    vvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrr      vvvvvvvvvvvvvvvvvvvvvv  
rrrrrrrrrrrrr       vvvvvvvvvvvvvvvvvvvvvv    
rr                vvvvvvvvvvvvvvvvvvvvvv      
rr            vvvvvvvvvvvvvvvvvvvvvvvv      rr
rrrr      vvvvvvvvvvvvvvvvvvvvvvvvvv      rrrr
rrrrrr      vvvvvvvvvvvvvvvvvvvvvv      rrrrrr
rrrrrrrr      vvvvvvvvvvvvvvvvvv      rrrrrrrr
rrrrrrrrrr      vvvvvvvvvvvvvv      rrrrrrrrrr
rrrrrrrrrrrr      vvvvvvvvvv      rrrrrrrrrrrr
rrrrrrrrrrrrrr      vvvvvv      rrrrrrrrrrrrrr
rrrrrrrrrrrrrrrr      vv      rrrrrrrrrrrrrrrr
rrrrrrrrrrrrrrrrrr          rrrrrrrrrrrrrrrrrr
rrrrrrrrrrrrrrrrrrrr      rrrrrrrrrrrrrrrrrrrr
rrrrrrrrrrrrrrrrrrrrrr  rrrrrrrrrrrrrrrrrrrrrr

       INSTRUCTION SETS WANT TO BE FREE
[    0.000000] Linux version 5.13.0-rc3-173670-g9dacadb8c086 (martin@martin-ThinkPad-T14-Gen-1) (riscv64-unknown-linux-gnu-gcc (GCC) 9.2.0, GNU ld (GNU Binutils) 2.34) #3 Wed Jun 9 01:39:48 CEST 2021
[    0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000
[    0.000000] Machine model: freechips,rocketchip-unknown
[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
[    0.000000] printk: bootconsole [sbi0] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000080200000-0x0000000083ffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080200000-0x0000000083ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080200000-0x0000000083ffffff]
[    0.000000] On node 0 totalpages: 15872
[    0.000000]   DMA32 zone: 248 pages used for memmap
[    0.000000]   DMA32 zone: 0 pages reserved
[    0.000000]   DMA32 zone: 15872 pages, LIFO batch:3
[    0.000000] SBI specification v0.1 detected
[    0.000000] riscv: ISA extensions acdfim
[    0.000000] riscv: ELF capabilities acdfim
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 15624
[    0.000000] Kernel command line: earlycon=sbi console=liteuart swiotlb=noforce
[    0.000000] Dentry cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.000000] Inode-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.000000] Sorting __ex_table...
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 47968K/63488K available (3810K kernel code, 4060K rwdata, 2048K rodata, 2808K init, 273K bss, 15520K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] riscv-intc: 64 local interrupts mapped
[    0.000000] plic: interrupt-controller@c000000: mapped 4 interrupts with 1 handlers for 2 contexts.
[    0.000000] random: get_random_bytes called from start_kernel+0x3a8/0x574 with crng_init=0
[    0.000000] riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [0]
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x1d854df40, max_idle_ns: 3526361616960 ns
[    0.000088] sched_clock: 64 bits at 1000kHz, resolution 1000ns, wraps every 2199023255500ns
[    0.006888] Console: colour dummy device 128x32
[    0.009369] Calibrating delay loop (skipped), value calculated using timer frequency.. 2.00 BogoMIPS (lpj=10000)
[    0.011540] pid_max: default: 32768 minimum: 301
[    0.019933] LSM: Security Framework initializing
[    0.025388] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.027537] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.109542] ASID allocator disabled
[    0.116651] EFI services will not be available.
[    0.132434] devtmpfs: initialized
[    0.222549] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.225565] futex hash table entries: 256 (order: 0, 6144 bytes, linear)
[    0.244647] NET: Registered protocol family 16
[    0.695985] clocksource: Switched to clocksource riscv_clocksource
[    1.215104] NET: Registered protocol family 2
[    1.222594] IP idents hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    1.245086] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
[    1.248627] TCP established hash table entries: 512 (order: 0, 4096 bytes, linear)
[    1.250868] TCP bind hash table entries: 512 (order: 0, 4096 bytes, linear)
[    1.252940] TCP: Hash tables configured (established 512 bind 512)
[    1.260748] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    1.263292] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    1.273146] NET: Registered protocol family 1
[    1.417378] workingset: timestamp_bits=46 max_order=14 bucket_order=0
[    2.349512] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
[    2.362676] Oops - load access fault [#1]
[    2.364155] CPU: 0 PID: 1 Comm: swapper Not tainted 5.13.0-rc3-173670-g9dacadb8c086 #3
[    2.365874] Hardware name: freechips,rocketchip-unknown (DT)
[    2.367127] epc : litex_soc_ctrl_probe+0x3e/0x106
[    2.369249]  ra : litex_soc_ctrl_probe+0x30/0x106
[    2.371239] epc : ffffffff802420a6 ra : ffffffff80242098 sp : ffffffe00125bbf0
[    2.373053]  gp : ffffffff80cb4df8 tp : ffffffe001250000 t0 : 0000000000000003
[    2.374847]  t1 : 00000000002dc800 t2 : 0000000000000000 s0 : ffffffe00125bc20
[    2.376517]  s1 : ffffffe00133da98 a0 : 0000000004001b00 a1 : ffffffe001205cd8
[    2.378139]  a2 : ffffffe0012b8368 a3 : ffffffe00133dac0 a4 : fffffffffffff000
[    2.379760]  a5 : ffffffd004001b04 a6 : ffffffff80ca7138 a7 : ffffffd004001000
[    2.381379]  s2 : ffffffe0012b8200 s3 : ffffffe0012b8210 s4 : 0000000000000000
[    2.383039]  s5 : ffffffff80c931f0 s6 : ffffffff80cb7018 s7 : ffffffff8060a278
[    2.384676]  s8 : ffffffff80600468 s9 : 0000000000000008 s10: ffffffff804000a8
[    2.386312]  s11: 0000000000000000 t3 : 0000000000001000 t4 : ffffffff80ca71c8
[    2.387921]  t5 : ffffffd004002000 t6 : 0000000000000000
[    2.389271] status: 0000000200000120 badaddr: ffffffd004001b04 cause: 0000000000000005
[    2.390956] Call Trace:
[    2.391978] [<ffffffff802420a6>] litex_soc_ctrl_probe+0x3e/0x106
[    2.394077] [<ffffffff80267f80>] platform_probe+0x3e/0x7c
[    2.395996] [<ffffffff802667ec>] really_probe+0x8e/0x26a
[    2.397861] [<ffffffff80266a00>] driver_probe_device+0x38/0x74
[    2.399806] [<ffffffff80266f70>] device_driver_attach+0x4c/0x50
[    2.401784] [<ffffffff80266fc8>] __driver_attach+0x54/0xc6
[    2.403810] [<ffffffff8026507e>] bus_for_each_dev+0x5a/0x98
[    2.405698] [<ffffffff80267054>] driver_attach+0x1a/0x22
[    2.407555] [<ffffffff8026581e>] bus_add_driver+0x12e/0x190
[    2.409417] [<ffffffff80267572>] driver_register+0x3a/0xd0
[    2.411331] [<ffffffff80268d32>] __platform_driver_register+0x1c/0x24
[    2.413384] [<ffffffff8041044e>] litex_soc_ctrl_driver_init+0x1a/0x22
[    2.415304] [<ffffffff80400cd8>] do_one_initcall+0x6e/0x134
[    2.417214] [<ffffffff80400f78>] kernel_init_freeable+0x172/0x1de
[    2.419164] [<ffffffff803b5018>] kernel_init+0x12/0xf0
[    2.421115] [<ffffffff8000302e>] ret_from_exception+0x0/0xc
[    2.426654] ---[ end trace c52f0c50dc8758d7 ]---
[    2.428148] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    2.429979] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---

@gsomlo
Copy link
Collaborator

gsomlo commented Jun 26, 2021 via email

@troibe
Copy link
Contributor Author

troibe commented Jun 27, 2021

If you are referring to my access strobe fix for blackparrot then I don't think so.
I didn't touch anything Litex or Rocket related for that fix just Blackparrot's sytemverilog code.
Therefore Rocket simulation shouldn't be affected.

Unless you also ran into some strobe access problems on Rocket that I'm not aware of.

@gsomlo
Copy link
Collaborator

gsomlo commented Jun 27, 2021 via email

@troibe
Copy link
Contributor Author

troibe commented Jun 27, 2021

Is there some sort of (legitimate) address translation happening?
The scratch register should be at 0x12000004 according to the CSR map.

@gsomlo
Copy link
Collaborator

gsomlo commented Jun 27, 2021 via email

@troibe
Copy link
Contributor Author

troibe commented Jun 29, 2021

Thanks for the EDIT now I have something to look into.
Up to this point I wasn't really sure what to look for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants