Skip to content

Latest commit

 

History

History
181 lines (146 loc) · 9.95 KB

0197-2024-04-02.md

File metadata and controls

181 lines (146 loc) · 9.95 KB
Raw

2 Apr 2024

Previous journal: Next journal:
0196-2024-03-30.md 0198-2024-04-03.md

Diagnosing tt04-raybox-zero ASIC malfunction

Summary

Sylvain (@tnt) has been testing the TT04 ASIC with raybox-zero 1.0 on it, and my design has some problems. This animation shows one exact, specific view, comparing how Verilator renders it and how the ASIC actually renders it:

raybox-zero 1.0 glitching on ASIC

Note the following:

  • This is taken from a raw capture at 3MHz, but it looks like the same artefacts occur at 19MHz (and maybe 24MHz).
  • Using the debug box in the top-right, the scale is very close between the two images, despite it looking like the glitchy one is shrunken. This suggests there is an overall coarse error in the distance calculation.
  • The colours are weird because of the specific RGB channels that Sylvain was able to capture at the time. I made some adjustments to match my Verilator version as closely as possible.
  • This version of raybox-zero (from memory?) starts the texel counter from the bottom of the wall column, rather than pre-calculating it via multiplication. It then stops rendering the texture when the texel counter overflows 1.0 (or 63?), which has led to some rendered examples showing what look like short walls, but they're actually errors (I think) in the texture scaling.

I think this generally suggests that the reciprocal is not (always) working correctly. It might have single-bit errors, but it's hard to tell so far.

The first step to trying to work out what's going on is to make sure we can regenerate the original GDS and other build/synthesis artefacts from the time I did this submission...

Other images of glitches

Reference Sylvain's capture
Reference view 1 Sylvain's capture 1
Reference view 2 Sylvain's capture 2
Reference view 3 Sylvain's capture 3
Reference view 4 Sylvain's capture 4

The pov_payload values of the images above are:

  1. 00110100011011100011111011011000000111101110000001000001111111000000011110
  2. 00101110000100100001110001101111010011111110100011100001011100111101001111
  3. 00111010000111000101101111111100100011011110010101100001101010100010001101
  4. 00011011010010000100101111111000111111111000000000000000000000000100000000

Another random view from Sylvain:

Sylvain random

Code sent to Sylvain with POV payloads

I created the spi_load_view.py snippet which I sent to Sylvain, that bit-bangs the SPI pov_payload values as shown above into Sylvain's test board, and it seems to work OK.

tt04-raybox-zero 1.0

Checking the repo:

  • Linked to raybox-zero v1.0 (commit 1029ddb)
  • There are no tests; I must've just used Verilator (the sim) the whole way thru!

Synth:

  • I recall I was doing synth on my L7 laptop, in which case likely VM is "Zerto to ASIC Course MPW8".
  • Other possible VM is "Zero to ASIC Course MPW8" on my desktop PC. It has a snapshot called "Before TT04 OpenLane", but then "Before Analog".
  • I will go with checking on the former, first...
  • Default environment:
    • PDK dirs were created 2022-12-29
    • PDK (sky130A) hash is: 3af133706e554a740cfe60f21e773d9eaa41838c
  • ~/CUP linked to: /home/zerotoasic/asic_tools/caravel_user_project
  • CUP is the ew version
  • Final TT04 submission was around 2023-09-11, so look for journals near there.
  • ~/anton/projects/tt04-raybox-zero is dated 2023-09-10... could be the one I'm looking for.

Instructions for doing synth locally:

Background:

Steps:

  1. Before I muck around with anything, I'm doing a snapshot of the VM: Before TT04 2024 reharden
  2. Clone a new copy of tt04-raybox-zero at version 1.0: 
    cd ~/anton
git clone \
    -b 1.0 \
    --recurse-submodules \
    git@github.com:algofoogle/tt04-raybox-zero \
    bringup-tt04-raybox-zero
cd bringup-tt04-raybox-zero
  3. Set up OpenLane environment to match what I originally had for my TT04 submission: 
    export OPENLANE_ROOT=~/tt@tt04/openlane
export PDK_ROOT=~/tt@tt04/pdk
export PDK=sky130A
# This OpenLane version is using the one specified
# in tt-gds-action@tt04 from 2023-09-13 (24271d1)
# (https://github.com/TinyTapeout/tt-gds-action/blob/24271d1a569576b6a161ee93ef04fa2aa2e641ab/action.yml#L14-L15)
# ...which at least seems to be the same one I was using back in the day
# (https://github.com/algofoogle/journal/blob/master/0127-2023-08-20.md)
export OPENLANE_TAG=2023.06.26
export OPENLANE_IMAGE_NAME=efabless/openlane:3bc9d02d0b34ad032921553e512fbe4bebf1d833
  4. Clone TT support tools (@tt04) into 'tt' subdidr of bringup-tt04-raybox-zero, and prep its Python env: 
    cd ~/anton/bringup-tt04-raybox-zero
git clone -b tt04 https://github.com/TinyTapeout/tt-support-tools tt
#NOTE: This wasn't the version I used at the time, because that tt04 branch
# is now up to 2023-11-10. I would've used somewhere around here:
# https://github.com/TinyTapeout/tt-support-tools/commit/dac3f4a0a4527ca7c65ce44987bdb2eaa25f5525

python3 -m venv ~/tt@tt04/venv
source ~/tt@tt04/venv/bin/activate
pip install -r tt/requirements.txt
  5. Set up OpenLane to match the required version for TT04: 
    git clone --depth=1 --branch $OPENLANE_TAG https://github.com/The-OpenROAD-Project/OpenLane.git $OPENLANE_ROOT
cd $OPENLANE_ROOT
make # Takes about 3 minutes and eats 2.4GB of disk space in ~/tt@tt04
  6. Do the harden: 
    cd ~/anton/bringup-tt04-raybox-zero
source ~/tt@tt04/venv/bin/activate
./tt/tt_tool.py --create-user-config
time ./tt/tt_tool.py --harden
# To my surprise this took only 07:48 on my VM in turtle mode.
  7. Attempt to verify:
    • Look at the GDS: Render GDS using Anton's updated GHA method: 
      ./tt/tt_tool.py --create-svg
#NOTE: The following fails with this design on Ubuntu 20.04
# (as used by MPW8) because that uses a version of rsvg-convert that only
# supports up to 200,000 XML elements. Either need to convert
# on Ubuntu 22.04, or handle it on Windows?
rsvg-convert --unlimited gds_render.svg -s <(echo 'text{display:none;}') -o gds_render_png24.png --no-keep-image-data
      ...I rendered instead in Photoshop, which took ages. It resembles the "shoe" that it was expected to look like: Photoshop render of GDS SVG for raybox-zero 1.0
    • Can also view the GDS in KLayout like this: 
      summary.py --top tt_um_algofoogle_raybox_zero --design . --run 0 --caravel --gds
    • Run ./tt/tt_tool.py --print-stats and thankfully I got exactly the same results as a recent GHA re-run of 1.0 (but I'm not sure about the original final run):
      Utilisation (%) Wire length (um)
      47.55 207590

Next steps

  • Create cocotb stub for RTL: DONE - See: https://github.com/algofoogle/tt04-raybox-zero/tree/1.0-test/src/test
  • See if GL sim is possible
  • Patch in GL sim from more recent (EW?) raybox-zero, if necessary
  • Document recent TT04 raybox-zero ASIC findings so far
  • Option for Verilator sim to write out a data stream, e.g. to compare with ASIC sigrok capture.
  • Fix L7 VM VT-x.

Theories about incorrect rendering

Lessons learned

  • Have more debug output options, e.g. there could be a mode to stream out the actual trace internals instead of image data. We have 6 channels (RGB222) to use easily:
    1. Trace vector X
    2. Trace vector Y
    3. Distance
    4. Height (distance reciprocal)
    5. Texture scaler
    6. ?

Other notes

  • SR (sigrok) file: From conversation here: tt04-33.sr -- Capture of ASIC's outputs.