[DRAFT] Diz 3.0 rewrite - early pull request for comments / review

[binary1230]

Diz 3.0

WARNING: This is super-early super-rough alpha-level code. Don't use yet unless you're ready to jump in with a C# debugger and hack the unit tests :) Do not even attempt to open prior projects yet. I am mostly opening this PR to start making my own release checklists. The code is now starting to cool down enough that it could be worth checking out now. The UI needs the most polish work, the underlying data model is starting to be pretty solid.

This is basically another major rewrite of Diz, which will be version 3.0. Major new features include:

• Port to .NET5 !
• Full overhaul of the data model (see below). Flexibility extends far beyond SNES now
• Overhaul the UI and enable multiple simultaneous views of byte data from multiple projects and ROMs at once
• Decoupling the UI from the mapping to the underlying data, making it more flexible
• Easy to add more UI components and tie them into the underlying view architecture (i.e. do you want a graphics viewer or a hex editor? implement a new View interface and it's a piece of cake)
• Rewrite of the SNES assembly output engine, easier to mod and test
• Add Powershell command-line interface, to integrate Diz into your automation workflows (easy to port to other stuff like Python/etc)
• Diz.Core is now a standalone library that can be included in other projects or emulators. Or, swap in your own CPU engines/etc or pipe data through from emulators/etc
• Heavy unit test coverage on Diz.Core ByteSource/Storage
• Flexible XML output to be able to accomodate future expansion
• Use Github actions for builds / testing
• CPU stepping functionality refactored into a way that's easier to control and subclass, so adding new CPUs or variants should be easy.
• Diz's workhouse datagrid has been supercharged to allow displaying arbitrary columns. Just pass it a new class with different properties and it'll automatically change things up and format according to attributes
• Datagrid has intelligent caching to preload and keep in memory the most recently viewed bytes. Should make dealing with the largest projects a breeze

Data Model

• Probably the most exciting part: Diz now utilizes a series of arbitrarily nested regions of memory, called ByteSources, which overlay, combine, and resolve into a final view for the UI and assembly output
• For example, for SNES, a typical setup will be: the root ByteSource is the SNES Address Bus, which contains a ChildSource
• Customize as you like, for example, create a mapping that overrides underlying data
• Mirroring should be handled well now: Map a region of WRAM into several mirrored regions. Tag bytes on WRAM with labels/annotations and they will magically show up in the correct mirrored addresses in the address space
• The original RomByte class has been broken up into several "Annotation" classes. These can be attached to a byte on any ByteSource layer
• Underlying storage can either be set to continuous (list based), great for quick and direct access in ROMs. or it can be set to sparse (dictionary based), which is great for large empty address spaces like CPU busses.
• The view of the various byte regions largely "does the right thing" and is easy to consume from the SNES Address bus level. However, the underlying low-level data model is able to return a graph that lets you traverse through the layers and collect / aggregate the underlying data for more advanced usages.
• Data can be setup with different policies. For example, a "patch" could be setup as a ByteSource that maps to 100 bytes of a bank of the SNES ROM, and anything in the path hides the data underneath. Or, take dumps of RAM at various points in the game and cycle through them.

Future stuff now easier to implement

• Diz is currently setup just for SNES, but, the overhauls make it far easier to add in support for other systems and CPUs.
• Memory mapping system was designed so it can map non-linearly if needed (i.e. map a compressed region of ROM into RAM via a custom algorithm)
• Dependency injection support coming soon to add extra plugins/etc
• Expression parsing for instructions that can be constrained to resolve back to the original ROM bytes (mirrored labels giving you the wrong offsets? it'll be no prob)
• No reason this stuff couldn't be used to make a web-based visualizer via React/Blazor/etc.
• It is my hope that the Diz XML file format can be used as an archival format for being both human-collaborator-friendly and a long-term storage friendly way to preserve videogame history

Checklist before putting this in:

• Update credits to include @furious who contributed some of the .NET5 work / Github CI, and has a few more patches we need to integrate. also work done in Upgrade to .NET 5 (latest version) #42 was the foundation of this PR
• Finish UI, it's a bit haggard right now after the data model
• Probably merge this as a rebase and not the individual commits, my commits are kind of sloppy and not atomic up to this point
• Finish polish on MVC architecture, it got a little beat up in the refactor
• Backwards compatibility for previous version XML files
• Optimize performance, if needed, for XML serialization
• oy. so much more.
• Get decent code coverage on remaining fringes of Data model (we're actually pretty decent right now)
• Cleanup lots of commented and junk code leftover
• Test some real-world .dizraw files I have
• Notify the SNESlab discord of progress
• Code review.
• Consider un-renaming some of the files to make the diffs easier to look at?
• IPropertyChanged stuff is horribly broken at the moment, re-enable and test. unit tests

Closes #46

[rdrpenguin04]

Will this be compatible cross-platform?

[binary1230]

Will this be compatible cross-platform?

I'm hoping it will. Everything already is except the GUI, and there's another branch off this branch that is attempting to replace the winforms GUI with AvaloniaUI, which is cross platform.

[rdrpenguin04]

Okay, thank you!

[rdrpenguin04]

Update: Tried on avalonia-ui-test2 (on Linux), and dotnet build still doesn't work, saying "Windows is required to build Windows desktop applications." If you want I can send you the full error here; otherwise, it may be better to open this as an issue on your repo.

[binary1230]

Update: Tried on avalonia-ui-test2 (on Linux), and dotnet build still doesn't work, saying "Windows is required to build Windows desktop applications." If you want I can send you the full error here; otherwise, it may be better to open this as an issue on your repo.

yea, unfortunately, it's still referencing winforms, the avalonia-ui-test2 branch is BOTH avalonia and winforms together, with the intention of slowly replacing one with the other.

also there's little avalonia UI actually written (I do have the main grid in a simplified form). the big thing going on first is to get the UI using ReactiveUI under the hood.

If you really did want to try it under linux, I -think- in that fork I had put all the winforms stuff in one project. you could modify the solution to not build that, and start avalonia.

all this is super-untested and very early, I'd have to look around and make sure it compiles/etc at the moment. I have though run it with the minimal avaloniaUI on linux. it might have been with mono, I forget (stock .net -should- work OK here now on non-windows)

[binary1230]

Update: Tried on avalonia-ui-test2 (on Linux), and dotnet build still doesn't work, saying "Windows is required to build Windows desktop applications." If you want I can send you the full error here; otherwise, it may be better to open this as an issue on your repo.

yea, unfortunately, it's still referencing winforms, the avalonia-ui-test2 branch is BOTH avalonia and winforms together, with the intention of slowly replacing one with the other.

also there's little avalonia UI actually written (I do have the main grid in a simplified form). the big thing going on first is to get the UI using ReactiveUI under the hood.

If you really did want to try it under linux, I -think- in that fork I had put all the winforms stuff in one project. you could modify the solution to not build that, and start avalonia.

all this is super-untested and very early, I'd have to look around and make sure it compiles/etc at the moment. I have though run it with the minimal avaloniaUI on linux. it might have been with mono, I forget (stock .net -should- work OK here now on non-windows)

the core of the project though IS separated out so it works without a gui (there's a powershell command-line-only project included that demonstrates using the Diz library functionality from non-gui). that means it can easily work inside another GUI or even in webassembly like on a webpage, or as a backend with a react-ui javascript frontend/etc.

sorry, I mostly ran out of time to keep working on this for the forseeable future. if you were interested in digging in, I'm down to help, a LOT of the backend datamodel overhaul has been completed in this branch already, the UI is just not caught up to it great.

[zoncabe]

hi guys, im really interested into making this work on linux
in what state is this project currently?