Ditherpunk roadmap

[bunnie]

The ditherpunk feature allows images to be displayed on Precursor using diffusion dithering to create black and white bitmaps.

Currently, the feature is gated off. There are a few issues that need to be resolved before the feature can be turned on by default, and they fall into two categories: functional, and contextual.

Functional

• Understand where stack and heap allocation requirements are coming from. System will not run correctly unless the default stack size is increased from 128kiB to 1024kiB. At the moment static analysis shows that the "big allocations" are now gone, but still, during runtime, we get failures during boot, so something is still funky.
• Fix "black bar on right edge" for images that aren't a perfect multiple of 32 bits
• Fix autoscale/rotate (or do we even need the feature?) currently it is commented out, but it does not handle the case of smaller-than-screensize images being scaled up and centered to fill the screen.
• Recode the Tile passing opcodes to pass all 6 tiles at once, instead of in a loop

Contextual

• What image formats should we allow? Does it make sense even to have Precursor doing auto-rotate, scale, and transformation from image formats on-device, or should we expect application developers and server operators to pre-format the data for the client?
• What are the functions and roles of images? This needs to be understood contra the security risks it can bring up from e.g. images of fake password boxes being introduced on the screen in a chat.
• How do we safely display images that can violate the rule of "only secure processes can do inverse text rendering"? Granted, this rule is weak, and other improvements need to be made such as changing the color of the status bar during high security events (an area of the screen that on process can modify except the most trusted processes).

This discussion is opened to help record decisions made around this feature.

[nworbnhoj]

• confirmed that the "black bar on right edge" issue exists in hosted mode on small images. I will submit a PR for this when I have it fixed.
• confirmed that small images are not centered correctly in hosted mode. I will submit a PR for this when I have it fixed.
• Image files of arbitrary size can arrive in xous via a messaging App or whatever. To handle these images at all it is important to get them right-sized for precursor ASAP. This is challenging, so it is better to do it once optimally so that the App developer need not re-invent the wheel. So I think it is important to provide a central (optional Server?) to "do it right" including decode, resize, rotate (best use of an already constrained screen), and the future possibility of "pan". A dedicated optional Image Server is looking like a better option than employing the Modal infrastructure due to the significantly different requirements.
• Untrusted areas of the screen are degraded with randomised lines. In similar form, perhaps an image could be "degraded" with a single dynamic line ricocheting around the image? In this way an image is called-out without permanently obscuring any part of the image.

[nworbnhoj]

The standard dither techniques will not work terribly well for "stickers and visuals" or very small images. This could be improved somewhat by designing a dither technique specifically for tiny images (interesting technical challenge). The problem is two-fold. Firstly, to improve performance on large images, the standard dither techniques disperse the quantization error to the right and down, and simply discard it at the edges of the image. The distortion and information loss becomes very apparent on very small images. Secondly, dithering simply needs a bit of space to trick the eye into seeing grey.

[bunnie]

I don't mean to prevent anyone from doing this, but, the model of 'your app is the enclave' breaks down when you have two apps running with such vastly different attack surfaces and threat models. In order to keep very precious passwords safe from a more featureful and thus more easily compromised chat app, you then want an enclave in your enclave... and we return to the model already found on most phones and devices today.

So, I'd say there is no fundamental reason you can't run both at once, but it's not better than the situation you have today where we have condensed all the security factors into a single device.

The only bit of 'magic' that a Precursor gets you is simplicity, and the more one drifts from that the less magic you have.

The other thing I'd point out is that operationally, there are reasons to want chat apps on burner devices... so being able to choose to discard / destroy a device rather than have a contact discovered is a significant operational freedom. If you also put your passwords in the same device as your chat, you would also be destroying your password store if you were forced into such a dilemma. Basically, chat has a more ephemeral nature, and protecting the contacts is often more valuable than the nature of the chat itself; there is a stronger case for wiring up the self destruct mechanism. Password stores have a more vault like nature, and perhaps you would want to fall back more on plausible deniability rather than opt for outright self destruction of the entire store.

[bunnie]

The standard dither techniques will not work terribly well for "stickers and visuals" or very small images. This could be improved somewhat by designing a dither technique specifically for tiny images (interesting technical challenge). The problem is two-fold. Firstly, to improve performance on large images, the standard dither techniques disperse the quantization error to the right and down, and simply discard it at the edges of the image. The distortion and information loss becomes very apparent on very small images. Secondly, dithering simply needs a bit of space to trick the eye into seeing grey.

that's a good point... probably any UX to use stickers or emoji would need to have an exaggerated area to display such things. I would not expect clean inter op with full color high DPI, but you could at least get as idea of what is going on...

[nworbnhoj]

Some thoughts on "How do we safely display images...". We could ....

1. deface all images permanently as per other untrusted areas of the screen. Easy but nasty.
2. deface images dynamically with a line richochetting slowly around the image. Requires a pump in graphics-server to update line position. A little bit of resource, but does not leave areas of the image permanently obscured.
3. deface images with lines by default - which are cleared when a key or key-combo is held. Requires a graphics-server opcode and dedicated key-action. The line would be drawn with bitwise not across the image and then stored to be reversed on key-action.
4. deface images with substantial black triangles in each corner - which are cleared when a key or key-combo is held. Requires a graphics-server opcode and dedicated key action. Leaves the central area visible and obvious on small images/icons.

[nworbnhoj]

Issues with "black bar on right edge" and "cener alignment" are now fixed with 39815cc596 and 1e266b3389 respectively.

[nworbnhoj]

Bitmaps are centered based on the expected Modal dimensions in services/modals/src:

const MODAL_WIDTH: u32 = 300;
const MODAL_HEIGHT: u32 = 370;

But with small images the actual Modal is only about 140 px high - so the small bitmap is moved to a position below the Modal clip rectangle.
This could be resolved in couple of ways:

• the Modal could guarantee the maximum height available (say 300x800) and update these constants accordingly
• the Modal server could provide a API to query the guaranteed modal dimensions and center the bitmap on these dynamic response,
• The Modal API could accept a dimension request.
• The Modal Server could center and size the Modal tightly around Images dynamically
• something else

[bunnie]

Ah that matches what I was seeing...the image was drawing "under" the visible area so I saw a small blank box, and when I looked at the Y bounds it was already below the bottom of the box.

The behavior of the Modal should probably be driven by the application envisioned for it -- more thoughts on that shortly...

[bunnie]

Turns out i just didn't wait long enough, and the box i was thinking was the test modal was from the other test thread. I reduced the iterations to 400000, and was able to have it finish in a couple of minutes (didn't time it exactly). The pattern is pretty faint with that many iterations, but it's there!

[nworbnhoj]

Great news thank you. I can tweak the clifford generator to cut down the generation time easily enough - and it may not impact the quality significantly, given the purpose and end result.

I have almost completed converting the entire imaging processing chain to a series of adaptor iterators - to process a pixel at a time rather than an image at a time. This should have a significant impact. Once this is complete then I can have a look at giving the decoder the same treatment.

[bunnie]

Jut got PR #190 and it works great. Had to make a minor adjustment to box rendering size: 1e266b3 so that the size of the box always shows up as full size since the centering calculation relies on it.

[bunnie]

btw, progress on issue #188 has allowed us to see this for the first time:

[nworbnhoj]

We are beginning to "understand where stack and heap allocation requirements are coming from." with insight in to the binary bloat associated with the enum_dispatch idiom. And we are circling around some sort of a "App framework" which involves "a chat app, which maybe initially has nothing but just frameworks for UX and typing text".

There are a few ways that images could be part of this "App framework".

• Image functionality could be bundled directly into the "chat App", but I suspect this App will quickly grow to be unweildy.
• Image functionality could be provided as a dedicated service Server, available to the chat App but also serving icons etc to other Apps. This seems like a tidy arrangement. And would be a natural place to hold and serve up Tiles when the graphics -server completes a redraw.

An image service could replace the current Imagemodal with an equivalent very basic Image "viewer". Could this be the point of separation between the "password wallet" and "chat" applications - where the graphics-server and gam hold the basic capacity to process Tiles and Bitmaps but the "password wallet" config for xous would omit the image service?

[nworbnhoj]

One simple way to disrupt malicious "images of fake password boxes introduced on the screen in a chat" is for the graphics_server to blank all images for 1 sec every time and alpha-numeric key is pressed.