future of io

[schveiguy]

I wanted to talk about possibilities for io. @jmdavis asked me what I thought about io, and I said I'd take a look here.

These are my thought on it, and I'm also OK with not doing any of this, and just doing it on my own packages.

D i/o is currently based on C i/o.

This is very limiting, and precludes doing things like event-loop based i/o. Walter, Andrei and I have discussed this in the past, and I had a very sketchy scheme that I planned to implement a long time ago when I thought I could replace Phobos i/o with what would eventually become iopipe. But I think phobos v3 has a real shot at doing this. What I would propose is:

1. Provide a D-only solution to all i/o.
2. Provide a wrapping type for the D API that has the ability to directly copy to a C FILE *, but doesn't have to.
3. The default write[f][ln] functions should use the C-wrapper-based i/o, but with an option to replace with direct D-only i/o.
4. stdin/stdout/stderr can be of this type, but it should NOT be the normal File structure. This structure should be exclusively D i/o. You should not pay the penalty of using C i/o with the default File infrastructure.

Some examples of WTF moments with D and std.stdio:

1. All File instances require locking, even if they aren't shared.
2. std.process uses File for pipes! The API is terrible for this, and the buffering mostly gets in the way, especially on the write side.
3. std.socket does not implement things using File. So abstracting networking with disk i/o requires wrappers not present in Phobos.
4. File is primarily text-based. Many files have nothing to do with text. Using File on binary files is clunky to say the least.

Async i/o is critical

Making it easy to do asynchronous i/o is a must for any programming language designed for servers.

Packages such as vibe, arsd, photon, and mecca all use an event loop (possibly others). I also plan on writing my own, as I find vibe eventcore to be too complex to deal with.

In order for these packages to all work together, they need to be a common implementation. If not a common implementation, then you must pick one implementation and only use packages built for it.

An event loop does a lot more than just i/o. Timers and other events (such as UI events) are also relevant.

Many languages (javascript, swift, C#, etc.) all have a basic batteries-included event loop system. async/await features depend on it being part of the language.

I think the gold standard here is a callback-based event loop, with Fibers to allow writing synchronous-looking code. Stackless coroutines are also possible, but I haven't explored that yet.

My projects

I currently have inherited Martin Nowak's io package (ambitiously packaged as std.io), which has cross-platform low-level io primitives. The concept was to provide a basic simple low-level i/o that could be built upon with buffers, formatting, etc. The plan was originally to provide multiple i/o "drivers", but I think at this point I will remove that possibility, and pick just async or sync i/o based on what you are doing. This will remove a lot of the complexity currently there.

I also have my iopipe library, which is somewhat low-level, but I think has the right abstraction for dealing with streams.

The jsoniopipe library is a json parser which provides a stream agnostic way of parsing JSON. The idea is, it shouldn't matter how you make your stream, there should be a way to parse it. This includes normal arrays, which provide an easy way to test things. Your code stays the same, you just swap out the pieces. I envision a whole fleet of parsers that are built this way, and then you can plug them together however you want. I also intend to rework this project so the json parser simply produces an iopipe of json tokens that can be treated the same as any other iopipe (I had a blog post about this recently: https://www.schveiguy.com/blog/2023/06/new-iopipe-abstraction-segmentedpipe/)

In general the iopipe concept I view as ranges, but for streams. I would love to see Phobos adopt this.

[pbackus]

The default write[f][ln] functions should use the C-wrapper-based i/o

Is this really necessary? My impression is that other languages generally don't do this, and it leads to issues like not being able to read Unicode input from the terminal on Windows.

[schveiguy]

It was necessary, in fact a requirement, from Walter. At least back when I was attempting to get iopipe in as the replacement for std.stream. I'm assuming that hasn't changed.

For what it's worth though, I think it's not that big of a hill to climb. Most of the time, writing to a terminal is OK to be lower performing.

[schveiguy]

issues like not being able to read Unicode input from the terminal on Windows

So this is why I said, you should have the option to turn it off. Something like stdin.detachFromC(); or something like that.

[pbackus]

Why not have it do the non-broken thing by default, and have attachToC be an opt-in?

[schveiguy]

I'll leave that up to Walter to answer. I'm on your side 😉

[rikkimax]

This is very either-or thinking.

You can have both a console module as well as the standard IO module.

Focus on text with console, and binary for standard IO. Use what is appropriate to the program.

[pbackus]

The idea is, it shouldn't matter how you make your stream, there should be a way to parse it. This includes normal arrays, which provide an easy way to test things. Your code stays the same, you just swap out the pieces.

[...]

In general the iopipe concept I view as ranges, but for streams. I would love to see Phobos adopt this.

What differentiates streams from ranges? If I'm writing something like a parser, what rules or guidelines should I use to decide whether to use the stream interface or the range interface?

[crazymonkyyy]

imagine you have a file thats is some format with comments so you start your range chain with a .filter, a filter that should be generic and handle non-bulk data; then you need to look ahead and match () maybe you let symbols be 80 chars
so you want to add a buffer that stores 81 elements

is double buffering so bad and evil you cant afford 81 bytes and 1 extra copy per loop?

no it wont be a parallelized simd json parser thats winning hn upvotes, but I think your letting the perfect be an enemy of the good, because to work around the old range syntax or use a pre existing bloated format, id have to make a choice between .array and having the whole file in ram to do any look ahead or some 10k line dependency and ughhhhhh no

The goal should be to get into the range ecosystem with as many features as possible

[schveiguy]

@crazymonkyyy I've completely lost what you are trying to say.

Perhaps try and build the thing you are saying is "good enough" with ranges, and then I can understand it.

[crazymonkyyy]

I have "foobar[hi(1223423)..$] = foolknnasdjlkn".elementslice!".byascii"('[',']').writeln; => "hi(1223423)..$" with a iterator concept that treats data stuctures as a generic type and a range concept with indexing in https://github.com/crazymonkyyy/raylib-2024/blob/master/lib/myalgorithms.d#L140C2-L140C85

id want to make my iterator concept have a climbing scale of array features
where you start with index then opIndex(type(index), and opSlice(type(index),type(index)) that seems like a logical progression to me; I haven't made demos because it be a nightmare to take on the whole of std algorithms alone

If you make indexing and arrays more important in the range interface then

If you can come up with a scheme for ranges that adds similar functionality, I'm happy to listen.

you could write a .bufferWith!ringarray that would be yes double buffering but would have lookahead in file io without saving the whole file

[schveiguy]

So how does one use this to parse JSON for instance? How do I get forward range-like functionality from this? What things does bufferWith expose to enable buffer management?

Honestly, bufferWith is exactly what iopipe does. See https://schveiguy.github.io/iopipe/iopipe/bufpipe/bufd.html

My expectation is as you flesh out your idea, you get to the same point. Which is why I say, please show me the design. And I don't mean hand-wavy not-yet-defined things, I mean actually try it.

[crazymonkyyy]

My expectation is as you flesh out your idea, you get to the same point.

window, release, extand

I want lookahead not a sliding window and the api to be front, pop, empty and the metaprogramming that goes along with that scheme

"iterate over all char's, if you have a '\' lookahead one char for special syntax otherwise return front" is how I think about escaped chars in english

for(auto buffer=File(...); ! buffer.empty;(){buffer.release(1);buffer.extend(1);}){
  auto w=buffer.window;
  if(w[0]=='\\'){//find excaped chars
    ...

would feel like a c++ iterator ritual

Im very very sold on front, pop, empty being related what loops usually are and being the correct starting point for an iterator api

I'm saying id let the double buffering happen to let terse metaprogramming things that I may not be able to articulate happen.

please show me the design.

#13
https://i.kym-cdn.com/entries/icons/original/000/022/524/pepe_silvia_meme_banner.jpg

So how does one use this to parse JSON for instance?

I was using json as an example of a bad file type I think theres to much complexity to simplify, for good data types that have good complexity levels; consider usv/lazy csv and stacking "bufferUntil"(store values until you find _, return buffer)

How do I get forward range-like functionality from this?

I want to kill the old hierarchy, bufferwith would need an array like datastucture to return some arraylike features

What things does bufferWith expose to enable buffer management?

nothing; if ringarray throws bufferWith!ringarray throws; if ringarray silently overwrites, bufferWith does; crashes on underflow,...; warns on overflow,..; etc.

[atilaneves]

I'll leave that up to Walter to answer. I'm on your side 😉

Or me, since this is Phobos.

[schveiguy]

I'm just saying what the situation was years ago. Maybe Walter has come around and is not so focused on printf compatibility?

[atilaneves]

My understanding is he likes printf because writeln means templates. I think the solution is to eliminate whatever problems templates have.

[schveiguy]

So to reiterate, the problem that was cited was that using both printf and some other buffering scheme interleaving would cause confusing output results (i.e. lines from one scheme split by lines from printf, or vice versa). I think it has more to do with using C code that uses printf than using printf directly.

[dukc]

I have a bit of comments about the proposed package structure with regards of the IO modules.

First, in my opinion std.process should belong to them. Instantiating other processes and invoking terminal commands is very much IO in my books.

If we do that, I think it means nothing else in lib.sys depends on the io modules anymore. Therefore the io package can and should become a direct member of lib.

I also suspect path shouldn't belong to the IO package. Apart from expandTilde it only handles paths, but doesn't actually query or pass them to the operating system. Maybe the string module would be a good place for it?

[rikkimax]

First, in my opinion std.process should belong to them. Instantiating other processes and invoking terminal commands is very much IO in my books.

I'm currently dealing with process creation for pipes in my own stuff.

It does interact with the eventloop, and uses the cleanup timer stuff that sockets also need.

So I can confirm its IO tied.

Note: writes on Windows are blocking, reads can be done through other means.