Binary Postgresql encoding of (b)vector & other structures

[zopieux]

For anyone seeing this in the future: I cannot be bothered to go through publishing a crate for this, but here is the code:
https://gist.github.com/zopieux/971956ec8105b9931ad818fefc36a805

---

Hello,

I am trying to write the sqlx encoder/decoder for pgvevcto.rs structures. I'm starting with bvector(N). Sadly, I can't find the Rust code in this repository doing the binary encoding amongst the thousands of files, probably with a serde layer in the middle.

From "reverse engineering" from existing rows inserted using normal SQL, it seems like it the first two bytes encode the length as LE i16, but the packing of the actual bytes is then unclear, especially at high bvector sizes likes 12345 for instance. It seems like there is some padding or similar happening.

Can you point me to the code encoding/decoding (b)vector into bytestreams for Postgresql storage? Thanks!

[usamoi]

Code for binary representation is in https://github.com/tensorchord/pgvecto.rs/blob/main/src/datatype/binary_bvecf32.rs. Code for storage layout is in https://github.com/tensorchord/pgvecto.rs/blob/main/src/datatype/memory_bvecf32.rs.

[zopieux]

Thanks @usamoi!

For anyone interested in the result, https://gist.github.com/zopieux/971956ec8105b9931ad818fefc36a805 implements vector & bvector for sqlx, and the minimal tests are passing.

[spikecodes]

I was looking for this exact thing, thank you @zopieux! It would be great also to have this documented by the pgvecto.rs maintainers, given that SQLx is most likely the driver Rust users would use with this extension.

[spikecodes]

Using Claude 3.5 Sonnet, I converted the unsafe code to a safe code alternative. It passed all of my tests so I believe it should function the same: https://gist.github.com/zopieux/971956ec8105b9931ad818fefc36a805?permalink_comment_id=5137314#gistcomment-5137314

[xieydd]

@VoVAllen @gaocegege PTAL