StructArray or Arc<StructArray> #492
-
Hi, I'm doing some kind of streaming applications in Rust at the moment to do some batch streaming into Redshift. Basically we have a kafka consumer, processors which will turn events into StructArray, and async writer to write them to S3, then call load manager API to queue the file in order to be upload to Redshift. All queue using From the processors to async writers, should we parse the If my understanding are correct, both Thanks |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
Cool scenario :) I would go for They are both
Since you mentioned a streaming scenario where sometimes the size of the metadata matters, and there is a small overhead in cloning a |
Beta Was this translation helpful? Give feedback.
Cool scenario :)
I would go for
Arc<StructArray>
.They are both
Send+Sync
. TheArc
offers two things:StructArray
to be a trait objectArray
.Since you mentioned a streaming scenario where sometimes the size of the metadata matters, and there is a small overhead in cloning a
StructArray
due to the clone of the metadataVec<Field>
, I would go forArc<StructArray>
.