How to load data blob in C++ and then use segmenter?

[riajain0412]

I am using ICU4X library in C++. I was trying segmenter code with test data provider initially but now I want to use data blob with all locales and segmenter files. So, I want to know that how to load this data blob in C++?

[riajain0412]

Hi @sffc , I've written below code to load data file but it is not running:

UnicodeString str ("ພາສາລາວ");

  printUnicodeString(str);
  cout<<endl;
  std::ifstream file("checking.txt");
  
  if (!file.is_open()) {
      // handle error
      cout<<"here1"<<endl;
  }

  if (!file.good()) {
      cout<<"here2"<<endl;
      // handle error
  }
  file.seekg(0, std::ios::end);
  std::streamsize blob_size = file.tellg();
  file.seekg(0, std::ios::beg);
  cout<<"size buffer "<<blob_size<<endl;
  
  std::vector<uint8_t> buffer(blob_size);
  const diplomat::span<const uint8_t> blob(buffer.data(), buffer.size());
  const uint16_t* u16Ptr = reinterpret_cast<const uint16_t*>(str.getBuffer());
  
  diplomat::result<ICU4XDataProvider, ICU4XError> provider_result = ICU4XDataProvider::create_from_byte_slice(blob);
  
 const auto provider_result = ICU4XDataProvider::create_from_byte_slice(blob);

 const auto segmenter_auto =    ICU4XWordSegmenter::create_auto(provider_result.).ok().value();
 const ICU4XWordSegmenter* segmenters[] = {&segmenter_auto};

  size_t sizeInBytes = str.length() * sizeof(UChar);
  const size_t size = sizeInBytes / sizeof(uint16_t); // Compute the number of elements in the array
  diplomat::span<const uint16_t> span(u16Ptr,size);
  auto iterator = segmenters[0]->segment_utf16(span);
  int32_t breakpoint = iterator.next();
  int32_t breakpoint2 = iterator.next();
  cout<<breakpoint2;

blob_size is coming as -1 which is why the folllowing error is coming: libc++abi: terminating with uncaught exception of type std::length_error: vector

[riajain0412]

Okay, So Am I going in the correct direction of using create_from_byte_slice API in C++.

[Manishearth]

Yes, though you need to call .ok().value() on provider_result to get the actual provider before using it

[riajain0412]

Okay, Even after writing to buffer I get bad optional access error. Has anyone in your knowledge used data blob in C++? I'm having hard time loading data-blob in C++.

[sffc]

It looks to me that the code you've written in C++ to read a file to a buffer is not correct. Make sure you have a variable with the correct buffer before handing anything to ICU4X.

[riajain0412]

I've tried loading the blob data into buffer using C as well as C++ . Below is the new C++ code which I've written:

int main(){
  UnicodeString str ("ພາສາລາວ");
  printUnicodeString(str);
  cout<<endl;

//C++ code
  std::ifstream fin("my_data_blob", std::ifstream::ate | std::ifstream::binary);
  fin.seekg(0, std::ios::end);
  std::streamsize blob_size = fin.tellg();
  cout<<"size buffer "<<blob_size<<endl;
  fin.seekg(0, std::ios::beg);
  char* buffer = new char[blob_size];
  fin.read(buffer, blob_size);
  cout<<buffer<<endl;
  std::vector<uint8_t> f_buffer(blob_size);
  auto sample = reinterpret_cast<uint8_t*>(buffer);
  auto sample2 = reinterpret_cast<uint8_t*>(buffer + blob_size);
  f_buffer.insert(f_buffer.end(), reinterpret_cast<const uint8_t*>(buffer), reinterpret_cast<const uint8_t*>(buffer+ blob_size));

  const diplomat::span<const uint8_t> blob(f_buffer.data(), f_buffer.size());
  const auto provider_result = ICU4XDataProvider::create_from_byte_slice(blob).ok().value();

  const auto segmenter_auto = ICU4XWordSegmenter::create_auto(provider_result).ok().value();
}

But this code is giving bad optional access error. Maybe there is some issue in loading blob into buffer. But this is the only way which I found to read blob's data into buffer. Are you aware of how to load your data blob into buffer?