Skip to content
This repository has been archived by the owner on Jul 7, 2022. It is now read-only.

Adding preliminary support for converting json schema to apache parquet schema. #19

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

Adding preliminary support for converting json schema to apache parquet schema. #19

wants to merge 2 commits into from

Commits on Oct 9, 2020

  1. Adding preliminary support for converting json schema to apache parqu… 

    …et schema.

Related to issue: pennsignals#2
Objective: hoping to get advice about approach.

Simple test:
$ aptos convert -format parquet tests/schema/person
message ParquetSchema {
  Person {
    required binary firstName (UTF8);
    optional binary lastName (UTF8);
    required int32 age;
    Address {
      optional binary street_name (UTF8);
      optional int32 zipcode;
    }
  }
}
    lalade committed Oct 9, 2020
    Configuration menu
    Copy the full SHA
    024a503 View commit details
    Browse the repository at this point in the history

  2. Fixing object field name to contain definition and type sub-fields. 

    Sample run:
$ aptos convert -format parquet tests/schema/person
message ParquetSchema {
  optional group Person {
    required binary firstName (UTF8);
    optional binary lastName (UTF8);
    required int32 age;
    optional group Address {
      optional binary street_name (UTF8);
      optional int32 zipcode;
    }
  }
}
    lalade committed Oct 9, 2020
    Configuration menu
    Copy the full SHA
    6ec8cfe View commit details
    Browse the repository at this point in the history