Using Protocol Buffer

In the repository, I demo how to use protocol buffers. The protocol buffers is an efficient storing approach. It basically consists of three parts.

• The protocol is defined in a .proto file.
• The file stores the data in the binary format.
• The script can be generated by the tool protoc for different targets, like Python, Java, C++, etc.

It is a standard defined and released by Google. It is common to compare between protocol buffers (protobuf) and JSON for the reason of transmission and storing data.

Items	Protobuf	JSON
Usage	Most for gRPC	HTTP API
Content	Binary	Human Readable
Contract	Necessary (.proto)	Optional (e.g. OpenAPI)
Prescriptiveness	Constraint	Loose

How to start

The protobuf

For the definition, you can simply create a file with extension .proto. The following content is an example.

syntax = "proto2";

package tutorial;

message Person {
  optional string name = 1;
  optional int32 id = 2;
  optional string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    optional string number = 1;
    optional PhoneType type = 2 [default = HOME];
  }

  repeated PhoneNumber phones = 4;
}

message AddressBook {
  repeated Person people = 1;
}

• The version of protobuf is necessary, and it is defined by syntax.
• Use message to define the data object.
• Use enum to define the similiar type.
• Use repeated to note this field can be repeated many times. The optional represents the field can be set or not. The required means the field is necessary. Notice that the reqired fields are necessary even if you edit the protocol definition.
• The number is the identity for the attribute, it should not be resued while editing the protocol file.

Generating the script

Next you can generate the script or code based on your platforms, like Python, C++, etc. In the first you have to install the protobuf compiler. Generally, you can install it by one of the following approaches.

• Building from the scratch.

sudo apt-get install libprotobuf-dev

git clone https://github.com/google/protobuf
cd protobuf
./autogen.sh
./configure
make check
sudo make install
sudo ldconfig

• Installing the pre-compiled version

sudo apt update
sudo apt install protobuf-compiler

After that, you can check the installation by protoc --version. If the installation is complete, you can now generate the script or code. Here I will demo generating the Python script.

protoc --python_out=. *.proto

After that, you will get the *.py script. You can use them in your script like import *.py in Python.

Generating the binary file

After you import the module generated, you can manipulate it to generate the binary file. Before generating the binary file, you have to install necessary Python packages.

python3 -m virtualenv -p python3 env
source ./en/bin/activate
pip3 install --no-cache-dir -r ./requirements.txt

You now can start programming.

# for example
import addressbook_pb2

You can edit the main.py and run it. The basic flow is like the following script.

def AddPerson(address_book_path):
  # use the pre-generated Python script
  ADDRESS_BOOK = addressbook_pb2.AddressBook()

  try:
    # read the existing address book serialized data
    with open(address_book_path, "rb") as fin:
      ADDRESS_BOOK.ParseFromString(fin.read())
  except Exception as err:
    print("Failed in loading the address book {}.".format(address_book_path))
    print("Can't load the address book. Create a new now.")

  PromptForPerson(ADDRESS_BOOK.people.add())

  # write the serialized data to the local file
  with open(address_book_path, "wb") as fout:
    fout.write(ADDRESS_BOOK.SerializeToString())

After programming, you can run the command to generate the binary file.

"""
Usage:
	python3 main.py <binary_file_path> <operation>

	operation: add|list
"""

# add the data
python3 main.py address_book add

# retrieve the data
python3 main.py address_book list