Protocol Buffers

lastmod 2026-01-09created 2015-10-0640 edits

Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

Why Use Protocol Buffers

The example we're going to use is a very simple address book application that can read and write people's contact details to and from a file. Each person in the address book has a name, an ID, an email address, and a contact phone number. How do you serialize and retrieve structured data like this? There are a few ways to solve this problem.

The raw in-memory data structures can be stored or sent in binary form. Over time, this is a fragile approach, as the receiving/reading code must be compiled with exactly the same memory layout. Also, as files accumulate data in the raw format and copies of software that are wired for that format are spread around, it's very hard to extend the format.
You can invent an ad-hoc way to encode the data items into a single string – such as encoding 4 ints as "12:3-23:67". This is a simple and flexible approach, although it does require writing one-off encoding and parsing code, and the parsing imposes a small run-time cost. This works best for encoding very simple data.
Serialize the data to XML. This approach can be very attractive since XML is (sort of) human readable and there are binding libraries for lots of languages. This can be a good choice if you want to share data with other applications/projects. However, XML is notoriously space intensive, and encoding/decoding it can impose a huge performance penalty on applications. Also, navigating an XML DOM tree is considerably more complicated than navigating simple fields in a class normally would be.

Protocol buffers are the flexible, efficient, automated solution to solve exactly this problem. With protocol buffers, you write a .proto description of the data structure you wish to store. From that, the protocol buffer compiler creates a class that implements automatic encoding and parsing of the protocol buffer data with an efficient binary format. The generated class provides getters and setters for the fields that make up a protocol buffer and takes care of the details of reading and writing the protocol buffer as a unit. Importantly, the protocol buffer format supports the idea of extending the format over time in such a way that the code can still read data encoded with the old format.

How to install

$ ./autogen.sh
$ ./configure
$ make
$ make check
$ make install

protoc

Executable for building.

protoc -I$INCLUDE ... $PROTO_FILES

Define syntax

A variable is defined as follows.

한정자 데이터타입 변수명 = 순서 [default = 기본값];

Qualifier

required

A value for the field must be provided, otherwise the message will be considered "uninitialized". If libprotobuf is compiled in debug mode, serializing an uninitialized message will cause an assertion failure. In optimized builds, the check is skipped and the message will be written anyway. However, parsing an uninitialized message will always fail (by returning false from the parse method). Other than this, a required field behaves exactly like an optional field.

optional

The field may or may not be set. If an optional field value isn't set, a default value is used. For simple types, you can specify your own default value, as we've done for the phone number type in the example. Otherwise, a system default is used: zero for numeric types, the empty string for strings, false for bools. For embedded messages, the default value is always the "default instance" or "prototype" of the message, which has none of its fields set. Calling the accessor to get the value of an optional (or required) field which has not been explicitly set always returns that field's default value.

repeated

The field may be repeated any number of times (including zero). The order of the repeated values will be preserved in the protocol buffer. Think of repeated fields as dynamically sized arrays.

oneof

syntax = "proto3";

package ai.eam.dataops.rsync;
option go_package = "gen/protobuf/ai/eam/dataops/rsync";

import "eamai/dataops/image_data.proto";
import "eamai/dataops/indy_data.proto";
import "eamai/dataops/pointcloud_data.proto";
import "eamai/job.proto";

message DownloadResponse {
  oneof request {
    ai.eam.dataops.imagedata.UploadRequest image_data = 1;
    ai.eam.dataops.pointclouddata.UploadRequest pointcloud_data = 2;
    ai.eam.dataops.indydata.UploadRequest indy_data = 3;
    ai.eam.StartJobRequest start_job = 4;
    ai.eam.FinishJobRequest finish_job = 5;
  }
}

Protocol Buffers Well-Known Types

Protocol Buffers Well-Known Types | Protocol Buffers Documentation

You need an additional import as follows.

import "google/protobuf/timestamp.proto"

Any (message)
Api (message)
BoolValue (message)
BytesValue (message)
DoubleValue (message)
Duration (message)
Empty (message)
Enum (message)
EnumValue (message)
Field (message)
Field.Cardinality (enum)
Field.Kind (enum)
FieldMask (message)
FloatValue (message)
Int32Value (message)
Int64Value (message)
ListValue (message)
Method (message)
Mixin (message)
NullValue (enum)
Option (message)
SourceContext (message)
StringValue (message)
Struct (message)
Syntax (enum)
Timestamp (message)
Type (message)
UInt32Value (message)
UInt64Value (message)
Value (message)

Example

message Contents {
    required string name = 1;
    optional int32 file_size = 2 [default = 0];
    optional int32 remark = 3;

    enum ContentsType {
        MOVIE = 0;
        TEXT = 1;
        IMAGE = 2;
        MUSIC = 3;
    }
    optional ContentsType type = 4 [default = TEXT];
}

Method options

Things you put in the body of an RPC function.

HTTP annotations (most common)

Used for REST API mapping in gRPC-Gateway or Google Cloud APIs:

import "google/api/annotations.proto";

service Greeter {
  rpc SayHello (HelloRequest) returns (HelloReply) {
    option (google.api.http) = {
      post: "/v1/hello"  // HTTP POST /v1/hello로 매핑
      body: "*"          // 전체 request body 사용
    };
  }

  rpc GetUser (GetUserRequest) returns (User) {
    option (google.api.http) = {
      get: "/v1/users/{user_id}"  // URL 패스 파라미터
    };
  }
}

Marking as deprecated

service Greeter {
  rpc OldMethod (Request) returns (Response) {
    option deprecated = true;  // 이 메서드는 더 이상 사용하지 않음을 표시
  }
}

Custom options

import "google/protobuf/descriptor.proto";

extend google.protobuf.MethodOptions {
  string my_custom_option = 50000;  // 커스텀 옵션 정의
}

service Greeter {
  rpc SayHello (HelloRequest) returns (HelloReply) {
    option (my_custom_option) = "custom_value";  // 커스텀 옵션 사용
  }
}

Idempotency (idempotency) setting (Google Cloud APIs)

service Greeter {
  rpc CreateUser (CreateUserRequest) returns (User) {
    option idempotency_level = IDEMPOTENT;  // 멱등성 보장
  }
}

Troubleshooting

Python import error

The following error may occur in Python.

ImportError: No module named google.protobuf

In this case, install the Protobuf package for Python as follows.

$ sudo pip install protobuf

Couldn't build proto file into descriptor pool: duplicate file name

../../../.pyenv/versions/opy-your-3.9.11/lib/python3.9/site-packages/proj/proto/daemon/daemon_api_pb2.py:19: in <module>
    DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'...')
E   TypeError: Couldn't build proto file into descriptor pool: duplicate file name (daemon_api.proto)

A .proto with the same name was built. Find it and remove the duplicate.

wrong type for method ProtoReflect

Running staticcheck in go printed the following error:

-: # main
main.go:249:13: cannot use pbMsg.(interface{ProtoReflect()}) (comma, ok expression of type interface{ProtoReflect()}) as "google.golang.org/protobuf/proto".Message value in argument to protojson.MarshalOptions{…}.Marshal: interface{ProtoReflect()} does not implement protoreflect.ProtoMessage (wrong type for method ProtoReflect)
                have ProtoReflect()
                want ProtoReflect() protoreflect.Message (compile)

First check the import:

import "google.golang.org/protobuf/proto"

Code like the following:

if pbMsg, ok := obj.(interface {
  ProtoReflect() interface{}
}); ok {
  jsonBytes, err = protojson.MarshalOptions{
    Multiline:       true,
    Indent:          "  ",
  }
  // ...
}

was changed as follows:

if pbMsg, ok := obj.(proto.Message); ok {
  jsonBytes, err = protojson.MarshalOptions{
    Multiline:       true,
    Indent:          "  ",
  }
  // ...
}

protoc-gen-mypy: program not found or is not executable

protoc-gen-mypy: program not found or is not executable

Please specify a program using absolute path or make sure the program is available in your PATH system variable

--mypy_out: protoc-gen-mypy: Plugin failed with status code 1.

This error occurs when protoc tries to generate Python type hints (.pyi files) from .proto files but cannot find the plugin program protoc-gen-mypy responsible for that task.

First install the mypy-protobuf package:

pip install mypy-protobuf

Check (or fix) the PATH to locate the protoc-gen-mypy executable.

If you want to specify it directly on the command line:

# protoc-gen-mypy의 위치를 먼저 찾습니다.
# 예: /usr/local/bin/protoc-gen-mypy 또는 ~/.local/bin/protoc-gen-mypy
protoc --plugin=protoc-gen-mypy=/실제/경로/protoc-gen-mypy \
       --python_out=. \
       --mypy_out=. \
       your_file.proto

Local Download

Protocol Buffers v3.0.0-beta-2: Protobuf-3.0.0-beta-2.tar.gz

Protocol Buffers v3.2.0 (2017-02-08): Protobuf-3.2.0.tar.gz

Favorite site

Documentation

What is ProtoBuf

Guide

Google Protocol Buffers basic usage
Google Protocol Buffer (Google Protocl Buffer) ¹
[Recommended] Protocol Buffer Basics: C++ ²
Using Protocol Buffers as packets

References

Knight76.tistory.com_-_googleProtocolBuffer.pdf ↩
Protocol_Buffer_Basics_Cpp_-_Notes.pdf ↩

author Zer0 sitemap