01 — gRPC and Protobuf

gRPC is the network language most large-scale C++ backends speak. Google, ByteDance, Databricks, Nvidia’s Triton, and every serious ML inference service you have heard of speaks either gRPC or something gRPC-shaped (Envoy xDS, Triton’s KServe interface, TensorFlow Serving). If your resume says “C++ engineer” and does not have a gRPC service on it, hiring managers assume you have only done homework problems. This file fixes that.

Protobuf is the wire format. gRPC is the RPC framework on top of Protobuf + HTTP/2. You will learn them together because in practice you never touch one without the other.

1. Protobuf: the schema you commit to disk

Protobuf 3 (proto3) is the version you write. Proto2 still exists in Google-internal code and old repos; you should recognize it (required/optional/default keywords) but not write new code in it.

A minimal .proto file:

syntax = "proto3";
package miniserve.v1;

message InferenceRequest {
  string model_name = 1;
  repeated float features = 2;
  string request_id = 3;
}

message InferenceResponse {
  repeated float logits = 1;
  int64 latency_us = 2;
}

service Inference {
  rpc Predict(InferenceRequest) returns (InferenceResponse);
  rpc PredictStream(stream InferenceRequest) returns (stream InferenceResponse);
}

Five things to internalize about this file:

  1. Field numbers are the contract. = 1, = 2 are wire tags. Never change or reuse them. Deleting a field means reserving its number (reserved 3;).

  2. Everything is optional in proto3. All scalar fields default to zero-values. “Was this actually set?” is a design problem you solve with wrapper types (google.protobuf.StringValue) or explicit has_x() bools.

  3. repeated is a RepeatedField<T> in C++, not std::vector. It supports move semantics and arenas.

  4. bytes is std::string. Yes, really. Bytes are stored as strings in the C++ API. This trips up everyone once.

  5. Package name maps to C++ namespace. package miniserve.v1; → miniserve::v1::InferenceRequest. Version your packages from day one; renaming later is painful.

2. Compiling and linking (CMake)

Use the modern CMake targets, not the legacy PROTOBUF_GENERATE_CPP macro:

find_package(Protobuf CONFIG REQUIRED)
find_package(gRPC CONFIG REQUIRED)

add_library(miniserve_proto
  proto/miniserve.proto)
target_link_libraries(miniserve_proto PUBLIC
  protobuf::libprotobuf gRPC::grpc++)

protobuf_generate(TARGET miniserve_proto LANGUAGE cpp)
protobuf_generate(TARGET miniserve_proto LANGUAGE grpc
  GENERATE_EXTENSIONS .grpc.pb.h .grpc.pb.cc
  PLUGIN "protoc-gen-grpc=\$<TARGET_FILE:gRPC::grpc_cpp_plugin>")

Install via vcpkg (vcpkg install grpc protobuf) or Homebrew on macOS (brew install grpc protobuf). Do not build gRPC from source unless you have a reason — the build tree is 45 minutes and 3GB.

3. The four RPC types

gRPC gives you four call shapes. You need one working example of each committed to your repo.

Unary (request → response)

The default. 90% of RPCs in most services are unary.

Status Predict(ServerContext* ctx,
               const InferenceRequest* req,
               InferenceResponse* resp) override {
  resp->mutable_logits()->Add(0.9f);
  resp->set_latency_us(120);
  return Status::OK;
}

Server streaming (request → stream of responses)

Use when the response is large or produced incrementally (log tailing, training progress).

Status StreamLogs(ServerContext* ctx, const LogRequest* req,
                  ServerWriter<LogLine>* writer) override {
  for (auto& line : tail(req->path())) {
    if (ctx->IsCancelled()) return Status::CANCELLED;
    writer->Write(line);
  }
  return Status::OK;
}

Client streaming (stream of requests → single response)

Use when many small inputs collapse into one summary (metric ingestion, file upload).

Bidirectional streaming (stream ↔ stream)

The interesting one for ML. This is how you do dynamic batching. Client keeps a persistent stream open, server accumulates N requests or waits T ms, runs one batch through the model, writes N responses back. Triton’s core loop is exactly this. Your P6.1 will implement it.

Status PredictStream(ServerContext* ctx,
                     ServerReaderWriter<InferenceResponse,
                                        InferenceRequest>* stream) override {
  InferenceRequest req;
  while (stream->Read(&req)) {
    // enqueue, batch, respond
    InferenceResponse resp = run_batched(req);
    stream->Write(resp);
  }
  return Status::OK;
}

4. Sync vs async — and why you must learn async

The synchronous API above is easy to write. It is also thread-per-RPC. On a 16-core box with 10K concurrent streams, you have 10K threads and a scheduler on fire. This does not scale.

Production gRPC C++ code uses one of two async APIs:

a) Completion queue API (older, more control, still used everywhere). You allocate a grpc::CompletionQueue, tag every async operation with a void*, and drive a loop:

void* tag; bool ok;
while (cq_.Next(&tag, &ok)) {
  auto* call = static_cast<CallData*>(tag);
  call->Proceed(ok);
}

Each CallData is a state machine (CREATE → PROCESS → FINISH). It is verbose. It is also what every gRPC internals talk on YouTube spends 40 minutes on. You must be able to read it. G-Research’s blog post on async streaming (“Lessons Learnt from Writing Asynchronous Streaming gRPC Services In C++”) is the canonical tour — read it before writing P6.1.

b) Callback API (newer, since gRPC 1.39, uses grpc::CallbackServerContext). Same performance, less boilerplate. Uses Reactor classes and OnDone/OnReadDone/OnWriteDone callbacks. For new code in 2026, use the callback API. For reading existing code, know the CQ API.

Your W37 exercise: implement one unary and one bidi service twice, once with each API. Feel the difference.

5. Deadlines, cancellation, and error status

The three details that separate real services from demos.

Deadlines. Every RPC should have a deadline. Client sets it:

ClientContext ctx;
ctx.set_deadline(std::chrono::system_clock::now() + 200ms);

Server checks it periodically with ctx->IsCancelled() in long-running or streaming handlers. If you skip this, one slow client blocks a worker forever.

Cancellation. Same signal as deadline expiry on the server side. In bidi streams, stream->Read() returns false when the peer cancels. Handle it — do not leak the batch you were accumulating.

Error status. Return a grpc::Status with a code from the enum: INVALID_ARGUMENT, NOT_FOUND, RESOURCE_EXHAUSTED, UNAVAILABLE, DEADLINE_EXCEEDED, INTERNAL. Do not stuff exceptions across the wire; do not use OK with error payloads. Codes are language-neutral and clients can retry on them.

6. TLS — non-negotiable for P6.1

Plain HTTP/2 gRPC is a demo. TLS-terminated gRPC is a service.

grpc::SslServerCredentialsOptions opts;
opts.pem_key_cert_pairs.push_back({key_pem, cert_pem});
auto creds = grpc::SslServerCredentials(opts);
ServerBuilder builder;
builder.AddListeningPort("0.0.0.0:9443", creds);

Generate certs with mkcert for local dev, Let’s Encrypt (via certbot) for the deployed VM. Do not commit private keys to git. Your P6.1 acceptance criterion explicitly requires TLS — this is why.

7. Interceptors, deadline propagation, and metadata

Briefly, because you will need these in P6.2:

  • Metadata = key/value string headers, like HTTP headers but on RPCs. Used for auth tokens, trace IDs, tenant IDs. Access with ctx->AddInitialMetadata and ctx->client_metadata().

  • Interceptors = middleware. Wrap every RPC to log, trace, or auth. OpenTelemetry’s gRPC instrumentation is exactly an interceptor.

  • Deadline propagation = when service A calls service B on behalf of a caller with 200ms budget, A should pass 200ms − elapsed to B. gRPC does not do this automatically; you must forward the deadline via metadata or explicit code.

8. Tooling you will actually use

  • grpcurl — curl for gRPC. Reflection-based. Test your services from the terminal.

  • ghz — gRPC load generator. Concurrency, RPS, latency percentiles. This is what you use for P6.1’s acceptance load test.

  • buf — Protobuf linter and breaking-change detector. Add to CI once you have >1 .proto file.

What most people get wrong

  • Reusing field numbers after deleting a field. Breaks every deployed client silently. Always reserved.

  • Not setting deadlines on the client side. A hung server can then hang your entire fleet.

  • Assuming Status::OK means “success” in streams. In server-streaming, OK just means the stream ended cleanly — the individual messages may still have carried error payloads if you designed the schema that way. Prefer standard status codes.

  • Writing sync-only gRPC and then load testing. Your throughput will look 20x worse than a comparable async service, and you will blame gRPC.