Protocol Buffers (Protobuf) have become a popular choice for microservices communication thanks to their compact binary encoding and language-agnostic design. But when your services evolve over time—adding new fields, renaming existing ones, or deprecating old fields—it’s critical to design your Protobuf schemas to be evolvable without breaking consumers.
This article will walk you through best practices for schema evolution, backward compatibility, optional fields, and versioning, so your microservices can grow safely.
Why Schema Evolution Matters
Unlike JSON, where unknown fields are typically ignored, Protobuf requires careful planning. Microservices might:
- Be updated at different times.
- Communicate across teams or organizational boundaries.
- Persist Protobuf-encoded data in databases or logs.
A careless change—like removing a field or reusing a tag number—can corrupt data or break consumers.
Reference:
Protocol Buffers Language Guide
Core Principles of Evolvable Protobuf
✅ 1. Field Numbers Are Forever
Each field has a unique numeric tag:
message User {
int32 id = 1;
string name = 2;
string email = 3;
}
Rule:
Never reuse or renumber fields, even if you delete them.
Instead, mark them as reserved:
message User {
int32 id = 1;
string name = 2;
reserved 3;
}
This prevents accidentally reassigning 3 to a different meaning.
✅ 2. Use Optional Fields
In proto3, all fields are optional by default. However, starting with proto3 version 3.12, you can explicitly declare fields as optional:
message Order {
int32 id = 1;
optional string comments = 2;
}
Benefits:
- The sender can omit the field.
- Receivers handle missing fields safely.
- You can distinguish between unset and set-to-default.
✅ 3. Avoid Changing Field Types
Changing a field’s type (e.g., from int32 to string) or its label (e.g., from repeated to optional) breaks binary compatibility.
✅ Do:
- Add new fields with new tag numbers.
- Keep the old field for backward compatibility.
✅ Avoid:
- Changing the type of an existing field.
- Changing
optionaltorepeatedor vice versa.
✅ 4. Use Reserved Tags and Names
When deprecating a field, always reserve both the tag number and the name:
message Product {
int32 id = 1;
string name = 2;
reserved 3, 4;
reserved "old_description", "legacy_field";
}
This prevents accidental reuse in the future.
✅ 5. Be Careful with Enums
Adding new enum values is safe.
Changing numeric values or removing existing ones is not safe.
Example:
enum Status {
UNKNOWN = 0;
PENDING = 1;
APPROVED = 2;
REJECTED = 3;
}
✅ You can add CANCELLED = 4.
❌ You should never change APPROVED from 2 to 4.
✅ 6. Prefer Singular Over Repeated Fields (When Possible)
Adding a repeated field later can cause deserialization problems if older clients expect a single value.
If you anticipate multiple values, use repeated from the start.
✅ 7. Use Wrapper Types for Nullability
Primitive types like int32 cannot distinguish between “unset” and “set to zero.”
If you need that distinction, use Google’s well-known wrapper types:
import "google/protobuf/wrappers.proto";
message User {
google.protobuf.Int32Value age = 1;
}
This allows:
ageunset (null).ageset to zero.ageset to other integers.
✅ 8. Version Your APIs Thoughtfully
Protobuf schemas themselves don’t require a v1, v2, etc. in the message name, but it can help avoid confusion:
message PaymentV1 { ... }
message PaymentV2 { ... }
Alternatives:
- Use package names for versioning:
package payments.v1;
Example: Evolving a Message Safely
Let’s walk through a realistic evolution.
Version 1:
message Invoice {
int32 id = 1;
double amount = 2;
}
Version 2:
- You add a due date.
- You deprecate the
amountfield (but don’t remove it).
message Invoice {
int32 id = 1;
reserved 2;
optional string due_date = 3;
double amount = 2 [deprecated = true];
}
Note:
deprecated = truesignals thatamountis obsolete, but still supported.reserved 2;prevents reusing the field number if you do remove it in the future.
Testing Compatibility
When evolving schemas:
- Test old clients reading new messages.
- Test new clients reading old messages.
- Use tools like:
- Buf Schema Registry for linting and breaking-change detection.
- Prototool for validation.
Best Practices Checklist
✅ Never reuse field numbers.
✅ Always reserve removed field numbers and names.
✅ Avoid changing field types or labels.
✅ Use optional and wrapper types for nullability.
✅ Version your APIs if major changes are unavoidable.
✅ Document your schema evolution policies.
✅ Automate compatibility checks in CI pipelines.
Resources
- Protocol Buffers Language Guide
- Buf Schema Registry
- Designing Robust APIs with Protocol Buffers
- Google Well-Known Types
Conclusion
Designing evolvable Protobuf schemas is essential for microservices to remain reliable and backward compatible over time. By following these best practices, you’ll avoid common pitfalls and confidently iterate your APIs without breaking clients.
