Configuration and File Discovery

Build configuration overview

Buf is configured through a buf.yaml file that should be checked in to the root of your repository. Buf will automatically read this file if present. Configuration can also be provided via the command-line flag --config, which accepts a path to a .json or .yaml file, or direct JSON or YAML data.

All Buf operations that use your local .proto files as Input rely on a valid build configuration. This configuration tells Buf where to search for .proto files, and how to handle imports. As opposed to protoc, where all .proto files are manually specified on the command-line, buf operates by recursively discovering all .proto files under configuration and building them.

The following is an example of all configuration options for build.

version: v1beta1
build:
roots:
- proto
- vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis
excludes:
- proto/foo/bar

The build section only has two options:

roots

This is a list of the directories that contain your .proto files. The directory paths must be relative to the root of your repository, and cannot point to a location outside of your repository. They also represent the root of your import paths within your .proto files.

For those familiar with protoc, roots corresponds to your --proto_paths, aliased as -I with protoc - that is, these are the directories that the compiler uses to search for imports.

As an example, if your repository has two files, proto/foo/bar/bar.proto and proto/foo/baz/baz.proto, we recommend having the following configuration:

version: v1beta1
build:
roots:
- proto

If baz.proto wants to import bar.proto, it does so relative to proto/:

// proto/foo/baz/baz.proto
syntax = "proto3";
package foo.baz;
import "foo/bar/bar.proto";

Generally, for a given Protobuf repository, we expect one main root that contains all your .proto files. For most repositories, we recommend this be a directory proto/. This is the pattern that the buf-example repository uses.

Other repositories that have no vendored .proto files may choose to use the root of the repository as the only root. This is what googleapis does, as an example - all imports derive from the root of the repository.

excludes

This is a list of the directories to ignore from file discovery. Any directories added to this list will be completely skipped and considered not part of the Protobuf schema. We do not recommend using this option in general, however in some situations it is unavoidable.

Root requirements

There are two additional requirements that buf imposes on your .proto file structure for compilation to succeed that are not enforced by protoc, both of which are very important for successful modern Protobuf development across a number of languages

1. Roots must not overlap, that is one root can not be a sub-directory of another root.

For example, the following is not a valid configuration:

version: v1beta1
# THIS IS INVALID AND WILL RESULT IN A PRE-COMPILATION ERROR
build:
roots:
- foo
- foo/bar

This is important to make sure that across all your .proto files, imports are consistent In the above example, for a given file foo/bar/bar.proto, it would be valid to import this file as either bar/bar.proto or bar.proto. Having inconsistent imports leads to a number of major issues across the Protobuf plugin ecosystem.

2. All .proto file paths must be unique relative to the roots.

For example, consider the following configuration:

version: v1beta1
build:
roots:
- foo
- bar

Given the above configuration, it is invalid to have the following two files:

  • foo/baz/baz.proto
  • bar/baz/baz.proto

This results in two files having the path baz/baz.proto. Given the following third file bar/baz/bat.proto:

// THIS IS DEMONSTRATING SOMETHING BAD
syntax = "proto3";
package bar.baz;
import "baz/baz.proto";

Which file is being imported? Is it foo/baz/baz.proto? bar/baz/baz.proto? The answer depends on the order of the -I flags given to protoc, or (if Buf didn't error in this scenario pre-compilation, which Buf does) the order of the imports given to the internal compiler. If the authors are being honest, we can't remember if it's the first -I or second -I that wins - we have outlawed this in our own builds for a long time.

While the above example is relatively contrived, the common error that comes up is when you have vendored .proto files. For example, grpc-gateway has it's own copy of the google.api definitions it needs. While these are usually in sync, the google.api schema can change. If we allowed the following:

version: v1beta1
# THIS IS INVALID AND WILL RESULT IN A PRE-COMPILATION ERROR
build:
roots:
- proto
- vendor/github.com/googleapis/googleapis
- vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis

Which copy of google/api/*.proto wins? The answer is no one wins, so this is not allowed.

Manual proto_path and file specification

In most scenarios, it is best to let Buf run file discovery on it's own - this will result in the least error-prone usage of Buf, and the most consistent results from the linter and breaking change detector. However, there are scenarios where you do not want Buf to search for .proto files, or you want to specify the entire build configuration on the command line, for example with Bazel integration.

Buf allows this through the --path flag and the various configuration flags on each command. The --path flag can be specified multiple times, and specifies the exact files or directories to use for Protobuf compilation. For every Buf command, configuration can also be passed on the command line.

As an example, let's say we want to replicate protoc's build behavior with buf build (described in more detail in the next section). Assume we have:

  • A single root, proto/
  • Two files, proto/foo/bar.proto and proto/foo/baz.proto

The following protoc commands:

# no imports or source code info
$ protoc \
-I proto \
-o image.bin \
proto/foo/bar.proto proto/foo/baz.proto
# with imports and source code info
$ protoc \
-I proto \
-o image.bin \
--include_imports \
--include_source_info \
proto/foo/bar.proto proto/foo/baz.proto

Can be replicated by the following buf commands:

# no imports or source code info
$ buf build \
--config '{"build":{"roots":["proto"]}}' \
--exclude-imports \
--exclude-source-info \
-o image.bin \
--path proto/foo/bar.proto --path proto/foo/baz.proto
# with imports and source code info
$ buf build \
--config '{"build":{"roots":["proto"]}}' \
-o image.bin \
--path proto/foo/bar.proto --path proto/foo/baz.proto