Use protoc input instead of the internal compiler

Buf uses a self-contained compiler that is verified to produce exactly equivalent output to protoc. This compiler has a number of advantages over using protoc separately, and is tested against thousands of Protobuf definitions including Protobuf grammar edge cases that almost all users will never hit. However, we don't expect you to take our word for it - we wouldn't trust that statement ourselves without our own verification.

Buf is able to use a variety of Inputs, one of which is binary Images. Since Images are FileDescriptorSets (and vice versa), Buf is able to use output from protoc as input.

Let's return to our lint example. Comment out the ENUM_NO_ALLOW_ALIAS and IMPORT_NO_PUBLIC exceptions from your buf.yaml file.

version: v1beta1
lint:
use:
- BASIC
- FILE_LOWER_SNAKE_CASE
except:
#- ENUM_NO_ALLOW_ALIAS
#- IMPORT_NO_PUBLIC
- PACKAGE_AFFINITY
- PACKAGE_DIRECTORY_MATCH
- PACKAGE_SAME_DIRECTORY
breaking:
use:
- WIRE_JSON

Run the following:

$ protoc -I . --include_source_info $(find . -name '*.proto') -o /dev/stdout 2>/dev/null | buf check lint -
google/appengine/v1/app_yaml.proto:53:5:Enum option "allow_alias" on enum "ErrorCode" must be false.
google/appengine/v1/app_yaml.proto:264:3:Enum option "allow_alias" on enum "SecurityLevel" must be false.
google/cloud/policytroubleshooter/v1/checker.proto:20:1:Import "google/cloud/policytroubleshooter/v1/explanations.proto" must not be public.
google/cloud/recommendationengine/v1beta1/catalog.proto:132:5:Enum option "allow_alias" on enum "StockState" must be false.
google/cloud/securitycenter/v1/securitycenter_service.proto:19:1:Import "google/cloud/securitycenter/v1/run_asset_discovery_response.proto" must not be public.
google/cloud/securitycenter/v1p1beta1/securitycenter_service.proto:20:1:Import "google/cloud/securitycenter/v1p1beta1/run_asset_discovery_response.proto" must not be public.
google/cloud/websecurityscanner/v1beta/scan_config_error.proto:35:5:Enum option "allow_alias" on enum "Code" must be false.
google/storage/v1/storage.proto:1902:5:Enum option "allow_alias" on enum "Values" must be false.

Note we added 2>/dev/null to suppress the unused import warnings. In general, you should not do this, as both warnings and build errors are printed to stderr from protoc, and there is no way to suppress warnings. We recommend doing this in two steps instead - first, create a .bin file, then, use this .bin file as the input for buf check lint.

This is the equivalent of our buf check lint example earlier in the tour. What this does:

  • Runs protoc to build a FileDescriptorSet with SourceCodeInfo for all Protobuf files in the repository.
  • Outputs this FileDescriptorSet to stdout.
  • Uses this FileDescriptorSet as a binary Image file inputted on stdin for buf check lint.

Note we had to include SourceCodeInfo to have line and column references. If we do not, the following is outputted, and any comment checks will not work:

$ protoc -I . $(find . -name '*.proto') -o /dev/stdout 2>/dev/null | buf check lint -
google/appengine/v1/app_yaml.proto:1:1:Enum option "allow_alias" on enum "ErrorCode" must be false.
google/appengine/v1/app_yaml.proto:1:1:Enum option "allow_alias" on enum "SecurityLevel" must be false.
google/cloud/policytroubleshooter/v1/checker.proto:1:1:Import "google/cloud/policytroubleshooter/v1/explanations.proto" must not be public.
google/cloud/recommendationengine/v1beta1/catalog.proto:1:1:Enum option "allow_alias" on enum "StockState" must be false.
google/cloud/securitycenter/v1/securitycenter_service.proto:1:1:Import "google/cloud/securitycenter/v1/run_asset_discovery_response.proto" must not be public.
google/cloud/securitycenter/v1p1beta1/securitycenter_service.proto:1:1:Import "google/cloud/securitycenter/v1p1beta1/run_asset_discovery_response.proto" must not be public.
google/cloud/websecurityscanner/v1beta/scan_config_error.proto:1:1:Enum option "allow_alias" on enum "Code" must be false.
google/storage/v1/storage.proto:1:1:Enum option "allow_alias" on enum "Values" must be false.

As per the Inputs documentation, Buf will still look for a buf.yaml file in your current directory. However, you can override this to be a different JSON or YAML file, or direct JSON or YAML input.

The below avoids any file I/O by buf other than reading the Image from stdin:

$ protoc -I . --include_source_info $(find . -name '*.proto') -o /dev/stdout 2>/dev/null | buf check lint --config '{"lint":{"use":["ENUM_NO_ALLOW_ALIAS"]}}' -
google/appengine/v1/app_yaml.proto:53:5:Enum option "allow_alias" on enum "ErrorCode" must be false.
google/appengine/v1/app_yaml.proto:264:3:Enum option "allow_alias" on enum "SecurityLevel" must be false.
google/cloud/recommendationengine/v1beta1/catalog.proto:132:5:Enum option "allow_alias" on enum "StockState" must be false.
google/cloud/websecurityscanner/v1beta/scan_config_error.proto:35:5:Enum option "allow_alias" on enum "Code" must be false.
google/storage/v1/storage.proto:1902:5:Enum option "allow_alias" on enum "Values" must be false.

Using protoc to build is a perfectly fine way to use Buf, and may be preferable in situations where you already are using protoc as part of your build environment, such as for repositories managed by Bazel. However, you do have the following disadvantages:

  • Buf does additional checks to verify your Protobuf layout is sane, namely it does not allow overlapping proto_paths, and makes sure all file paths are unique relative to the proto paths. This is an extremely important check to perform in most scenarios.
  • Buf utilizes all available cores to complete compilation, and will generally be much faster, although Protobuf compilation is generally very quick regardless. As a comparison on a system with four cores:
$ time protoc -I . $(find . -name '*.proto') -o /dev/null 2>/dev/null
real 0m4.345s
user 0m4.158s
sys 0m0.187s
# Buf reverses the flags --include-imports and --include-source-info to exclude so
# we exclude them here for equivalent performance testing to protoc.
$ time buf build --exclude-imports --exclude-source-info
real 0m0.794s
user 0m4.347s
sys 0m0.356s
  • Buf cannot compute relative or absolute paths for lint or breaking change output when Images are used as input. For example, if you ran buf check lint with different directory inputs, you get equivalent output paths, however if using Images, you always get the paths relative to the proto paths.
# The below commands assume ENUM_NO_ALLOW_ALIAS and IMPORT_NO_PUBLIC are commented out from our example.
$ cd ..
$ buf check lint googleapis
googleapis/google/appengine/v1/app_yaml.proto:53:5:Enum option "allow_alias" on enum "ErrorCode" must be false.
googleapis/google/appengine/v1/app_yaml.proto:264:3:Enum option "allow_alias" on enum "SecurityLevel" must be false.
googleapis/google/cloud/policytroubleshooter/v1/checker.proto:20:1:Import "google/cloud/policytroubleshooter/v1/explanations.proto" must not be public.
googleapis/google/cloud/recommendationengine/v1beta1/catalog.proto:132:5:Enum option "allow_alias" on enum "StockState" must be false.
googleapis/google/cloud/securitycenter/v1/securitycenter_service.proto:19:1:Import "google/cloud/securitycenter/v1/run_asset_discovery_response.proto" must not be public.
googleapis/google/cloud/securitycenter/v1p1beta1/securitycenter_service.proto:20:1:Import "google/cloud/securitycenter/v1p1beta1/run_asset_discovery_response.proto" must not be public.
googleapis/google/cloud/websecurityscanner/v1beta/scan_config_error.proto:35:5:Enum option "allow_alias" on enum "Code" must be false.
googleapis/google/storage/v1/storage.proto:1902:5:Enum option "allow_alias" on enum "Values" must be false.
$ buf check lint $(pwd)/googleapis
/home/foo/googleapis/google/appengine/v1/app_yaml.proto:53:5:Enum option "allow_alias" on enum "ErrorCode" must be false.
/home/foo/googleapis/google/appengine/v1/app_yaml.proto:264:3:Enum option "allow_alias" on enum "SecurityLevel" must be false.
/home/foo/googleapis/google/cloud/policytroubleshooter/v1/checker.proto:20:1:Import "google/cloud/policytroubleshooter/v1/explanations.proto" must not be public.
/home/foo/googleapis/google/cloud/recommendationengine/v1beta1/catalog.proto:132:5:Enum option "allow_alias" on enum "StockState" must be false.
/home/foo/googleapis/google/cloud/securitycenter/v1/securitycenter_service.proto:19:1:Import "google/cloud/securitycenter/v1/run_asset_discovery_response.proto" must not be public.
/home/foo/googleapis/google/cloud/securitycenter/v1p1beta1/securitycenter_service.proto:20:1:Import "google/cloud/securitycenter/v1p1beta1/run_asset_discovery_response.proto" must not be public.
/home/foo/googleapis/google/cloud/websecurityscanner/v1beta/scan_config_error.proto:35:5:Enum option "allow_alias" on enum "Code" must be false.
/home/foo/googleapis/google/storage/v1/storage.proto:1902:5:Enum option "allow_alias" on enum "Values" must be false.
$ protoc -I googleapis --include_source_info $(find googleapis -name '*.proto') -o /dev/stdout 2>/dev/null | buf check lint --config googleapis/buf.yaml -
google/appengine/v1/app_yaml.proto:53:5:Enum option "allow_alias" on enum "ErrorCode" must be false.
google/appengine/v1/app_yaml.proto:264:3:Enum option "allow_alias" on enum "SecurityLevel" must be false.
google/cloud/policytroubleshooter/v1/checker.proto:20:1:Import "google/cloud/policytroubleshooter/v1/explanations.proto" must not be public.
google/cloud/recommendationengine/v1beta1/catalog.proto:132:5:Enum option "allow_alias" on enum "StockState" must be false.
google/cloud/securitycenter/v1/securitycenter_service.proto:19:1:Import "google/cloud/securitycenter/v1/run_asset_discovery_response.proto" must not be public.
google/cloud/securitycenter/v1p1beta1/securitycenter_service.proto:20:1:Import "google/cloud/securitycenter/v1p1beta1/run_asset_discovery_response.proto" must not be public.
google/cloud/websecurityscanner/v1beta/scan_config_error.proto:35:5:Enum option "allow_alias" on enum "Code" must be false.
google/storage/v1/storage.proto:1902:5:Enum option "allow_alias" on enum "Values" must be false.