Skip to main content

Overview

Modules#

A module is a collection of Protobuf files that are configured, built, and versioned as a logical unit. By moving away from individual .proto files, the module simplifies file discovery and eliminates the need for complex build scripts to -I include, exclude, and configure your Protobuf sources.

BSR module
How modules map to Buf YAML configs

Storing modules in the BSR, a Protobuf-aware registry, protects you from publishing broken builds. Module consumers have confidence that the modules that they pull compile, something that isn't possible with traditional version control systems.

The module's name uniquely identifies and gives ownership to a collection of Protobuf files, which means you can push modules to authorized repositories within the BSR, add hosted modules as dependencies, consume modules as part of code generation, and much more.

A module is identified by a name key in the buf.yaml file, which is placed at the root of the Protobuf source files it defines. This tells buf where to search for .proto files, and how to handle imports. Unlike protoc, where you manually specify .proto files, buf recursively discovers all .proto files under configuration to build the module.

buf.yaml
version: v1name: buf.build/acme/weather

The module name is composed of three parts: the remote, owner, and repository:

Module name syntax
(buf.build)/{owner}/{repository}
Examplebuf.build/acme/weather
Legend:(default){variable}
  • Remote: The DNS name for the server hosting the BSR. This is always buf.build.

  • Owner: An entity that is either a user or organization within the BSR ecosystem.

  • Repository: Stores all versions of a single module

    While roughly analogous to Git repositories, a Buf repository is only a remote locationthere is no concept of a repository "clone" or "fork". Repositories do not exist in multiple locations.

    Every repository is identified by its module name, allowing it to be imported by other modules and uniquely identified within the BSR.

Many organizations with public Protobuf files are already using the BSR, and some of the bigger ones are officially maintained by Buf. These include

Documentation#

Every push to the BSR generates documentation. You may browse the documentation section of a repository by navigating to the Docs tab.

For more information, see Generated documentation.

Dependencies#

A module can declare dependencies on other modules, which is configured in the deps key of your buf.yaml. You can add dependencies by adding their module name to the deps list. For example:

buf.yaml
version: v1name: buf.build/acme/weatherdeps:  - buf.build/acme/units

Although we do not recommend it, in some situations you may need to pin a module to a specific version. Ideally, authors keeps modules backwards-compatible and avoid breaking changes so you can always rely on the latest version.

deps:  - buf.build/acme/units:1c473ad9220a49bca9320f4cc690eba5

Once a dependency is added to the configuration file, you need to run:

$ buf mod update

This updates all your deps to their latest version and is captured in a buf.lock file.

You can now import the Protobuf types just like you would if you had the files locally:

acme/weather/v1/weather.proto
package acme.weather.v1;
import "acme/units/v1/unit.proto";
message Forecast {  string description = 1;  acme.units.Degree temperature = 2;}

The buf CLI automatically resolves the module(s) specified in the deps list.

See the Usage section for a detailed example.

Referencing a module#

A module has different versions. Each version includes any number of changes, and each change is identifiable by a unique commit or tag. The collective set of module versions is housed within a repository.

Commit: Every push of new content to a repository is associated with a commit that identifies that change in the schema. The commit is created after a successful push. This means that unlike Git, the commit only exists on the BSR repository and not locally.

Tag: A reference to a single commit but with a human readable name, similar to a Git tag. It is useful for identifying commonly referenced commitslike a release.

Local modules with workspaces#

If you want to depend on local modules, you can set up a workspace to discover modules through your file system. If you are in a workspace, buf looks for deps in your workspace configuration before attempting to find it on the BSR.

This makes workspaces a good way to iterate on multiple modules at the same time before pushing any changes to the BSR.

For an in-depth example check out the Tour - Use a Workspace

Module cache#

buf caches files it downloads as part of module resolution in a folder on the local filesystem to avoid incurring the cost of downloading modules repeatedly. To choose where to cache the files, it checks these, in order:

  • The value of $BUF_CACHE_DIR, if set.
  • The value of $XDG_CACHE_HOME falling back to $HOME/.cache on Linux and Mac and %LocalAppData% for Windows.

Code generation#

Hosting modules on the BSR means anyone with proper access can consume those modules. This solves the need to coordinate and sync Protobuf files manually amongst multiple consumers, which is error prone and quite often leads to drift.

Instead, users bring their own plugins and generate code from a single source of truth: a hosted module on the BSR. This is especially useful when consuming a Protobuf-based API that requires a client SDK for your language(s) of choice.

In your buf.gen.yaml define plugins and their respective options, and then generate your code with the buf CLI by referencing a BSR module:

$ buf generate buf.build/acme/weather

See the Usage section for a detailed example.

Although beyond the scope of this overview, we suggest taking a look at managed mode as it relates to code generation. Historically, consumer concerns are conflated with producer concerns due to hardcoding of language-specific options in Protobuf files, which in turn restricts their usefulness to consumers. Managed mode addresses existing limitations and offers a better separation of concerns.