Workspaces: organize multiple modules

Workspaces are collections of modules that enable you to iterate on modules and manage cross-module dependencies without having to commit to the Buf Schema Registry (BSR). They also allow you to more easily do buf operations like breaking change detection and linting across related modules.

When iterating on related modules without workspaces, you can get into a dependency loop where you have to push modules that are dependencies to the BSR, and then update the local dependencies of the modules that rely on them. This is both frustrating and prone to error.

With workspaces, you define all of the related modules in a simple configuration file, and they can locally vendor each other as you iterate without involving the BSR. And for all local buf operations, the workspace is treated as the input without the need to specify each module.

If you're familiar with protoc, a workspace is similar to specifying multiple include -I paths, but with the added consistency of defining dependencies in version-controlled config files.

Configuration

A workspace requires at least one module, which is defined by a buf.yaml file. The workspace config file, buf.work.yaml, is generally one level above the module directories, often at the root of a VCS. Below is a complete example of a workspace that includes a Pets API and a Payments API, where the Pets API is importing the Payments API. It contains a buf.work.yaml configuration file and a buf.yaml configuration file, and shows our recommended directory structure.

The buf.work.yaml file lists the directories of the modules it includes, and the buf.yaml files define the dependencies between modules.

.
├── buf.work.yaml
├── paymentapis
│   ├── acme
│   │   └── payment
│   │       └── v2
│   │           └── payment.proto
│   └── buf.yaml
└── petapis
    ├── acme
    │   └── pet
    │       └── v1
    │           └── pet.proto
    └── buf.yaml

version: v1
directories:
  - paymentapis
  - petapis

version: v1
name: buf.build/acme/petapis
deps:
  - buf.build/acme/paymentapis

See the buf.work.yaml config file reference for more information about its fields.

Additional requirements

The Buf CLI imposes two additional requirements on your .proto file structure for compilation to succeed, both of which are essential to successful modern Protobuf development across a number of languages.

1. Workspace modules must not overlap. A workspace module can't be a sub-directory of another workspace module.

This, for example, isn't a valid configuration:

version: v1 # THIS IS INVALID AND RESULTS IN A PRE-COMPILATION ERROR
  directories:
    - foo
    - foo/bar

Following this rule ensures that imports are consistent across all your .proto files. Without it, in the above example a file foo/bar/bar.proto could be imported as either bar/bar.proto or bar.proto. Having inconsistent imports leads to a number of major issues across the Protobuf plugin ecosystem, so we don't allow it.

2. All .proto file paths must be unique relative to each workspace module.

Consider this configuration:

version: v1
directories:
  - foo
  - bar

Given the above configuration, it's invalid to have these two files:

• foo/baz/baz.proto
• bar/baz/baz.proto

because it results in two files having the path baz/baz.proto. If you add the following file to the mix, the issue becomes apparent:

// THIS IS DEMONSTRATING SOMETHING BAD
syntax = "proto3";

package bar.baz;

import "baz/baz.proto";

Which file is being imported, foo/baz/baz.proto or bar/baz/baz.proto? With protoc the answer depends on the order of the -I flags. The Buf CLI errors out pre-compilation instead, alerting you to the issue. Though the above example is relatively contrived, vendoring .proto files is a common practice that can cause this situation.

Importing across modules

In a workspace, imports are resolved relative to each module's root, or the placement of the buf.yaml file. For the example directory structure shown above, paymentapis/acme/payment/v2/payment.proto is included in the workspace as acme/payment/v2/payment.proto and the petapis/acme/pet/v1/pet.proto file imports it like this:

import "acme/payment/v2/payment.proto";

message PurchasePetRequest {
  string pet_id = 1;
  acme.payment.v2.Order order = 2;
}

Multiple-module operations

If the input for a buf command is a directory containing a buf.work.yaml file, the command acts upon all of the modules defined in the buf.work.yaml.

For example, suppose that we update both the paymentapis and petapis directories with some lint failures, such as using a camel case field name. We can easily lint all of the modules defined in a buf.work.yaml with a single command:

$ ls
Output
buf.work.yaml  paymentapis  petapis

$ buf lint
Output
paymentapis/acme/payment/v2/payment.proto:29:10:Field name "recipientID" should be lower_snake_case, such as "recipient_id".
petapis/acme/pet/v1/pet.proto:51:27:Field name "orderV2" should be lower_snake_case, such as "order_v2".

Interaction with module cache

As mentioned above, workspaces enable you to work on multiple modules in parallel, such as introducing a new Protobuf message in one module and depending on it in another.

Without a workspace, the Buf CLI relies on the module's buf.lock manifest to read its dependencies from the local module cache. This requires that you push changes that create new dependencies to the BSR and run buf mod update in the module that requires them before they can be used.

With a workspace, the module cache is only used for dependencies not defined in the workspace. For all directories listed in the buf.work.yaml file, the workspace overrides the module cache and allows you to use the new changes without pushing and updating.

Pushing modules from workspaces

It's important to note that workspaces only apply to local operations. Though you don't need to define modules in the deps section of the buf.yaml file to use them locally, you do need to do so before pushing modules to the BSR.

version: v1
name: buf.build/acme/petapis
deps:
  - buf.build/acme/paymentapis

A current limitation of workspaces is that each module needs to be pushed to the BSR independently in dependency order, starting with the leaf modules. As you push each module, run the buf mod update command in the next downstream module to update its dependencies, and continue to push each of your modules until all of your local changes are published to the BSR.

We're working on ways to streamline this workflow and the configuration files in general, so keep an eye out for updates.