Resolving Python Import Failures in Bazel with `proto_library` Targets

TL;DR:


If Python imports fail when a package depends on multiple proto_library targets, it's because Bazel generates non-namespace packages by default.

Add the following flag to your .bazelrc file to resolve this issue:

build --incompatible_default_to_explicit_init_py

At some point in the future it might be enabled by default

Problem Overview

While working with Bazel to build Python code from Protocol Buffers, I encountered a perplexing runtime error:

• Importing the first package in PYTHONPATH works, but subsequent imports from other proto_library targets fail.


• Reversing the dependency order changes which package is successfully imported.

This issue arises because Bazel-generated Python packages for Protocol Buffers are not namespace packages by default. When two packages share the same top-level path (e.g., proto.common), Python loads only the first one it encounters, masking the others.

Environment and Setup

We're using Bazel with bzlmod in our (auxillis.ai) monorepo, which has the following structure:

proto/

   common/

      some_lib/

         some_lib.proto

         BUILD.bazel

   service/

       some_service/

            some_service.proto

            BUILD.bazel

OurMODULE.bazel file includes dependencies for Python and Protocol Buffers:

# Protobuf

bazel_dep(name = "protobuf", version = "27.3", repo_name = "com_google_protobuf")

bazel_dep(name = "rules_proto", version = "6.0.2")

bazel_dep(name = "rules_proto_grpc_python", version = "5.0.0")



# Python

PYTHON_VERSION = "3.12.4"

bazel_dep(name = "rules_python", version = "0.35.0")

python = use_extension("@rules_python//python/extensions:python.bzl", "python")

python.toolchain(python_version = PYTHON_VERSION, is_default = True)

Reproducing the Issue

The following example shows how the issue manifests. If you run the main.py with bazel run, the imports would fail.

BUILD.bazel for common/some_lib

proto_library(

    name = "auxillis_ai_common_some_lib_proto",

    srcs = ["some_lib.proto"],

    visibility = ["//visibility:public"],

    deps = ["@com_google_protobuf//:struct_proto"],

)



python_proto_library(

    name = "py",

    protos = [":auxillis_ai_common_some_lib_proto"],

    visibility = ["//visibility:public"],

)

BUILD.bazel for service/some_service

proto_library(

    name = "auxillis_ai_service_some_service_proto",

    srcs = ["some_service.proto"],

    visibility = ["//visibility:public"],

    deps = [

        "//proto/common/some_lib:auxillis_ai_common_some_lib_proto",

        "@com_google_protobuf//:any_proto",

    ],

)



python_grpc_library(

    name = "py",

    protos = [":auxillis_ai_service_some_service_proto"],

    visibility = ["//visibility:public"],

)

Python Script (main.py)

import os



def main():

    python_path = os.getenv("PYTHONPATH")



    python_path = python_path.split(":")

    common_prefix = os.path.commonprefix(python_path)

    print("Common prefix: ", common_prefix)

    for path in python_path:

        print(path.replace(common_prefix + "/", ""))

    print("-------------------")

    try:

        from proto.common.some_lib import some_lib_pb2

        from proto.service.some_service import some_service_pb_2, some_service_pb_2_grpc

        print(some_lib_pb2)

        print(some_service_pb_2)

        print(some_service_pb_2_grpc)

    except ImportError:

        raise 



if __name__ == "__main__":

    main()

BUILD.bazel for main.py

py_binary(

    name = "main",

    srcs = ["main.py"],

    deps = [

        "//proto/common/some_lib:py",

        "//proto/service/some_service:py",

        "@pip//grpcio",

    ],

)

Root Cause

Inspecting the PYTHONPATH environment variable reveals that Bazel adds generated Python packages to PYTHONPATH in the order specified in the deps argument. However:

1. Each proto_library generates a directory with an __init__.py file, making it a regular Python package.


2. When Python loads a package (e.g., proto.common.some_lib), any other packages sharing the same namespace are masked.

Solution

To fix this, you need to configure Bazel to generate namespace packages. Add the following to your .bazelrc:

build --incompatible_default_to_explicit_init_py

This ensures Bazel creates namespace packages, allowing multiple directories to contribute to the same Python package hierarchy. After adding the flag, all imports work as expected.

Key Takeaways

• Bazel’s default behavior for Python Protocol Buffer packages can lead to runtime import issues when dependencies overlap.


• Adding --incompatible_default_to_explicit_init_py makes Bazel generate namespace packages, resolving the issue.


• Use flag if your monorepo has shared Protocol Buffer dependencies.