Andreas Farre: How to make Firefox builds1 17% faster2

In the , which lets you write custom wrappers for programs that aren’t compilers in the traditional sense. With is small. In dom/bindings/Makefile.in, we now conditionally pass $(CCACHE) as a command wrapper to the py_action call:

WEBIDL_CCACHE=

ifdef MOZ_USING_BUILDCACHE

WEBIDL_CCACHE=$(CCACHE)

endif



webidl.stub: $(codegen_dependencies)

	$(call py_action,webidl $(relativesrcdir),$(srcdir),,$(WEBIDL_CCACHE))

	@$(TOUCH) $@

The py_action macro in config/makefiles/functions.mk is what runs Python build actions. The ability to pass a command wrapper as a fourth argument was also introduced in this bug. When buildcache is configured as the compiler cache, this means the webidl action is invoked as buildcache python3 -m mozbuild.action.webidl ... instead of just python3 -m mozbuild.action.webidl .... That’s all buildcache needs to intercept it.

Note the ifdef MOZ_USING_BUILDCACHE guard. This is specific to buildcache because ccache and sccache don’t have a mechanism for caching arbitrary commands. Buildcache does, through its Lua wrappers.

The Lua wrapper

Buildcache’s Lua plugin system lets you write a script that tells it how to handle a program it doesn’t natively understand. The wrapper for WebIDL codegen, , bringing the total down to 1m12s ^{repo and point buildcache at it via lua_paths in ~/.buildcache/config.json:}

{

"lua_paths": ["/path/to/buildcache-wrappers/mozilla"],

"max_cache_size": 10737418240,

"max_local_entry_size": 2684354560

}

Alternatively, you can set the BUILDCACHE_LUA_PATH environment variable. A convenient place to do that is in your mozconfig:

mk_add_options "export BUILDCACHE_LUA_PATH=/path/to/buildcache-wrappers/mozilla/"

The large max_local_entry_size (2.5 GB) is needed because some Rust crates produce very large cache entries.

What’s next

The Lua plugin system is the interesting part here. The WebIDL wrapper is a proof of concept, but the same technique applies to any deterministic build step that takes known inputs and produces known outputs. There are other codegen actions in the Firefox build that could get the same treatment, and I plan to explore those next.

Notes