A collection of easy-to-use and extensible commands to be used in your xtask CLI based on clap.
We rely on these commands in each of our Tracel repositories. By centralizing our redundant commands we save a big amount of code duplication, boilerplate and considerably lower their maintenance cost. This also provides a unified interface across all of our repositories.
These commands are not specific to Tracel repositories and they should be pretty much usable in any Rust repositories with a cargo workspace as well as other repositories where Rust is not necessarily the only language. The commands can be easily extended using handy proc macros and by following some patterns described in this README.
- Create a new Cargo workspace:
cargo new my_workspace
cd my_workspace- Create the
xtaskbinary crate:
cargo new xtask --bin- Configure the workspace:
Edit the Cargo.toml file in the root of the workspace to include the following:
[workspace]
members = ["xtask"]- Add the
tracel-xtaskdependency:
In the xtask/Cargo.toml file, add the following under [dependencies]:
[dependencies]
tracel-xtask = "2.0"- Build the workspace:
cargo buildYour workspace is now set up with a xtask binary crate that depends on tracel-xtask version 2.0.x.
- In the
main.rsfile of the newly createdxtaskcrate, import thetracel_xtaskprelude module and then declare aCommandenum. Select the base commands you want to use by adding themacros::base_commandsattribute:
use tracel_xtask::prelude::*;
#[macros::base_commands(
Build,
Check,
Fix
Test,
)]
pub enum Command {}- Update the
mainfunction to initialize xtask and dispatch the base commands:
fn main() -> anyhow::Result<()> {
let args = init_xtask::<Command>(parse_args::<Command>()?)?;
match args.command {
// dispatch_base_commands function is generated by the commands macro
_ => dispatch_base_commands(args),
}
}-
Build the workspace with
cargo buildat the root of the repository to verify that everything is. -
You should now be able to display the main help screen which lists the commands you selected previously:
cargo xtask --helpInvoking the xtask binary with cargo is very verbose and not really usable as is. Happily we can create a cargo alias to make it really effortless to invoke it.
Create a new file .cargo/config.toml in your repository with the following contents:
[alias]
xtask = "run --target-dir target/xtask --package xtask --bin xtask --"This saves quite a few characters to type as you can now invoke xtask directly like this:
cargo xtaskTry it with cargo xtask --help.
We can save even more typing by creating a shell alias for cargo xtask.
For instance we can set the alias to cx. Here is how to do it in various shells.
- For bash:
nano ~/.bashrc
# add this to the file
alias cx='cargo xtask'
# save and source the file or restart the shell session
source ~/.bashrc- For zsh:
nano ~/.zshrc
# add this to the file
alias cx='cargo xtask'
# save and source the file or restart the shell session
source ~/.zshrc- For fish:
nano ~/.config/fish/config.fish
# add this to the file
alias cx='cargo xtask'
# save and source the file or restart the shell session
source ~/.config/fish/config.fish- For powershell:
notepad $PROFILE
# add this at the end of file
function cx {
cargo xtask $args
}
# save and quit then open a new powershell terminalTry it with cx --help at the root of the repository.
All our repositories follow the same directory hierarchy:
- a
cratesdirectory which contains all the crates of the workspace - an
examplesdirectory which holds all the examples crates - a
xtaskdirectory which is the binary crate for our xtask CLI usingtracel-xtask
As per Cargo convention, Integration tests are tests contained in a tests directory of a crate besides its src directory.
Inline tests in src directory are called Unit tests.
tracel-xtask allows to easily execute them separately using the test command.
There are 4 default targets provided by tracel-xtask:
workspacewhich targets the cargo workspace, this is the default targetcratesare all the binary crates and library cratesexamplesare all the example cratesall-packagesare bothcratesandexamplestargets
workspace and all-packages are different because workspace uses the --workspace flag of cargo whereas all-packages
relies on crates and examples targets which use the --package flag. So all-packages executes a command for each crate
or example individually.
Here are some examples:
# run all the crates tests
cargo xtask test --target crates all
# check format for examples, binaries and libs
cargo xtask check --target all-packages unit
# build the workspace
cargo xtask build --target workspace
# workspace is the default target so this has the same effect
cargo xtask buildThe following options are global and precede the actual command on the command line.
-e, --environment
cargo xtask -e production buildIts main role is to inform your custom commands or dispatch functions about the targeted environment which can be:
dev(default) for development,testfor test,stagfor staging,prodfor production.
It also automatically loads the following environment variables files if they exists in the working directory:
.envfor any set environment,.env.{environment}(example:.env.dev) for the non-sensitive configuration,.env.{environment}.secrets(example.env.dev.secrets) for the sensitive configuration like password. These files must be added to the ignore file of your VCS tool (for git add.env.*.secretsto the.gitignorefile at the root of your repository).
This parameter is passed to all the handle_command function as env.
-c, --context
cargo xtask --context no-std buildThis argument has no effect in the base commands. Its purpose is to provide additional context to your custom commands
or dispatch functions — for example, to indicate whether the current build context is std or no-std.
This parameter is passed to all the handle_command function as context.
--enable-coverage
It setups the Rust toolchain to generate coverage information.
We use the derive API of clap which is based on structs, enums and attribute proc macros. Each base command is a
submodule of the base_commands module. If the command accepts arguments there is a corresponding struct named <command>CmdArgs
which declare the options, arguments and subcommands. In the case of subcommands a corresponding enum named <command>SubCommand
is defined.
Here is an example with a foo command:
#[macros::declare_command_args(Target, FooSubCommand)]
struct FooCmdArgs {}
pub enum FooSubCommand {
/// A sub command for foo (usage on the command line: cargo xtask foo print-something)
PrintSomething,
}Note that it is possible to have an arbitrary level of nested subcommands but deeper nested subcommands cannot be extended, in other words, only the first level of subcommands can be extended. If possible, try to design commands with only one level of subcommands to keep the interface simple.
In the following sections we will see how to create completely new commands as well how to extend existing base commands.
-
First, we organize commands by creating a
commandsmodule. Create a filextask/src/commands/mycommand.rsas well as the correspondingmod.rsfile to declare the module contents. -
Then, in
mycommand.rsdefine the arguments struct with thedeclare_command_argsmacro and define thehandle_commandfunction. Thedeclare_command_argsmacro takes two parameters, the first is the type of the target enum and the second is the type of the subcommand enum if any. If the command has no target or no subcommand then putNonefor each argument respectively.Targetis the default target type provided bytracel-xtask. This type can be extended to support more targets as we will see in a later section.
use tracel_xtask::prelude::*;
#[macros::declare_command_args(Target, None)]
struct MyCommandCmdArgs {}
pub fn handle_command(_args: MyCommandCmdArgs) -> anyhow::Result<()> {
println!("Hello from my-command");
Ok(())
}- Make sure to update the
mod.rsfile to declare the command module:
pub(crate) mod my_command;
- We can now add a new variant to the
Commandenum inmain.rs:
mod commands;
use tracel_xtask::prelude::*;
#[macros::base_commands(
Bump,
Check,
Fix,
Test,
)]
pub enum Command {
MyCommand(commands::mycommand::MyCommandCmdArgs),
}- And dispatch its handling to our new command module:
fn main() -> anyhow::Result<()> {
let args = init_xtask::<Command>()?;
match args.command {
Command::NewCommand(args) => commands::new_command::handle_command(args),
_ => dispatch_base_commands(args),
}
}- You can now test your new command with:
cargo xtask my-command --help
cargo xtask my-commandLet's implement a new command called extended-target to illustrate how to extend the default Target enum.
-
Create a
commands/extended_target.rsfile and update themod.rsfile as we saw in the previous section. -
We also need to add a new
strumdependency to ourCargo.tomlfile:
[dependencies]
strum = {version = "0.26.3", features = ["derive"]}- Then we can extend the
Targetenum with themacros::extend_targetsattribute in ourextended_target.rsfile. Here we choose to add a new target calledfrontendwhich targets the frontend component we could find for instance in a monorepo:
use tracel_xtask::prelude::*;
#[macros::extend_targets]
pub enum MyTarget {
/// Target the frontend component of the monorepo.
Frontend,
}- Then we define our command arguments by referencing our newly created
MyTargetenum in thedeclare_command_argsattribute:
#[macros::declare_command_args(MyTarget, None)]
struct ExtendedTargetCmdArgs {}- Our new target is then available for use in the
handle_commandfunction:
pub fn handle_command(args: ExtendedTargetCmdArgs) -> anyhow::Result<()> {
match args.target {
// Default targets
MyTarget::AllPackages => println!("You chose the target: all-packages"),
MyTarget::Crates => println!("You chose the target: crates"),
MyTarget::Examples => println!("You chose the target: examples"),
MyTarget::Workspace => println!("You chose the target: workspace"),
// Additional target
MyTarget::Frontend => println!("You chose the target: frontend"),
};
Ok(())
}- Register our new command the usual way by adding it to our
Commandenum and dispatch it in themainfunction:
mod commands;
use tracel_xtask::prelude::*;
#[macros::base_commands(
Bump,
Check,
Fix,
Test,
)]
pub enum Command {
ExtendedTarget(commands::extended_target::ExtendedTargetCmdArgs),
}
fn main() -> anyhow::Result<()> {
let args = init_xtask::<Command>()?;
match args.command {
Command::ExtendedTarget(args) => commands::extended_target::handle_command(args),
_ => dispatch_base_commands(args),
}
}- Test the command with:
cargo xtask extended-target --help
cargo xtask extended-target --target frontendTo extend an existing command we use the macros::extend_command_args attribute which takes three parameters:
- first argument is the type of the base command arguments struct to extend,
- second argument is the target type (or
Noneif there is no target), - third argument is the subcommand type (or
Noneif there is no subcommand).
Let's use two examples to illustrate this, the first is a command to extend the build base command with
a new --debug argument; and the second is a new command to extend the subcommands of the check base command
to add a new my-check subcommand.
Note that you can find more examples in the xtask crate of this repository.
We create a new command called extended-build-args which will have an additional argument called --debug.
-
Create the
commands/extended_build_args.rsfile and update themod.rsfile as we saw in the previous section. -
Extend the
BuildCommandArgsstruct using the attributemacros::extend_command_argsand define thehandle_commandfunction. Note that the macro automatically implements theTryIntotrait which makes it easy to dispatch back to the base command ownhandle_commandfunction. Also note that if the base command requires a target then you need to provide a target as well in your extension, i.e. the target parameter of the macro cannot beNoneif the base command has aTarget.
use tracel_xtask::prelude::*;
#[macros::extend_command_args(BuildCmdArgs, Target, None)]
pub struct ExtendedBuildArgsCmdArgs {
/// Print additional debug info when set
#[arg(short, long)]
pub debug: bool,
}
pub fn handle_command(args: ExtendedBuildArgsCmdArgs) -> anyhow::Result<()> {
if args.debug {
println!("Debug is enabled");
}
base_commands::build::handle_command(args.try_into().unwrap())
}- Register the new command the usual way by adding it to the
Commandenum and dispatch it in themainfunction:
mod commands;
use tracel_xtask::prelude::*;
#[macros::base_commands(
Bump,
Check,
Fix,
Test,
)]
pub enum Command {
ExtendedBuildArgs(commands::extended_build_args::ExtendedBuildArgsCmdArgs),
}
fn main() -> anyhow::Result<()> {
let args = init_xtask::<Command>()?;
match args.command {
Command::ExtendedBuildArgs(args) => commands::extended_build_args::handle_command(args),
_ => dispatch_base_commands(args),
}
}- Test the command with:
cargo xtask extended-build-args --help
cargo xtask extended-build-args --debugFor this one we create a new command called extended-check-subcommands which will have an additional subcommand.
-
Create a
commands/extended_check_subcommands.rsfile and update themod.rsfile as we saw in the previous section. -
Extend the
CheckCommandArgsstruct using the attributemacros::extend_command_args:
use tracel_xtask::prelude::*;
#[macros::extend_command_args(CheckCmdArgs, Target, ExtendedCheckSubcommand)]
pub struct ExtendedCheckedArgsCmdArgs {}- Implement the
ExtendedCheckSubcommandenum by extending theCheckSubcommandbase enum with the macroextend_subcommands. It takes the name of the type of the subcommand enum to extend:
#[macros::extend_subcommands(CheckSubCommand)]
pub enum ExtendedCheckSubcommand {
/// An additional subcommand for our extended check command.
MySubcommand,
}- Implement the
handle_commandfunction to handle the new subcommand. Note that we must handle theAllsubcommand as well:
use strum::IntoEnumIterator;
pub fn handle_command(args: ExtendedCheckedArgsCmdArgs) -> anyhow::Result<()> {
match args.get_command() {
ExtendedCheckSubcommand::MySubcommand => run_my_subcommand(args.clone()),
ExtendedCheckSubcommand::All => {
ExtendedCheckSubcommand::iter()
.filter(|c| *c != ExtendedCheckSubcommand::All)
.try_for_each(|c| {
handle_command(
ExtendedCheckedArgsCmdArgs {
command: Some(c),
target: args.target.clone(),
exclude: args.exclude.clone(),
only: args.only.clone(),
},
)
})
}
_ => base_commands::check::handle_command(args.try_into().unwrap()),
}
}
fn run_my_subcommand(_args: ExtendedCheckedArgsCmdArgs) -> Result<(), anyhow::Error> {
println!("Executing new subcommand");
Ok(())
}- Register the new command the usual way by adding it to the
Commandenum and dispatch it in themainfunction:
mod commands;
use tracel_xtask::prelude::*;
#[macros::base_commands(
Bump,
Check,
Fix,
Test,
)]
pub enum Command {
ExtendedCheckSubcommand(commands::extended_check_subcommands::ExtendedCheckedArgsCmdArgs),
}
fn main() -> anyhow::Result<()> {
let args = init_xtask::<Command>()?;
match args.command {
Command::ExtendedCheckSubcommand(args) => commands::extended_check_subcommands::handle_command(args),
_ => dispatch_base_commands(args),
}
}- Test the command with:
cargo xtask extended-check-subcommands --help
cargo xtask extended-check-subcommands my-checktracel-xtask provides helper functions to easily execute custom builds or tests with specific features or build targets (do not confuse
Rust build targets which is an argument of the cargo build command with the xtask target we introduced previously).
For instance we can extend the build command to build additional crates with custom features or build targets using the helper function:
pub fn handle_command(mut args: tracel_xtask::commands::build::BuildCmdArgs) -> anyhow::Result<()> {
// regular execution of the build command
tracel_xtask::commands::build::handle_command(args)?;
// additional crate builds
// build 'my-crate' with all the features
tracel_xtask::utils::helpers::custom_crates_build(vec!["my-crate"], vec!["--all-features"], None, None, "all features")?;
// build 'my-crate' with specific features
tracel_xtask::utils::helpers::custom_crates_build(vec!["my-crate"], vec!["--features", "myfeature1,myfeature2"], None, None, "myfeature1,myfeature2")?;
// build 'my-crate' with a different target than the default one
tracel_xtask::utils::helpers::custom_crates_build(vec!["my-crate"], vec!["--target", "thumbv7m-none-eabi"], None, None, "thumbv7m-none-eabi target")?;
Ok(())
}Here is a example GitHub job which shows how to setup coverage, enable it and upload coverage information to codecov:
env:
GRCOV_LINK: "https://github.com/mozilla/grcov/releases/download"
GRCOV_VERSION: "0.8.19"
jobs:
my-job:
runs-on: ubuntu-22.04
steps:
- name: Checkout
uses: actions/checkout@v4
- name: install rust
uses: dtolnay/rust-toolchain@master
with:
components: rustfmt, clippy
toolchain: stable
- name: Install grcov
shell: bash
run: |
curl -L "$GRCOV_LINK/v$GRCOV_VERSION/grcov-x86_64-unknown-linux-musl.tar.bz2" |
tar xj -C $HOME/.cargo/bin
cargo xtask coverage install
- name: Build
shell: bash
run: cargo xtask build
- name: Tests
shell: bash
run: cargo xtask --enable-coverage test all
- name: Generate lcov.info
shell: bash
# /* is to exclude std library code coverage from analysis
run: cargo xtask coverage generate --ignore "/*,xtask/*,examples/*"
- name: Codecov upload lcov.info
uses: codecov/codecov-action@v4
with:
files: lcov.info
token: ${{ secrets.CODECOV_TOKEN }}By convention this command is responsible to run all the checks, builds, and/or tests that validate the code before opening a pull request or merge request.
The command Validate can been added via the macro tracel_xtask_macros::commands like the other commands.
By default all the checks from the check command are run as well as both unit and integration tests from
the test command.
You can make your own handle_command function if you need to perform more validations. Ideally this function
should only call the other commands handle_command functions.
For quick reference here is a simple example to perform all checks and tests against the workspace:
pub fn handle_command(args: ValidateCmdArgs) -> anyhow::Result<()> {
let target = Target::Workspace;
let exclude = vec![];
let only = vec![];
// checks
[
CheckSubCommand::Audit,
CheckSubCommand::Format,
CheckSubCommand::Lint,
CheckSubCommand::Typos,
]
.iter()
.try_for_each(|c| {
super::check::handle_command(CheckCmdArgs {
target: target.clone(),
exclude: exclude.clone(),
only: only.clone(),
command: Some(c.clone()),
ignore_audit: args.ignore_audit,
})
})?;
// tests
super::test::handle_command(TestCmdArgs {
target: target.clone(),
exclude: exclude.clone(),
only: only.clone(),
threads: None,
jobs: None,
command: Some(TestSubCommand::All),
})?;
Ok(())
}The check and fix commands are designed to help you maintain code quality during development.
They run various checks and fix issues, ensuring that your code is clean and follows best practices.
check and fix contains the same subcommands to audit, format, lint or proofread a code base.
While the check command only reports issues, the fix command attempts to fix them as they are encountered.
Each check can be executed separately or all of them can be executed sequentially using all.
Usage to lint the code base:
cargo xtask check lint
cargo xtask fix lint
cargo xtask fix allTesting is a crucial part of development, and the test command is designed to make this process easy.
This command makes the distinction between unit tests and integrations tests. Unit tests are inline tests under the
src directory of a crate. Integration tests are tests defined in files under the tests directory of a crate besides
the src directory.
Usage:
# execute workspace unit tests
cargo xtask test unit
# execute workspace integration tests
cargo xtask test integration
# execute workspace both unit tests and integration tests
cargo xtask test allNote that documentation tests are supported by the doc command.
Command to build and test the documentation in a workspace.
This is a command reserved for repository maintainers.
The bump command is used to update the version numbers of all first-party crates in the repository.
This is particularly useful when you're preparing for a new release and need to ensure that all crates have the correct version.
You can bump the version by major, minor, or patch levels, depending on the changes made. For example, if you’ve made breaking changes, you should bump the major version. For new features that are backwards compatible, bump the minor version. For bug fixes, bump the patch version.
Usage:
cargo xtask bump <SUBCOMMAND>This is a command reserved for repository maintainers and is typically used in publish GitHub workflows.
This command automates the process of publishing crates to crates.io, the Rust package registry.
By specifying the name of the crate, xtask handles the publication process, ensuring that the crate is available for others to use.
Usage:
cargo xtask publish <NAME>As mentioned, this command is often used in a GitHub workflow. We provide a Tracel's reusable publish-crate workflow that makes use of this command. Here is a simple example with a workflow that publishes two crates A and B with A depending on B.
name: publish all crates
on:
push:
tags:
- "v*"
jobs:
publish-B:
uses: tracel-ai/github-actions/.github/workflows/publish-crate.yml@v1
with:
crate: B
secrets:
CRATES_IO_API_TOKEN: ${{ secrets.CRATES_IO_API_TOKEN }}
# --------------------------------------------------------------------------------
publish-A:
uses: tracel-ai/github-actions/.github/workflows/publish-crate.yml@v1
with:
crate: A
needs:
- publish-B
secrets:
CRATES_IO_API_TOKEN: ${{ secrets.CRATES_IO_API_TOKEN }}This command provide a subcommand to install the necessary dependencies for performing code coverage and a subcommand to generate the
coverage info file that can then be uploaded to a service provider like codecov. See dedicated section Enable and generate coverage information.
The docker command provides up and down commands to start and stop stacks. The command is integrated with the environment
configuration mechanism of tracel-xtask.
The name of the compose file must follow the template docker-compose.{env}.yml with env being the shorthand environment name.
For instance for the development environment the file is named docker-compose.dev.yml.
The command also requires a mandatory project name for the stack in order to have idempotent up commands.
Various additional subcommands about dependencies.
deny make sure that all dependencies meet requirements using cargo-deny.
unused detects dependencies in the workspace that are not in ussed.
This command makes it easier to execute sanitizers as described in the Rust unstable book.
These sanitizers require a nightly toolchain.
Run the specified vulnerability check locally. These commands must be called with 'cargo +nightly'
Usage: xtask vulnerabilities <COMMAND>
Commands:
all Run all most useful vulnerability checks
address-sanitizer Run Address sanitizer (memory error detector)
control-flow-integrity Run LLVM Control Flow Integrity (CFI) (provides forward-edge control flow protection)
hw-address-sanitizer Run newer variant of Address sanitizer (memory error detector similar to AddressSanitizer, but based on partial hardware assistance)
kernel-control-flow-integrity Run Kernel LLVM Control Flow Integrity (KCFI) (provides forward-edge control flow protection for operating systems kerneljs)
leak-sanitizer Run Leak sanitizer (run-time memory leak detector)
memory-sanitizer Run memory sanitizer (detector of uninitialized reads)
mem-tag-sanitizer Run another address sanitizer (like AddressSanitizer and HardwareAddressSanitizer but with lower overhead suitable for use as hardening for production binaries)
nightly-checks Run nightly-only checks through cargo-careful `<https://crates.io/crates/cargo-careful>`
safe-stack Run SafeStack check (provides backward-edge control flow protection by separating stack into safe and unsafe regions)
shadow-call-stack Run ShadowCall check (provides backward-edge control flow protection - aarch64 only)
thread-sanitizer Run Thread sanitizer (data race detector)
help Print this message or the help of the given subcommand(s)
tracel-xtask gives access to two useful macros register_cleanup and handle_cleanup to easily define some cleanup functions to be executed at
a given time during the program as well as whenever the user presses CTRL+c.
It is very useful to guard processes while executing some tests and make sure that the state is still cleaned up even if the program is interrupted by
the user.
Since the cleanup handler is stored in a static variable, it's drop function is not automatically called when the program exits normally and the handle_cleanup macro needs to be called manually. However, when the program is interrupted by the user with CTRL+c, the cleanup is automatically called.
Example:
Register cleanup functions in your commands, say you have a customized test command that spins up some container.
pub(crate) async fn handle_command(
args: TestCmdArgs,
env: Environment,
ctx: Context,
) -> anyhow::Result<()> {
match args.get_command() {
TestSubCommand::Integration => {
// spin up containers
// ...
// register cleanup command for them
register_cleanup!("Integration tests: Docker compose stack", move || {
base_commands::docker::handle_command(
DockerCmdArgs {
build: false,
project: super::DOCKER_COMPOSE_PROJECT_NAME.to_string(),
command: Some(DockerSubCommand::Down),
services: vec![],
},
Environment::Test,
ctx.clone(),
)
.expect("Should be able to stop docker compose stack");
});
// Execute xtask test base command
base_commands::test::run_integration(&cmd_args.target, &cmd_args)
}
TestSubCommand::All => { ... }
}
}Then call the handle_cleanup macro at the end of your main function to force a cleanup:
use tracel_xtask::prelude::*;
#[macros::base_commands(
Build,
Check,
Fix,
Test
)]
pub enum Command {}
fn main() -> anyhow::Result<()> {
let args = init_xtask::<Command>(parse_args::<Command>()?)?;
match args.command {
Command::Test(cmd_args) => {
commands::test::handle_command(cmd_args, args.environment, args.context).await
}
_ => dispatch_base_commands(args),
}
handle_cleanup!();
}