Skip to content

Fix: Resolve protobuf generation bloat with sparse mode filtering #1525

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Naq302 wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Fix: Resolve protobuf generation bloat with sparse mode filtering #1525

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
13c02d5
fix: improve THORChain genMsgs.sh script robustness and error handling
Naq302 Aug 26, 2025
2b64e59
Merge branch 'master' into fix/improve-genmsg-scripts
Naq302 Sep 5, 2025
e0cc58c
Updates to scripts and reverted protos
Naq302 Sep 5, 2025
8e0eead
Merge remote-tracking branch 'origin/master' into fix/improve-genmsg-…
Naq302 Sep 5, 2025
a079d62
resolve protobuf generation bloat with sparse mode filtering
Naq302 Sep 5, 2025
cb614fe
Code Rabbit Feedback
Naq302 Sep 5, 2025
1ca3399
Reduced clone time and adding back in coin for thorchain. wip
Naq302 Sep 5, 2025
4f9cad6
Fix THORChain protobuf bloat using minimal common.proto
Naq302 Sep 5, 2025
4d2ddf2
CR feedback
Naq302 Sep 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 75 additions & 9 deletions packages/xchain-mayachain/genMsgs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,31 +1,97 @@
#!/bin/bash

# this script checks out mayanode master and generates the proto3 typescript buindings for MsgDeposit and MsgSend
# This script updates MAYAChain Protobuf bindings for MsgDeposit and MsgSend
set -e # Exit on any error

MSG_COMPILED_OUTPUTFILE=src/types/proto/MsgCompiled.js
MSG_COMPILED_TYPES_OUTPUTFILE=src/types/proto/MsgCompiled.d.ts

TMP_DIR=$(mktemp -d)
# Using local minimal proto files - no need to clone mayanode repository

# Download cosmos/base/v1beta1/coin.proto from cosmossdk
COSMOS_COIN_PROTO="proto/cosmos/base/v1beta1/coin.proto"
if [ ! -f "$COSMOS_COIN_PROTO" ]; then
tput setaf 2
echo "Downloading cosmos/base/v1beta1/coin.proto from cosmossdk"
tput sgr0
mkdir -p "proto/cosmos/base/v1beta1"
if ! curl -f -o "$COSMOS_COIN_PROTO" \
"https://raw.githubusercontent.com/cosmos/cosmos-sdk/main/proto/cosmos/base/v1beta1/coin.proto"; then
echo "Error: Failed to download cosmos coin.proto"
exit 1
fi
echo "✓ Downloaded cosmos coin.proto"
else
echo "✓ cosmos coin.proto already exists"
fi

# Verify our minimal proto files exist
tput setaf 2
echo "Checking out https://gitlab.com/mayachain/mayanode to $TMP_DIR"
echo "Checking minimal proto files"
tput sgr0
(cd $TMP_DIR && git clone https://gitlab.com/mayachain/mayanode)
MISSING_FILES=0
for proto_file in \
"proto/common/minimal_common.proto" \
"proto/types/minimal_msg_deposit.proto" \
"proto/types/minimal_msg_send.proto" \
"$COSMOS_COIN_PROTO"; do
if [ ! -f "$proto_file" ]; then
echo "Error: $(basename "$proto_file") missing"
MISSING_FILES=1
else
echo "✓ $(basename "$proto_file") found"
fi
done

if [ $MISSING_FILES -eq 1 ]; then
echo "Error: Required proto files are missing"
exit 1
fi

# Generate msgs
# Generate Protobuf JS bindings using minimal proto files to prevent over-inclusion
tput setaf 2
echo "Generating $MSG_COMPILED_OUTPUTFILE"
tput sgr0
yarn run pbjs -w commonjs -t static-module $TMP_DIR/mayanode/proto/mayachain/v1/common/common.proto $TMP_DIR/mayanode/proto/mayachain/v1/x/mayachain/types/msg_deposit.proto $TMP_DIR/mayanode/proto/mayachain/v1/x/mayachain/types/msg_send.proto $TMP_DIR/mayanode/third_party/proto/cosmos/base/v1beta1/coin.proto -o $MSG_COMPILED_OUTPUTFILE
if ! yarn pbjs -w commonjs -t static-module \
-p proto \
"proto/common/minimal_common.proto" \
"proto/types/minimal_msg_deposit.proto" \
"proto/types/minimal_msg_send.proto" \
"$COSMOS_COIN_PROTO" \
-o "$MSG_COMPILED_OUTPUTFILE" 2>pbjs_errors.txt; then
echo "Error: Failed to generate JavaScript bindings"
cat pbjs_errors.txt
exit 1
fi

# Fix import to be ESM-compatible (no omitted file extension)
sed -i -E 's|"(protobufjs/minimal)"|"\1.js"|' "$MSG_COMPILED_OUTPUTFILE"

Comment on lines 96 to 98
Copy link
Contributor

@coderabbitai coderabbitai bot Sep 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

BSD/GNU sed portability: use -i.bak to support macOS.

-sed -i -E 's|"(protobufjs/minimal)"|"\1.js"|' "$MSG_COMPILED_OUTPUTFILE"
+sed -i.bak -E 's|"(protobufjs/minimal)"|"\1.js"|' "$MSG_COMPILED_OUTPUTFILE" && rm -f "$MSG_COMPILED_OUTPUTFILE.bak"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# Fix import to be ESM-compatible (no omitted file extension)
sed -i -E 's|"(protobufjs/minimal)"|"\1.js"|' "$MSG_COMPILED_OUTPUTFILE"
# Fix import to be ESM-compatible (no omitted file extension)
sed -i.bak -E 's|"(protobufjs/minimal)"|"\1.js"|' "$MSG_COMPILED_OUTPUTFILE" && \
rm -f "$MSG_COMPILED_OUTPUTFILE.bak"
🤖 Prompt for AI Agents 
In packages/xchain-mayachain/genMsgs.sh around lines 96 to 98, the sed -i usage
is not portable to macOS; update the sed invocation to use the backup-style flag
(e.g., replace -i with -i.bak) so it works with BSD sed, and after running sed
remove the generated .bak backup file (remove the specific backup for
MSG_COMPILED_OUTPUTFILE) to avoid leaving stray files.

# Generate TypeScript definitions
tput setaf 2
echo "Generating $MSG_COMPILED_TYPES_OUTPUTFILE"
tput sgr0
yarn run pbts $MSG_COMPILED_OUTPUTFILE -o $MSG_COMPILED_TYPES_OUTPUTFILE
if ! yarn pbts "$MSG_COMPILED_OUTPUTFILE" -o "$MSG_COMPILED_TYPES_OUTPUTFILE" 2>pbts_errors.txt; then
echo "Error: Failed to generate TypeScript definitions"
cat pbts_errors.txt
exit 1
fi

# Verify generated files
if [ ! -f "$MSG_COMPILED_OUTPUTFILE" ] || [ ! -s "$MSG_COMPILED_OUTPUTFILE" ]; then
echo "Error: Generated JavaScript file is missing or empty"
exit 1
fi

if [ ! -f "$MSG_COMPILED_TYPES_OUTPUTFILE" ] || [ ! -s "$MSG_COMPILED_TYPES_OUTPUTFILE" ]; then
echo "Error: Generated TypeScript definitions file is missing or empty"
exit 1
fi

# Clean up error files if they're empty
[ ! -s pbjs_errors.txt ] && rm -f pbjs_errors.txt
[ ! -s pbts_errors.txt ] && rm -f pbts_errors.txt

tput setaf 2
echo "Removing $TMP_DIR/mayanode"
echo "✓ Successfully generated protobuf bindings"
tput sgr0
rm -rf $TMP_DIR
6 changes: 6 additions & 0 deletions packages/xchain-mayachain/src/types/proto/MsgCompiled.d.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ export namespace common {

/** Asset trade */
trade?: (boolean|null);

/** Asset secured */
secured?: (boolean|null);
}

/** Represents an Asset. */
Expand All @@ -45,6 +48,9 @@ export namespace common {
/** Asset trade. */
public trade: boolean;

/** Asset secured. */
public secured: boolean;

/**
* Creates a new Asset instance using the specified properties.
* @param [properties] Properties to set
Expand Down
22 changes: 22 additions & 0 deletions packages/xchain-mayachain/src/types/proto/MsgCompiled.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ $root.common = (function() {
* @property {string|null} [ticker] Asset ticker
* @property {boolean|null} [synth] Asset synth
* @property {boolean|null} [trade] Asset trade
* @property {boolean|null} [secured] Asset secured
*/

/**
Expand Down Expand Up @@ -86,6 +87,14 @@ $root.common = (function() {
*/
Asset.prototype.trade = false;

/**
* Asset secured.
* @member {boolean} secured
* @memberof common.Asset
* @instance
*/
Asset.prototype.secured = false;

/**
* Creates a new Asset instance using the specified properties.
* @function create
Expand Down Expand Up @@ -120,6 +129,8 @@ $root.common = (function() {
writer.uint32(/* id 4, wireType 0 =*/32).bool(message.synth);
if (message.trade != null && Object.hasOwnProperty.call(message, "trade"))
writer.uint32(/* id 5, wireType 0 =*/40).bool(message.trade);
if (message.secured != null && Object.hasOwnProperty.call(message, "secured"))
writer.uint32(/* id 6, wireType 0 =*/48).bool(message.secured);
return writer;
};

Expand Down Expand Up @@ -169,6 +180,9 @@ $root.common = (function() {
case 5:
message.trade = reader.bool();
break;
case 6:
message.secured = reader.bool();
break;
default:
reader.skipType(tag & 7);
break;
Expand Down Expand Up @@ -219,6 +233,9 @@ $root.common = (function() {
if (message.trade != null && message.hasOwnProperty("trade"))
if (typeof message.trade !== "boolean")
return "trade: boolean expected";
if (message.secured != null && message.hasOwnProperty("secured"))
if (typeof message.secured !== "boolean")
return "secured: boolean expected";
return null;
};

Expand All @@ -244,6 +261,8 @@ $root.common = (function() {
message.synth = Boolean(object.synth);
if (object.trade != null)
message.trade = Boolean(object.trade);
if (object.secured != null)
message.secured = Boolean(object.secured);
return message;
};

Expand All @@ -266,6 +285,7 @@ $root.common = (function() {
object.ticker = "";
object.synth = false;
object.trade = false;
object.secured = false;
}
if (message.chain != null && message.hasOwnProperty("chain"))
object.chain = message.chain;
Expand All @@ -277,6 +297,8 @@ $root.common = (function() {
object.synth = message.synth;
if (message.trade != null && message.hasOwnProperty("trade"))
object.trade = message.trade;
if (message.secured != null && message.hasOwnProperty("secured"))
object.secured = message.secured;
return object;
};

Expand Down
99 changes: 70 additions & 29 deletions packages/xchain-thorchain/genMsgs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,45 +1,68 @@
#!/bin/bash

# This script updates THORChain Protobuf bindings for MsgDeposit and MsgSend
set -e # Exit on any error

MSG_COMPILED_OUTPUTFILE=src/types/proto/MsgCompiled.js
MSG_COMPILED_TYPES_OUTPUTFILE=src/types/proto/MsgCompiled.d.ts

TMP_DIR=$(mktemp -d)
# Using local minimal proto files - no need to clone thornode repository

tput setaf 2
echo "Checking out https://gitlab.com/thorchain/thornode to $TMP_DIR"
tput sgr0
(cd "$TMP_DIR" && git clone --branch develop https://gitlab.com/thorchain/thornode)
# Download cosmos/base/v1beta1/coin.proto from cosmossdk
COSMOS_COIN_PROTO="proto/cosmos/base/v1beta1/coin.proto"
if [ ! -f "$COSMOS_COIN_PROTO" ]; then
tput setaf 2
echo "Downloading cosmos/base/v1beta1/coin.proto from cosmossdk"
tput sgr0
mkdir -p "proto/cosmos/base/v1beta1"
if ! curl -f -o "$COSMOS_COIN_PROTO" \
"https://raw.githubusercontent.com/cosmos/cosmos-sdk/main/proto/cosmos/base/v1beta1/coin.proto"; then
echo "Error: Failed to download cosmos coin.proto"
exit 1
fi
echo "✓ Downloaded cosmos coin.proto"
else
echo "✓ cosmos coin.proto already exists"
fi

# Verify proto files exist
# Verify our minimal proto files exist
tput setaf 2
echo "Checking proto files"
echo "Checking minimal proto files"
tput sgr0
ls "$TMP_DIR/thornode/proto/thorchain/v1/common/common.proto" || echo "common.proto missing"
ls "$TMP_DIR/thornode/proto/thorchain/v1/types/msg_deposit.proto" || echo "msg_deposit.proto missing"
ls "$TMP_DIR/thornode/proto/thorchain/v1/types/msg_send.proto" || echo "msg_send.proto missing"
MISSING_FILES=0
for proto_file in \
"proto/common/minimal_common.proto" \
"proto/types/minimal_msg_deposit.proto" \
"proto/types/minimal_msg_send.proto" \
"$COSMOS_COIN_PROTO"; do
if [ ! -f "$proto_file" ]; then
echo "Error: $(basename "$proto_file") missing"
MISSING_FILES=1
else
echo "✓ $(basename "$proto_file") found"
fi
done

# Download cosmos/base/v1beta1/coin.proto from cosmossdk
tput setaf 2
echo "Downloading cosmos/base/v1beta1/coin.proto from cosmossdk"
tput sgr0
mkdir -p "$TMP_DIR/thornode/third_party/proto/cosmos/base/v1beta1"
curl -o "$TMP_DIR/thornode/third_party/proto/cosmos/base/v1beta1/coin.proto" \
"https://raw.githubusercontent.com/cosmos/cosmos-sdk/main/proto/cosmos/base/v1beta1/coin.proto"
if [ $MISSING_FILES -eq 1 ]; then
echo "Error: Required proto files are missing"
exit 1
fi

# Generate Protobuf JS bindings with include path
# Generate Protobuf JS bindings using minimal proto files to prevent over-inclusion
tput setaf 2
echo "Generating $MSG_COMPILED_OUTPUTFILE"
tput sgr0
yarn pbjs -w commonjs -t static-module \
-p "$TMP_DIR/thornode/proto" \
-p "$TMP_DIR/thornode/third_party/proto" \
"$TMP_DIR/thornode/proto/thorchain/v1/common/common.proto" \
"$TMP_DIR/thornode/proto/thorchain/v1/types/msg_deposit.proto" \
"$TMP_DIR/thornode/proto/thorchain/v1/types/msg_send.proto" \
"$TMP_DIR/thornode/third_party/proto/cosmos/base/v1beta1/coin.proto" \
-o "$MSG_COMPILED_OUTPUTFILE" 2>pbjs_errors.txt
if ! yarn pbjs -w commonjs -t static-module \
-p proto \
"proto/common/minimal_common.proto" \
"proto/types/minimal_msg_deposit.proto" \
"proto/types/minimal_msg_send.proto" \
"$COSMOS_COIN_PROTO" \
-o "$MSG_COMPILED_OUTPUTFILE" 2>pbjs_errors.txt; then
echo "Error: Failed to generate JavaScript bindings"
cat pbjs_errors.txt
exit 1
fi
Copy link
Contributor

@coderabbitai coderabbitai bot Sep 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Ensure output directory exists before pbjs; fail clearly if tools are missing

Avoids failures on clean checkouts; surfaces missing binaries early.

+mkdir -p "$(dirname "$MSG_COMPILED_OUTPUTFILE")"
+command -v yarn >/dev/null || { echo "Error: yarn not found"; exit 1; }
 if ! yarn pbjs -w commonjs -t static-module \
   -p proto \
   "proto/common/minimal_common.proto" \
   "proto/types/minimal_msg_deposit.proto" \
   "proto/types/minimal_msg_send.proto" \
   "$COSMOS_COIN_PROTO" \
   -o "$MSG_COMPILED_OUTPUTFILE" 2>pbjs_errors.txt; then
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if ! yarn pbjs -w commonjs -t static-module \
-p proto \
"proto/common/minimal_common.proto" \
"proto/types/minimal_msg_deposit.proto" \
"proto/types/minimal_msg_send.proto" \
"$COSMOS_COIN_PROTO" \
-o "$MSG_COMPILED_OUTPUTFILE" 2>pbjs_errors.txt; then
echo "Error: Failed to generate JavaScript bindings"
cat pbjs_errors.txt
exit 1
fi
# Ensure the output directory exists
mkdir -p "$(dirname "$MSG_COMPILED_OUTPUTFILE")"
# Fail early if yarn isn't installed
command -v yarn >/dev/null || { echo "Error: yarn not found"; exit 1; }
if ! yarn pbjs -w commonjs -t static-module \
-p proto \
"proto/common/minimal_common.proto" \
"proto/types/minimal_msg_deposit.proto" \
"proto/types/minimal_msg_send.proto" \
"$COSMOS_COIN_PROTO" \
-o "$MSG_COMPILED_OUTPUTFILE" 2>pbjs_errors.txt; then
echo "Error: Failed to generate JavaScript bindings"
cat pbjs_errors.txt
exit 1
fi
🤖 Prompt for AI Agents 
In packages/xchain-thorchain/genMsgs.sh around lines 55 to 65, the script runs
pbjs without ensuring the output directory exists or that required tools are
installed; update the script to first check for required binaries (yarn and
pbjs) and exit with a clear error if missing, then create (mkdir -p) the parent
directory of $MSG_COMPILED_OUTPUTFILE before invoking pbjs so the command won't
fail on clean checkouts; ensure errors from missing tools or directory creation
are printed and cause an exit with non-zero status.


# Fix import to be ESM-compatible (no omitted file extension)
sed -i -E 's|"(protobufjs/minimal)"|"\1.js"|' "$MSG_COMPILED_OUTPUTFILE"
Expand All @@ -48,9 +71,27 @@ sed -i -E 's|"(protobufjs/minimal)"|"\1.js"|' "$MSG_COMPILED_OUTPUTFILE"
tput setaf 2
echo "Generating $MSG_COMPILED_TYPES_OUTPUTFILE"
tput sgr0
yarn pbts --name types "$MSG_COMPILED_OUTPUTFILE" -o "$MSG_COMPILED_TYPES_OUTPUTFILE" 2>pbts_errors.txt
if ! yarn pbts --name types "$MSG_COMPILED_OUTPUTFILE" -o "$MSG_COMPILED_TYPES_OUTPUTFILE" 2>pbts_errors.txt; then
echo "Error: Failed to generate TypeScript definitions"
cat pbts_errors.txt
exit 1
fi

# Verify generated files
if [ ! -f "$MSG_COMPILED_OUTPUTFILE" ] || [ ! -s "$MSG_COMPILED_OUTPUTFILE" ]; then
echo "Error: Generated JavaScript file is missing or empty"
exit 1
fi

if [ ! -f "$MSG_COMPILED_TYPES_OUTPUTFILE" ] || [ ! -s "$MSG_COMPILED_TYPES_OUTPUTFILE" ]; then
echo "Error: Generated TypeScript definitions file is missing or empty"
exit 1
fi

# Clean up error files if they're empty
[ ! -s pbjs_errors.txt ] && rm -f pbjs_errors.txt
[ ! -s pbts_errors.txt ] && rm -f pbts_errors.txt

tput setaf 2
echo "Removing $TMP_DIR/thornode"
echo "✓ Successfully generated protobuf bindings"
tput sgr0
rm -rf "$TMP_DIR"