enh: stt file load perfermance

[facetosea]

Description

Issue(s)

• Close/close/Fix/fix/Resolve/resolve: Issue Link

Checklist

Please check the items in the checklist if applicable.

• Is the user manual updated?
• Are the test cases passed and automated?
• Is there no significant decrease in test coverage?

[gemini-code-assist]

Summary of Changes

Hello @facetosea, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on enhancing the performance of STT file loading by optimizing array initialization. It introduces a new utility function, taosArrayAddZeroData, which provides a more efficient mechanism for adding multiple zero-initialized elements to dynamic arrays. This function is then integrated into the loadSttStatisticsBlockData process, replacing a less efficient loop-based approach and thereby reducing overhead during data loading.

Highlights

• New Utility Function: Introduced taosArrayAddZeroData in include/util/tarray.h and implemented in source/util/src/tarray.c. This function efficiently adds a specified number of zero-initialized elements to an SArray by ensuring capacity and updating the array size, without explicitly writing zeros.
• Performance Optimization: Refactored loadSttStatisticsBlockData in source/dnode/vnode/src/tsdb/tsdbMergeTree.c to utilize the new taosArrayAddZeroData function. This replaces manual loops that previously pushed individual zero-initialized SValue objects, leading to improved performance during STT file loading.
• Code Simplification: The change simplifies the logic for initializing pFirstKey and pLastKey arrays within loadSttStatisticsBlockData, making the code cleaner and more concise.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new function taosArrayAddZeroData to optimize the loading of STT files by adding multiple zero-initialized elements to an array more efficiently than repeated pushes. While the performance improvement is a good goal, I've found a few critical issues.

The new function taosArrayAddZeroData has a bug where it doesn't actually zero-initialize the new array elements, which could lead to undefined behavior. Additionally, the error checking for this new function in tsdbMergeTree.c is incorrect, using a stale pointer for validation. I've also pointed out an issue with the documentation for the new function.

Please address these correctness and maintainability issues. Also, note the typo in the pull request title ("perfermance" should be "performance").

[gemini-code-assist]

The Doxygen comment for the new function taosArrayAddZeroData is incorrect. It seems to be copied from another function, as it references a @param pData which is not a parameter of this function. The comment should accurately describe the function's purpose, parameters, and return value.

/**
 * @brief Add nEles zero-initialized elements to the array.
 * @param pArray The array to add elements to.
 * @param nEles The number of elements to add.
 * @return 0 on success, or an error code otherwise.
 */

[Copilot]

Pull request overview

This PR aims to improve STT (Sorted Time-series Table) file load performance by optimizing the addition of zero-initialized elements to arrays. However, the implementation contains critical bugs that will prevent it from working correctly.

Key Changes:

• Added new function taosArrayAddZeroData() to add zero-initialized elements to arrays without a loop
• Replaced manual loops that pushed zero-initialized SValue structs with calls to the new function
• Added function declaration in the public header file

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File	Description
include/util/tarray.h	Adds declaration for new taosArrayAddZeroData() function with incomplete documentation
source/util/src/tarray.c	Implements taosArrayAddZeroData() but fails to zero-initialize memory
source/dnode/vnode/src/tsdb/tsdbMergeTree.c	Replaces loops with calls to taosArrayAddZeroData() but uses incorrect error checking macros

Critical Issues Identified:

1. The new taosArrayAddZeroData() function does not actually zero the memory - it only reserves space
2. Error handling uses TSDB_CHECK_NULL(px, ...) but px is no longer defined after the refactor
3. The function should use TSDB_CHECK_CODE macro instead since it returns an error code, not a pointer

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.