fix:stmt2 memleak #34169
fix:stmt2 memleak #34169
Conversation
Pengrongkun
requested review from
dapan1121,
guanshengliang and
hzcheng
as code owners
Summary of ChangesHello @Pengrongkun, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request primarily focuses on resolving a memory leak within the Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Pull request overview
This PR fixes memory leaks in the stmt2 (prepared statement version 2) implementation by improving memory cleanup across multiple components.
Key Changes:
- Moved
qDestroyBoundColInfofrom parser library to qcom library and enhanced it to properly clean up tag information - Changed parsed values data structure from
SArraytoSSHashObj(hash map) for more efficient storage and cleanup - Added missing memory cleanup calls for
aRowParray andpCreateTbReqstructures
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| source/libs/qcom/src/queryUtil.c | Added destroySTagsInfo and qDestroyBoundColInfo functions to properly clean up bound column info including nested tag data |
| source/libs/parser/src/parInsertUtil.c | Removed qDestroyBoundColInfo (moved to qcom), made destroyColVal non-static for reuse, added taosArrayDestroy call for aRowP |
| source/libs/parser/src/parInsertSql.c | Changed pParsedValues from SArray to SSHashObj, updated all references to use qDestroyBoundColInfo and hash operations |
| source/libs/parser/src/parInsertSml.c | Updated to call qDestroyBoundColInfo instead of insDestroyBoundColInfo |
| source/libs/parser/inc/parInsertUtil.h | Removed insDestroyBoundColInfo declaration (no longer needed) |
| source/client/src/clientStmt2.c | Renamed boundCols to fixedValueCols, implemented ownership transfer pattern, added cleanup for pCreateTbReq, improved stmtDeepReset cleanup logic |
| source/client/src/clientStmt.c | Updated function signature for stmtUpdateInfo to use SSHashObj** instead of SArray* |
| source/client/inc/clientStmt.h | Updated field name from boundCols to fixedValueCols with clarified comment |
| source/client/test/stmt2Test.cpp | Added test case for issue TD-6581610626 to verify fix, removed commented-out code |
| include/libs/qcom/query.h | Added qDestroyBoundColInfo function declaration |
| include/libs/parser/parser.h | Removed qDestroyBoundColInfo declaration (moved to qcom), updated SStmtCallback signature |
| include/common/tdataformat.h | Added destroyColVal function declaration for external use |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request effectively addresses a memory leak in stmt2 by improving memory management practices across several files. Key changes include:
- Refactoring cleanup logic in
stmtDeepResetto use a centralizedstmtCleanSQLInfofunction, which improves code maintainability. - Correctly freeing
pCreateTbReqinstmtSetTbTags2to prevent leaks. - Fixing a memory leak in
checkAndMergeSVgroupDataCxtByTbnameby destroyingaRowP. - Implementing a more complete cleanup in
qDestroyBoundColInfoto freeparseredTags, which was a source of memory leaks. - Changing
pParsedValuesfromSArraytoSSHashObjand setting a proper free function, enhancing both performance and memory safety.
The changes are well-implemented and consistent. The addition of a new test case for the fixed issue is also a good practice.
| void qDestroyBoundColInfo(void* pInfo) { | ||
| if (NULL == pInfo) { | ||
| return; | ||
| } | ||
| SBoundColInfo* pBoundInfo = (SBoundColInfo*)pInfo; | ||
|
|
||
| taosMemoryFreeClear(pBoundInfo->pColIndex); | ||
| destroySTagsInfo(pBoundInfo->parseredTags); | ||
| } |
There was a problem hiding this comment.
This is a crucial fix for a memory leak. The previous implementation of qDestroyBoundColInfo (or insDestroyBoundColInfo) only freed pColIndex. By adding the call to destroySTagsInfo(pBoundInfo->parseredTags), you are now correctly cleaning up the parseredTags and its contents, which was a source of the memory leak. The new destroySTagsInfo function also appears to be comprehensive in its cleanup.
|
|
||
| parserDebug("merge same uid data: %" PRId64 ", vgId:%d", pTbCtx->pData->uid, pVgCxt->vgId); | ||
|
|
||
| taosArrayDestroy(pTbCtx->pData->aRowP); |
There was a problem hiding this comment.
Adding taosArrayDestroy(pTbCtx->pData->aRowP); here is a critical fix. Previously, aRowP was not being destroyed when merging data for the same table, leading to a memory leak. This change correctly frees the array.
| if (pStmt->sql.fixValueTags) { | ||
| pStmt->sql.fixValueTags = false; | ||
| tdDestroySVCreateTbReq(pStmt->sql.fixValueTbReq); | ||
| pStmt->sql.fixValueTbReq = NULL; | ||
| } |
There was a problem hiding this comment.
Good improvement. Resetting fixValueTags to false and nulling out fixValueTbReq after freeing it is a good defensive programming practice. It prevents potential double-frees or use-after-free issues if stmtCleanSQLInfo is called multiple times.
| static int32_t stmtDeepReset(STscStmt2* pStmt) { | ||
| // Save state that needs to be preserved | ||
| char* db = pStmt->db; | ||
| TAOS_STMT2_OPTION options = pStmt->options; | ||
| uint32_t reqid = pStmt->reqid; | ||
| bool stbInterlaceMode = pStmt->stbInterlaceMode; | ||
|
|
||
| pStmt->errCode = 0; | ||
|
|
||
| // Wait for async execution to complete | ||
| if (pStmt->options.asyncExecFn && !pStmt->execSemWaited) { | ||
| if (tsem_wait(&pStmt->asyncExecSem) != 0) { | ||
| STMT2_ELOG_E("bind param wait asyncExecSem failed"); | ||
| } | ||
| pStmt->execSemWaited = true; | ||
| } | ||
|
|
||
| if (pStmt->stbInterlaceMode) { | ||
| if (pStmt->bindThreadInUse) { | ||
| while (0 == atomic_load_8((int8_t*)&pStmt->sql.siInfo.tableColsReady)) { | ||
| taosUsleep(1); | ||
| } | ||
| (void)taosThreadMutexLock(&pStmt->queue.mutex); | ||
| pStmt->queue.stopQueue = true; | ||
| (void)taosThreadCondSignal(&(pStmt->queue.waitCond)); | ||
| (void)taosThreadMutexUnlock(&pStmt->queue.mutex); | ||
|
|
||
| (void)taosThreadJoin(pStmt->bindThread, NULL); | ||
| pStmt->bindThreadInUse = false; | ||
| pStmt->queue.head = NULL; | ||
| pStmt->queue.tail = NULL; | ||
| pStmt->queue.qRemainNum = 0; | ||
|
|
||
| (void)taosThreadCondDestroy(&pStmt->queue.waitCond); | ||
| (void)taosThreadMutexDestroy(&pStmt->queue.mutex); | ||
| // Stop bind thread if in use (similar to stmtClose2) | ||
| if (stbInterlaceMode && pStmt->bindThreadInUse) { | ||
| while (0 == atomic_load_8((int8_t*)&pStmt->sql.siInfo.tableColsReady)) { | ||
| taosUsleep(1); | ||
| } | ||
| } | ||
|
|
||
| pStmt->sql.autoCreateTbl = false; | ||
| taosMemoryFree(pStmt->sql.pBindInfo); | ||
| pStmt->sql.pBindInfo = NULL; | ||
|
|
||
| taosMemoryFree(pStmt->sql.queryRes.fields); | ||
| pStmt->sql.queryRes.fields = NULL; | ||
|
|
||
| taosMemoryFree(pStmt->sql.queryRes.userFields); | ||
| pStmt->sql.queryRes.userFields = NULL; | ||
|
|
||
| pStmt->sql.type = 0; | ||
| pStmt->sql.runTimes = 0; | ||
| taosMemoryFree(pStmt->sql.sqlStr); | ||
| pStmt->sql.sqlStr = NULL; | ||
|
|
||
| qDestroyQuery(pStmt->sql.pQuery); | ||
| pStmt->sql.pQuery = NULL; | ||
|
|
||
| taosArrayDestroy(pStmt->sql.nodeList); | ||
| pStmt->sql.nodeList = NULL; | ||
|
|
||
| taosHashCleanup(pStmt->sql.pVgHash); | ||
| pStmt->sql.pVgHash = NULL; | ||
|
|
||
| if (pStmt->sql.fixValueTags) { | ||
| tdDestroySVCreateTbReq(pStmt->sql.fixValueTbReq); | ||
| pStmt->sql.fixValueTbReq = NULL; | ||
| } | ||
| pStmt->sql.fixValueTags = false; | ||
|
|
||
| void* pIter = taosHashIterate(pStmt->sql.pTableCache, NULL); | ||
| while (pIter) { | ||
| SStmtTableCache* pCache = (SStmtTableCache*)pIter; | ||
| (void)taosThreadMutexLock(&pStmt->queue.mutex); | ||
| pStmt->queue.stopQueue = true; | ||
| (void)taosThreadCondSignal(&(pStmt->queue.waitCond)); | ||
| (void)taosThreadMutexUnlock(&pStmt->queue.mutex); | ||
|
|
||
| qDestroyStmtDataBlock(pCache->pDataCtx); | ||
| qDestroyBoundColInfo(pCache->boundTags); | ||
| taosMemoryFreeClear(pCache->boundTags); | ||
| (void)taosThreadJoin(pStmt->bindThread, NULL); | ||
| pStmt->bindThreadInUse = false; | ||
| pStmt->queue.head = NULL; | ||
| pStmt->queue.tail = NULL; | ||
| pStmt->queue.qRemainNum = 0; | ||
|
|
||
| pIter = taosHashIterate(pStmt->sql.pTableCache, pIter); | ||
| (void)taosThreadCondDestroy(&pStmt->queue.waitCond); | ||
| (void)taosThreadMutexDestroy(&pStmt->queue.mutex); | ||
| } | ||
| taosHashCleanup(pStmt->sql.pTableCache); | ||
|
|
||
| if (pStmt->sql.stbInterlaceMode) { | ||
| // Clean all SQL and execution info (stmtCleanSQLInfo already handles most cleanup) | ||
| pStmt->bInfo.boundColsCached = false; | ||
| if (stbInterlaceMode) { | ||
| pStmt->bInfo.tagsCached = false; | ||
| } | ||
| STMT_ERR_RET(stmtCleanExecInfo(pStmt, false, true)); | ||
|
|
||
| resetRequest(pStmt); | ||
|
|
||
| if (pStmt->sql.siInfo.pTableCols) { | ||
| taosArrayDestroyEx(pStmt->sql.siInfo.pTableCols, stmtFreeTbCols); | ||
| pStmt->sql.siInfo.pTableCols = NULL; | ||
| } | ||
|
|
||
| if (pStmt->sql.siInfo.tbBuf.pBufList) { | ||
| taosArrayDestroyEx(pStmt->sql.siInfo.tbBuf.pBufList, stmtFreeTbBuf); | ||
| pStmt->sql.siInfo.tbBuf.pBufList = NULL; | ||
| } | ||
|
|
||
| if (pStmt->sql.siInfo.pTableHash) { | ||
| tSimpleHashCleanup(pStmt->sql.siInfo.pTableHash); | ||
| pStmt->sql.siInfo.pTableHash = NULL; | ||
| } | ||
|
|
||
| if (pStmt->sql.siInfo.pTableRowDataHash) { | ||
| tSimpleHashCleanup(pStmt->sql.siInfo.pTableRowDataHash); | ||
| pStmt->sql.siInfo.pTableRowDataHash = NULL; | ||
| } | ||
|
|
||
| if (pStmt->sql.siInfo.pVgroupHash) { | ||
| taosHashCleanup(pStmt->sql.siInfo.pVgroupHash); | ||
| pStmt->sql.siInfo.pVgroupHash = NULL; | ||
| } | ||
|
|
||
| if (pStmt->sql.siInfo.pVgroupList) { | ||
| taosArrayDestroy(pStmt->sql.siInfo.pVgroupList); | ||
| pStmt->sql.siInfo.pVgroupList = NULL; | ||
| } | ||
|
|
||
| if (pStmt->sql.siInfo.pDataCtx) { | ||
| qDestroyStmtDataBlock(pStmt->sql.siInfo.pDataCtx); | ||
| pStmt->sql.siInfo.pDataCtx = NULL; | ||
| } | ||
| STMT_ERR_RET(stmtCleanSQLInfo(pStmt)); | ||
|
|
||
| if (pStmt->sql.siInfo.pTSchema) { | ||
| taosMemoryFree(pStmt->sql.siInfo.pTSchema); | ||
| pStmt->sql.siInfo.pTSchema = NULL; | ||
| } | ||
| // Reinitialize resources (similar to stmtInit2) | ||
| if (stbInterlaceMode) { | ||
| pStmt->sql.siInfo.transport = pStmt->taos->pAppInfo->pTransporter; | ||
| pStmt->sql.siInfo.acctId = pStmt->taos->acctId; | ||
| pStmt->sql.siInfo.dbname = pStmt->taos->db; | ||
| pStmt->sql.siInfo.mgmtEpSet = getEpSet_s(&pStmt->taos->pAppInfo->mgmtEp); | ||
|
|
||
| if (pStmt->sql.siInfo.pRequest) { | ||
| taos_free_result(pStmt->sql.siInfo.pRequest); | ||
| pStmt->sql.siInfo.pRequest = NULL; | ||
| } | ||
| if (NULL == pStmt->pCatalog) { | ||
| STMT_ERR_RET(catalogGetHandle(pStmt->taos->pAppInfo->clusterId, &pStmt->pCatalog)); | ||
| } | ||
| pStmt->sql.siInfo.pCatalog = pStmt->pCatalog; | ||
|
|
||
| if (pStmt->stbInterlaceMode) { | ||
| STMT_ERR_RET(stmtResetStbInterlaceCache(pStmt)); | ||
|
|
||
| int32_t code = stmtIniAsyncBind(pStmt); | ||
| if (TSDB_CODE_SUCCESS != code) { | ||
| STMT2_ELOG("fail to reinit async bind in stmtDeepReset:%s", tstrerror(code)); | ||
| return code; | ||
| } | ||
| } | ||
|
|
||
| // Restore preserved state | ||
| pStmt->db = db; | ||
| pStmt->options = options; | ||
| pStmt->reqid = reqid; | ||
| pStmt->stbInterlaceMode = stbInterlaceMode; | ||
|
|
||
| pStmt->sql.pTableCache = taosHashInit(100, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), false, HASH_NO_LOCK); | ||
| if (NULL == pStmt->sql.pTableCache) { | ||
| STMT2_ELOG("fail to allocate memory for pTableCache in stmtResetStmt:%s", tstrerror(terrno)); | ||
| STMT2_ELOG("fail to allocate memory for pTableCache in stmtDeepReset:%s", tstrerror(terrno)); | ||
| return terrno; | ||
| } | ||
|
|
||
| pStmt->bInfo.needParse = true; | ||
| pStmt->sql.status = STMT_INIT; | ||
| pStmt->sql.siInfo.tableColsReady = true; | ||
|
|
||
| return TSDB_CODE_SUCCESS; | ||
| } |
There was a problem hiding this comment.
The refactoring of stmtDeepReset to use stmtCleanSQLInfo is a good improvement. It centralizes cleanup logic and removes a significant amount of duplicated code, which enhances maintainability and reduces the risk of introducing new bugs during future modifications.
Description
Issue(s)
Checklist
Please check the items in the checklist if applicable.