Replication sub task is stuck in the canceling state

[PotentialIngenuity]

Version
core - 3.90.0

Describe the bug

The replication task group has 1 task the is stuck in the canceling state. This stuck task is blocking all other tasks from processing.

http --verify false GET https://localhost/pulp/api/v3/task-groups/0199a5b0-3e09-742b-943c-903794dcbb07/ | jq 'del(.tasks)'
{
  "pulp_href": "/pulp/api/v3/task-groups/0199a5b0-3e09-742b-943c-903794dcbb07/",
  "prn": "prn:core.taskgroup:0199a5b0-3e09-742b-943c-903794dcbb07",
  "description": "Replication of repo05",
  "all_tasks_dispatched": false,
  "waiting": 0,
  "skipped": 0,
  "running": 0,
  "completed": 0,
  "canceled": 0,
  "failed": 0,
  "canceling": 1,
  "group_progress_reports": []
}

The specific task.

pulp task list --state canceling
[
  {
    "pulp_href": "/pulp/api/v3/tasks/0199a5b0-3e1c-7a31-b5b0-344924548223/",
    "prn": "prn:core.task:0199a5b0-3e1c-7a31-b5b0-344924548223",
    "pulp_created": "2025-10-02T16:10:15.009534Z",
    "pulp_last_updated": "2025-10-02T16:10:14.941567Z",
    "state": "canceling",
    "name": "pulpcore.app.tasks.replica.replicate_distributions",
    "logging_cid": "468e93cfbd1744589c3dbb54698f54ce",
    "created_by": "/pulp/api/v3/users/2/",
    "unblocked_at": null,
    "started_at": null,
    "finished_at": null,
    "error": null,
    "worker": null,
    "parent_task": null,
    "child_tasks": [],
    "task_group": "/pulp/api/v3/task-groups/0199a5b0-3e09-742b-943c-903794dcbb07/",
    "progress_reports": [],
    "created_resources": [],
    "reserved_resources_record": [
      "pdrn:018ccc7c-afbe-7fcf-b3ec-ac5099b3a3ea:servers",
      "shared:prn:core.domain:018ccc7c-afbe-7fcf-b3ec-ac5099b3a3ea"
    ],
    "result": null
  }
]

To Reproduce
Start a replicate and cancel all the tasks while it is processing.

Expected behavior
The task should cancel and I can start other processes.

Additional context
n/a

[PotentialIngenuity]

I found this thread and tried to run the suggested fix. I ran it inside one of the new workers. The canceling task did not clear.

https://discourse.pulpproject.org/t/stuck-sync-task-after-worker-restart-pulpcore-3-90-0/2131

docker exec -it ccd020e98b16 /bin/bash
bash-5.1$ pulpcore-manager shell -c "from pulpcore.app.models import AppStatus; print(AppStatus.objects.missing().delete())"
(0, {})
bash-5.1$

[mdellweg]

Do you use any sort of db-connection pool? That is not supported.
If your worker never receives the signal to cancel the task, you will need to stop it to free up the resources eventually.

[PotentialIngenuity]

No i do not use a connection pool. I am using the one service per container yaml you guys have as an example. Its just a single database.

[mdellweg]

Do you see any of these log messages?
"Attempt to set not incomplete task {} to canceling from '{}'."
"Cleaning up task %s in domain: %s and marking as %s. Reason: %s"
or even a stack trace in their vicinity?

[PotentialIngenuity]

I see this morning the worker finally cleaned up the task. There are now 0 tasks in the canceling state.

Oct 15 08:25:19 172.18.0.1 docker-pulp_worker-2[3087507]: pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 0199a5b0-3e1c-7a31-b5b0-344924548223 in domain: default and marking as canceled.

[pedro-psb]

@PotentialIngenuity If it happened only once it's hard to tell, but are you still experiencing these kind of tasks being stuck on cancelling?
if you consider this fixed, did you update your installation, or can you correlate this to any action you've taken?