Volume snapshot schedule shows error after successful retry #821
Description
Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT
What happened: Failed volume snapshot was retried, but the status on the volumesnapshotschedule still indicates an error.
# status item on the volume snapshot schedule
- creationTimestamp: "2021-04-09T00:01:09Z"
finishTimestamp: "2021-04-09T00:02:48Z"
name: REDACT-APPLICATION-weekly-cloud-daily-2021-04-09-000109
status: Error
# the completed volume snapshot
NAME PVC STATUS CREATED COMPLETED TYPE
REDACT-APPLICATION-weekly-cloud-daily-2021-04-09-000109 REDACT-APPLICATION Ready 09 Apr 21 01:01 BST 09 Apr 21 01:02 BST cloud
What you expected to happen:
The volume snapshot schedule to be correctly updated with completed.
How to reproduce it (as minimally and precisely as possible): Unsure how to reproduce the snapshot failure right now
This go error seems to be the cause of the initial failure;
E0409 00:02:12.212477 1 goroutinemap.go:150] Operation for "createREDACTED/REDACTED-APPLICATION-weekly-cloud-daily-2021-04-09-000109-992c7132-7f48-4955-83c3-418bc8f6c3eeREDACTED-APPLICATION" failed. No retries permitted until 2021-04-09 00:02:12.712449701 +0000 UTC m=+2214460.338826427 (durationBeforeRetry 500ms). Error: "Failed to take snapshot of the volume pvc-f74e5996-46f0-479e-a97b-7c3842a1087e: \"rpc error: code = Internal desc = Failed to create backup: Other cloudsnaps Requests pending, try little later\""
Anything else we need to know?: We're using portworx
Environment:
- Kubernetes version (use
kubectl version):
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.12-eks-7684af", GitCommit:"7684af4ac41370dd109ac13817023cb8063e3d45", GitTreeState:"clean", BuildDate:"2020-10-20T22:57:40Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
- Cloud provider or hardware configuration: AWS
- OS (e.g. from /etc/os-release): Amazon Linux 2
- Kernel (e.g.
uname -a):
Linux ip-ADDRESS.eu-central-1.compute.internal 5.4.58-32.125.amzn2.x86_64 #1 SMP Thu Sep 3 19:06:32 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
- Install tools:
- Others: