spKillSessionsForShardMappingLocal: Fix deadlock when the session is already gone

[PetrPospisil]

Fixed deadlock when the kill always returns 6106 forever for the session that is already gone. In this modification it will try to kill the session only once then moving to next in the next round.
Not sure I modified script versioning correctly, but you can see the idea of the fix.
This also probably address some issues mentioned in #166.

[ghost]

All CLA requirements met.

[zimaa77]

I see that this project is alive again. Do you plan to merge this PR? Or somehow fix the problem?

[stuartpa]

I don't have any context into this PR, and I don't see any tests for it (or even how to manually test the PR). Can anyone describe how to test this PR? Is it covered by an existing test? (I am not keen on merging PRs with no tests)

[calloncampbell]

We're also getting this issue in our production environment. It doesn't seem to be an issue in lower environments where there is less activity.

Here is the exception and stacktrace

Unhandled exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.ShardManagementException: Store Error: Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.. The error occurred while attempting to perform the underlying storage operation during 'Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.StoreException: Error occurred while performing store operation. See the inner SqlException for details.
 ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
 ---> System.ComponentModel.Win32Exception (258): The wait operation timed out.
   at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, SqlCommand command, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at Microsoft.Data.SqlClient.SqlDataReader.get_MetaData()
   at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean isAsync, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String method)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteReader()
   at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.SqlStoreTransactionScope.<>c__DisplayClass11_0.<ExecuteOperation>b__0()
   at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.SqlUtils.WithSqlExceptionHandling[TResult](Func`1 operation)
ClientConnectionId:d0de47bb-4e10-43ea-906f-5c56dac1c4ae
Error Number:-2,State:0,Class:11
   --- End of inner exception stack trace ---
   at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.SqlUtils.WithSqlExceptionHandling[TResult](Func`1 operation)
   at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.SqlStoreTransactionScope.ExecuteOperation(String operationName, XElement operationData)
   at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.UpdateMappingOperation.<KillConnectionsOnSourceShard>b__33_0()
   at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.SqlUtils.WithSqlExceptionHandling(Action operation)
   at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.UpdateMappingOperation.KillConnectionsOnSourceShard()
   at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.UpdateMappingOperation.DoLocalSourceExecute(IStoreTransactionScope ts)
   at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.StoreOperation.DoLocalSource()
   at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.StoreOperation.<Do>b__32_0()
   at Microsoft.Azure.SqlDatabase.ElasticScale.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.StoreOperation.Do()' operation on shard 'UpdatePointMappingMarkOffline'. See the inner StoreException for details.
   at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.StoreOperation.Do()
   at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.BaseShardMapper.Update[TMapping,TUpdate,TStatus](TMapping currentMapping, TUpdate update, Func`4 constructMapping, Func`2 statusAsInt, Func`2 intAsStatus, Guid lockOwnerId, MappingOptions options)
   at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.ListShardMapper`1.Update(PointMapping`1 currentMapping, PointMappingUpdate update, Guid lockOwnerId, MappingOptions options)
   at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.ListShardMap`1.UpdateMapping(PointMapping`1 currentMapping, PointMappingUpdate update, MappingLockToken mappingLockToken)
   at Microsoft.Azure.SqlDatabase.ElasticScale.ShardManagement.ListShardMap`1.UpdateMapping(PointMapping`1 currentMapping, PointMappingUpdate update)
   at ElasticShardSqlUtil.Services.ShardManagement.DeleteShardAction(ShardOptions options) in /home/runner/work/dni/dni/src/ElasticShardSqlUtil/DNI.ElasticSqlTool/Services/ShardManagement.cs:line 359
   at ElasticShardSqlUtil.Program.DeleteShardCommand(ShardOptions options, IHost host) in /home/runner/work/dni/dni/src/ElasticShardSqlUtil/DNI.ElasticSqlTool/Program.cs:line 307
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
   --- End of inner exception stack trace ---
   at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
   at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at System.CommandLine.NamingConventionBinder.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Hosting.HostingExtensions.<>c__DisplayClass1_0.<<UseHost>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass12_0.<<UseHelp>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseVersionOption>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<<UseTypoCorrections>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__18_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseParseDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__5_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass8_0.<<UseExceptionHandler>b__0>d.MoveNext()
PS E:\program files\Telmetrics\ElasticShardSqlUtil>