[Experimental] Can't use the Kubernetes Provider with RBAC/Managed Identity

[rdeveen]

When on a AKS cluster the Local Accounts are disabled and we can't use SSH Keys the Bicep Kubernetes Provider can't be used. Would like to use the provider with RBAC enabled and Managed Identity.

What I tried:

Create AKS Cluster without local accounts

resource cluster 'Microsoft.ContainerService/managedClusters@2022-11-01' = {
  name: clusterName
  properties: {
    disableLocalAccounts: true
    // linuxProfile: null
  }

Use listClusterAdminCredential() to generate kubeConfig.

module clusterRoleBinding './modules/cluster-role-binding.bicep' = {
  name: 'clusterRoleBinding'
  params: {
    kubeConfig: cluster.listClusterAdminCredential().kubeconfigs[0].value
  }
}

Use kubeConfig

@secure()
param kubeConfig string

import 'kubernetes@1.0.0' with {
  namespace: 'default'
  kubeConfig: kubeConfig
} as k8s

When deploying this, the following error occurs:
Getting static credential is not allowed because this cluster is set to disable local accounts.

Would like to see the provider works without local account and SSH Keys.

[rdeveen]

As a second option i'm generating the kubeconfig with az cli and passing it to the bicep template.

az aks get-credentials --name clusterName --resource-group rg

az deployment group create --resource-group rg --subscription guid --template-file main.bicep --parameters kubeConfig=$(cat ~/.kube/config | base64 -w0)

Gives the following error:

{
  "code": "InvalidTemplateDeployment",
  "message": "The template deployment 'clusterRoleBinding' is not valid according to the validation procedure. The tracking id is '<guid>'. See inner errors for details.",
  "details": [
    {
      "code": "InvalidImportConfig",
      "target": "/imports/kubernetes/config",
      "message": "external exec failed due to: An error occurred trying to start process 'kubelogin' with working directory 'S:\\SvcFab\\_App\\ExtensibilityHostType_App8\\Extensibility.Host.ApiServicePkg.Code.1.0.4169.0_master'. The system cannot find the file specified."
    }
  ]
}

[shenglol]

Thanks for reporting this. We'll need to see if we can work around the limitation with OAuth OBO flow, which can potentially also enable access to private clusters, but we need to talk to the AKS team to get their approvals.

[jeremy-morren]

'Not planned'? What kind of joke is this????

[shenglol]

@rdeveen Reopening this since we’ve started the design work, so it’s not “Not Planned”.