Dedication of public IP ranges broken after upgrade to 4.17.0.1

[Hudratronium]

ISSUE TYPE

• Bug Report

COMPONENT NAME

API, UI, SSVM, CPVM

CLOUDSTACK VERSION

4.17.0.1

CONFIGURATION

1 Zone, Advanced networking, 3 domains

OS / ENVIRONMENT

All running Ubuntu 20.04.5
Database mysql 8

SUMMARY

Existing configuration (working running CS 4.16.1)

Setting system.vm.public.reservation.mode.strictness = true

After upgrade:
While deploying SSVM / CPVM, dedicated IP Ranges for use of system VMs aren't concidert for deployment. With system.vm.public.reservation.mode.strictness enabled, a deployment of SSVM / CPVM fails "due to Insufficent address capacity".
With system.vm.public.reservation.mode.strictness set to 'false' deployment is successfull while using IP ranges not marked as dedicated for system VMs.
Newly created IP ranges with dedication for systemVMs aren't stored or found in db table domain_vlan_map. Also prior used ip ranges dedicated for systemVMs aren't referenced in domain_vlan_map.

Currently tested with public IP addresses. Not tested with "strictness" for management IP addresses.

STEPS TO REPRODUCE

• Create IP range for public IPs with dedication for system VMs enabled
• Change setting system.vm.public.reservation.mode.strictness to true
• Destroy existing CPVM / SSVM

EXPECTED RESULTS

- Table `domain_vlan_map` shows entry for newly created IP range (stored as new vlan) assigned to domain
- System VMs are successfully deployed using IPs from the dedicated IP range

ACTUAL RESULTS

- Table `domain_vlan_map` shows no entry
- Deployment of systemVMs  fails due to insufficent address capacity
- Changeing setting `system.vm.public.reservation.mode.strictness` to "false"
   --> deployment successfull using IP range of domain not dedicated for systemVMs

[shwstppr]

@Hudratronium I'm not able to reproduce this on 4.17 branch. I added a new public range and set system.vm.public.ip.reservation.mode.strictness to true. New system vms came up fine in the new range

(local) 🐓 > list zones id=8395094c-305b-46cd-878a-b1dcc01c37d0 
{
  "count": 1,
  "zone": [
    {
      "allocationstate": "Enabled",
      "dhcpprovider": "VirtualRouter",
      "dns1": "10.147.28.6",
      "guestcidraddress": "10.1.1.0/24",
      "hasannotations": false,
      "id": "8395094c-305b-46cd-878a-b1dcc01c37d0",
      "internaldns1": "10.147.28.6",
      "localstorageenabled": false,
      "name": "Sandbox-simulator",
      "networktype": "Advanced",
      "securitygroupsenabled": false,
      "tags": [],
      "zonetoken": "bfd27e96-3ba0-3df3-b387-f8ff690c63c5"
    }
  ]
}
(local) 🐗 > list systemvms zoneid=8395094c-305b-46cd-878a-b1dcc01c37d0 
{
  "count": 2,
  "systemvm": [
    {
      "activeviewersessions": 0,
      "created": "2022-09-05T12:36:52+0530",
      "dns1": "10.147.28.6",
      "gateway": "192.168.2.1",
      "hasannotations": false,
      "hostid": "fc35ad4a-bdc8-496a-94e9-46d9d228526a",
      "hostname": "SimulatedAgent.9d862bb4-6179-4a90-ab8a-f91b23fac374",
      "hypervisor": "Simulator",
      "id": "52ff1944-20d5-4954-8cfd-dcd0ed96b46f",
      "isdynamicallyscalable": false,
      "linklocalip": "169.254.196.124",
      "linklocalmacaddress": "0e:00:a9:fe:c4:7c",
      "linklocalnetmask": "255.255.0.0",
      "name": "v-9-VM",
      "podid": "ccdd5096-1dcc-40c5-9699-801e22f3f15f",
      "podname": "POD0",
      "privateip": "172.16.15.65",
      "privatemacaddress": "1e:00:54:00:00:40",
      "privatenetmask": "255.255.255.0",
      "publicip": "192.168.2.2",
      "publicmacaddress": "1e:00:91:00:00:c8",
      "publicnetmask": "255.255.255.0",
      "state": "Running",
      "systemvmtype": "consoleproxy",
      "templateid": "669011fa-2cdd-11ed-b1e6-645d8651f45a",
      "templatename": "SystemVM Template (simulator)",
      "zoneid": "8395094c-305b-46cd-878a-b1dcc01c37d0",
      "zonename": "Sandbox-simulator"
    },
    {
      "agentstate": "Up",
      "created": "2022-09-05T12:36:53+0530",
      "dns1": "10.147.28.6",
      "gateway": "192.168.2.1",
      "hasannotations": false,
      "hostid": "bdefc6c3-4a43-4d6a-9570-0fc9a6ac6d14",
      "hostname": "SimulatedAgent.d1b801a8-36b8-47e7-bfd7-710402b63a99",
      "hypervisor": "Simulator",
      "id": "eb726c05-b467-48b7-b42e-8a6c0a9bca42",
      "isdynamicallyscalable": false,
      "linklocalip": "169.254.181.229",
      "linklocalmacaddress": "0e:00:a9:fe:b5:e5",
      "linklocalnetmask": "255.255.0.0",
      "name": "s-10-VM",
      "podid": "ccdd5096-1dcc-40c5-9699-801e22f3f15f",
      "podname": "POD0",
      "privateip": "172.16.15.20",
      "privatemacaddress": "1e:00:75:00:00:13",
      "privatenetmask": "255.255.255.0",
      "publicip": "192.168.2.3",
      "publicmacaddress": "1e:00:16:00:00:c9",
      "publicnetmask": "255.255.255.0",
      "state": "Running",
      "systemvmtype": "secondarystoragevm",
      "templateid": "669011fa-2cdd-11ed-b1e6-645d8651f45a",
      "templatename": "SystemVM Template (simulator)",
      "version": "4.17.1.0-SNAPSHOT",
      "zoneid": "8395094c-305b-46cd-878a-b1dcc01c37d0",
      "zonename": "Sandbox-simulator"
    }
  ]
}
(local) 🐰 > list vlanipranges zoneid=8395094c-305b-46cd-878a-b1dcc01c37d0 
{
  "count": 1,
  "vlaniprange": [
    {
      "account": "system",
      "cidr": "192.168.2.0/24",
      "domain": "ROOT",
      "domainid": "fe476c7c-2cdc-11ed-b1e6-645d8651f45a",
      "endip": "192.168.2.200",
      "forsystemvms": false,
      "forvirtualnetwork": true,
      "gateway": "192.168.2.1",
      "id": "10b19120-cfd8-441c-81c8-e156c09c38e4",
      "netmask": "255.255.255.0",
      "networkid": "234ef426-bb2f-445e-83d9-5363fc6bfb38",
      "physicalnetworkid": "a44e9a10-6682-42e1-81b3-ffabfdfc30f9",
      "startip": "192.168.2.2",
      "vlan": "vlan://50",
      "zoneid": "8395094c-305b-46cd-878a-b1dcc01c37d0"
    }
  ]
}
(local) 🐈 > list configurations name=system.vm.public.ip.reservation.mode.strictness
{
  "configuration": [
    {
      "category": "Advanced",
      "description": "If enabled, the use of System VMs public IP reservation is strict, preferred if not.",
      "isdynamic": true,
      "name": "system.vm.public.ip.reservation.mode.strictness",
      "value": "false"
    }
  ],
  "count": 1
}
(local) 🐬 > create vlaniprange gateway=192.168.3.1 netmask=255.255.255.0 startip=192.168.3.2 endip=192.168.3.200 zoneId=8395094c-305b-46cd-878a-b1dcc01c37d0 vlan=51 forsystemvms=true forvirtualnetwork=true
{
  "vlan": {
    "account": "system",
    "cidr": "192.168.3.0/24",
    "domain": "ROOT",
    "domainid": "fe476c7c-2cdc-11ed-b1e6-645d8651f45a",
    "endip": "192.168.3.200",
    "forsystemvms": true,
    "forvirtualnetwork": true,
    "gateway": "192.168.3.1",
    "id": "edc376e0-cf2d-4ab7-aaac-0b7b14e2a931",
    "netmask": "255.255.255.0",
    "networkid": "234ef426-bb2f-445e-83d9-5363fc6bfb38",
    "physicalnetworkid": "a44e9a10-6682-42e1-81b3-ffabfdfc30f9",
    "startip": "192.168.3.2",
    "vlan": "vlan://51",
    "zoneid": "8395094c-305b-46cd-878a-b1dcc01c37d0"
  }
}
(local) 🦌 > update configuration name=system.vm.public.ip.reservation.mode.strictness value=true
{
  "configuration": {
    "category": "Advanced",
    "description": "If enabled, the use of System VMs public IP reservation is strict, preferred if not.",
    "isdynamic": true,
    "name": "system.vm.public.ip.reservation.mode.strictness",
    "value": "true"
  }
}
(local) 🐃 > destroy systemvm id=52ff1944-20d5-4954-8cfd-dcd0ed96b46f 
{
  "systemvm": {
    "created": "2022-09-05T12:36:52+0530",
    "dns1": "10.147.28.6",
    "hasannotations": false,
    "hostid": "fc35ad4a-bdc8-496a-94e9-46d9d228526a",
    "hostname": "SimulatedAgent.9d862bb4-6179-4a90-ab8a-f91b23fac374",
    "hypervisor": "Simulator",
    "id": "52ff1944-20d5-4954-8cfd-dcd0ed96b46f",
    "isdynamicallyscalable": false,
    "name": "v-9-VM",
    "podid": "ccdd5096-1dcc-40c5-9699-801e22f3f15f",
    "podname": "POD0",
    "state": "Running",
    "systemvmtype": "consoleproxy",
    "templateid": "669011fa-2cdd-11ed-b1e6-645d8651f45a",
    "templatename": "SystemVM Template (simulator)",
    "zoneid": "8395094c-305b-46cd-878a-b1dcc01c37d0",
    "zonename": "Sandbox-simulator"
  }
}
(local) 🌵 > destroy systemvm id=eb726c05-b467-48b7-b42e-8a6c0a9bca42 
{
  "systemvm": {
    "created": "2022-09-05T12:36:53+0530",
    "dns1": "10.147.28.6",
    "hasannotations": false,
    "hostid": "bdefc6c3-4a43-4d6a-9570-0fc9a6ac6d14",
    "hostname": "SimulatedAgent.d1b801a8-36b8-47e7-bfd7-710402b63a99",
    "hypervisor": "Simulator",
    "id": "eb726c05-b467-48b7-b42e-8a6c0a9bca42",
    "isdynamicallyscalable": false,
    "name": "s-10-VM",
    "podid": "ccdd5096-1dcc-40c5-9699-801e22f3f15f",
    "podname": "POD0",
    "state": "Running",
    "systemvmtype": "secondarystoragevm",
    "templateid": "669011fa-2cdd-11ed-b1e6-645d8651f45a",
    "templatename": "SystemVM Template (simulator)",
    "zoneid": "8395094c-305b-46cd-878a-b1dcc01c37d0",
    "zonename": "Sandbox-simulator"
  }
}
(local) 🐂 > list systemvms zoneid=8395094c-305b-46cd-878a-b1dcc01c37d0 
{
  "count": 2,
  "systemvm": [
    {
      "activeviewersessions": 0,
      "created": "2022-09-05T12:39:23+0530",
      "dns1": "10.147.28.6",
      "gateway": "192.168.3.1",
      "hasannotations": false,
      "hostid": "f0bcf07c-ab72-4137-a413-449684ae309c",
      "hostname": "SimulatedAgent.f83bd68a-5642-49bc-aac2-dd8a4785aedc",
      "hypervisor": "Simulator",
      "id": "01a9646f-17af-430d-aab4-a07b4a9eec35",
      "isdynamicallyscalable": false,
      "linklocalip": "169.254.170.181",
      "linklocalmacaddress": "0e:00:a9:fe:aa:b5",
      "linklocalnetmask": "255.255.0.0",
      "name": "v-11-VM",
      "podid": "ccdd5096-1dcc-40c5-9699-801e22f3f15f",
      "podname": "POD0",
      "privateip": "172.16.15.145",
      "privatemacaddress": "1e:00:60:00:00:90",
      "privatenetmask": "255.255.255.0",
      "publicip": "192.168.3.200",
      "publicmacaddress": "1e:00:40:00:03:f7",
      "publicnetmask": "255.255.255.0",
      "state": "Running",
      "systemvmtype": "consoleproxy",
      "templateid": "669011fa-2cdd-11ed-b1e6-645d8651f45a",
      "templatename": "SystemVM Template (simulator)",
      "zoneid": "8395094c-305b-46cd-878a-b1dcc01c37d0",
      "zonename": "Sandbox-simulator"
    },
    {
      "agentstate": "Up",
      "created": "2022-09-05T12:39:53+0530",
      "dns1": "10.147.28.6",
      "gateway": "192.168.3.1",
      "hasannotations": false,
      "hostid": "bdefc6c3-4a43-4d6a-9570-0fc9a6ac6d14",
      "hostname": "SimulatedAgent.d1b801a8-36b8-47e7-bfd7-710402b63a99",
      "hypervisor": "Simulator",
      "id": "02c6d3dd-13ae-44d4-bf90-0d20e9647525",
      "isdynamicallyscalable": false,
      "linklocalip": "169.254.102.29",
      "linklocalmacaddress": "0e:00:a9:fe:66:1d",
      "linklocalnetmask": "255.255.0.0",
      "name": "s-12-VM",
      "podid": "ccdd5096-1dcc-40c5-9699-801e22f3f15f",
      "podname": "POD0",
      "privateip": "172.16.15.138",
      "privatemacaddress": "1e:00:1f:00:00:89",
      "privatenetmask": "255.255.255.0",
      "publicip": "192.168.3.199",
      "publicmacaddress": "1e:00:fd:00:03:f6",
      "publicnetmask": "255.255.255.0",
      "state": "Running",
      "systemvmtype": "secondarystoragevm",
      "templateid": "669011fa-2cdd-11ed-b1e6-645d8651f45a",
      "templatename": "SystemVM Template (simulator)",
      "version": "4.17.1.0-SNAPSHOT",
      "zoneid": "8395094c-305b-46cd-878a-b1dcc01c37d0",
      "zonename": "Sandbox-simulator"
    }
  ]
}

[Hudratronium]

@shwstppr
Here the whole log with comments on database actions from my system. Out of interest: new created CS4.17 setup or upgraded from 4.16 / 4.15 / .... ?

Cann you maybe take a look into your db and lookig if you can find in domain_vlan_map a entry for referencing the vlan which is enabled for use with systemVMs and the domain you are actually using ?

###################################################################### Nic allocation

2022-08-31 18:44:21,203 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Allocating nics for VM instance {id: "86", name: "v-86-VM", uuid: "6f77acb4-c65e-49e8-ae03-1175b3974412", type="ConsoleProxy"}
2022-08-31 18:44:21,203 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-79, consoleproxy-1,
2022-08-31 18:44:21,204 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: start requested by: -Transaction.execute:39-NetworkOrchestrator.allocate:778-VirtualMachineManagerImpl$1.doInTransactionWithoutResult:478-TransactionCallbackWithExceptionNoReturn.doInTransaction:25-TransactionCallbackWithExceptionNoReturn.doInTransaction:21-Transaction.execute:40-VirtualMachineManagerImpl.allocate:469-VirtualMachineManagerImpl.allocate:528-ConsoleProxyManagerImpl.createProxyInstance:700-ConsoleProxyManagerImpl.startNew:572-ConsoleProxyManagerImpl.allocCapacity:809-ConsoleProxyManagerImpl.expandPool:1537
2022-08-31 18:44:21,204 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: has already been started.
2022-08-31 18:44:21,204 TRACE [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) allocating networks for v-86-VM(template c20799e4-a990-449a-aae6-4b67374c7aff); 3 networks
2022-08-31 18:44:21,204 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,206 TRACE [c.c.u.d.T.Connection] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Creating a DB connection with  txn:  for 0: dbconn623913035. Stack: -TransactionLegacy.prepareStatement:466-TransactionLegacy.prepareAutoCloseStatement:459-GenericDaoBase.searchIncludingRemoved:396-GenericDaoBase.searchIncludingRemoved:359-GenericDaoBase.search:348-GenericDaoBase.search:1319-TemplateDeployAsIsDetailsDaoImpl.listDetailsByTemplateIdMatchingPrefix:64-TemplateDeployAsIsDetailsDaoImpl.listNetworkRequirementsByTemplateId:69-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:566
2022-08-31 18:44:21,206 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT template_deploy_as_is_details.id, template_deploy_as_is_details.template_id, template_deploy_as_is_details.name, template_deploy_as_is_details.value FROM template_deploy_as_is_details WHERE template_deploy_as_is_details.template_id = ?  AND template_deploy_as_is_details.name LIKE ?

#### template_deploy_as_is_details = empty --> no results returned

2022-08-31 18:44:21,206 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT template_deploy_as_is_details.id, template_deploy_as_is_details.template_id, template_deploy_as_is_details.name, template_deploy_as_is_details.value FROM template_deploy_as_is_details WHERE template_deploy_as_is_details.template_id = 213  AND template_deploy_as_is_details.name LIKE x'6E6574776F726B2D25'
2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Connection] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: Not closing DB connection because we're still in a transaction.
2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT networks.id, networks.mode, networks.broadcast_domain_type, networks.traffic_type, networks.name, networks.display_text, networks.broadcast_uri, networks.gateway, networks.cidr, networks.network_cidr, networks.network_offering_id, networks.vpc_id, networks.physical_network_id, networks.data_center_id, networks.related, networks.guru_name, networks.state, networks.redundant, networks.dns1, networks.domain_id, networks.account_id, networks.set_fields, networks.guru_data, networks.dns2, networks.network_domain, networks.removed, networks.created, networks.reservation_id, networks.uuid, networks.guest_type, networks.acl_type, networks.restart_required, networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr, networks.display_network, networks.network_acl_id, networks.streched_l2, networks.external_id FROM networks WHERE networks.id = ?  AND networks.removed IS NULL
2022-08-31 18:44:21,209 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Allocating nic for vm VM instance {id: "86", name: "v-86-VM", uuid: "6f77acb4-c65e-49e8-ae03-1175b3974412", type="ConsoleProxy"} in network Network {"id": 200, "name": "null", "uuid": "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1} with requested profile NicProfile {"broadcastUri":null,"iPv4Address":null,"id":0,"reservationId":null,"vmId":0}
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT data_center.id, data_center.name, data_center.description, data_center.dns1, data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2, data_center.router_mac_address, data_center.guest_network_cidr, data_center.domain_id, data_center.domain, data_center.networktype, data_center.dns_provider, data_center.dhcp_provider, data_center.gateway_provider, data_center.vpn_provider, data_center.userdata_provider, data_center.lb_provider, data_center.firewall_provider, data_center.mac_address, data_center.zone_token, data_center.removed, data_center.allocation_state, data_center.uuid, data_center.is_security_group_enabled, data_center.is_local_storage_enabled, data_center.sort_key FROM data_center WHERE data_center.id = ?  AND data_center.removed IS NULL

### Allocating public nic

2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT networks.id, networks.mode, networks.broadcast_domain_type, networks.traffic_type, networks.name, networks.display_text, networks.broadcast_uri, networks.gateway, networks.cidr, networks.network_cidr, networks.network_offering_id, networks.vpc_id, networks.physical_network_id, networks.data_center_id, networks.related, networks.guru_name, networks.state, networks.redundant, networks.dns1, networks.domain_id, networks.account_id, networks.set_fields, networks.guru_data, networks.dns2, networks.network_domain, networks.removed, networks.created, networks.reservation_id, networks.uuid, networks.guest_type, networks.acl_type, networks.restart_required, networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr, networks.display_network, networks.network_acl_id, networks.streched_l2, networks.external_id FROM networks WHERE networks.id = 200  AND networks.removed IS NULL
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT account.id, account.account_name, account.type, account.role_id, account.domain_id, account.state, account.created, account.removed, account.cleanup_needed, account.network_domain, account.uuid, account.default_zone_id, account.default FROM account WHERE account.id = ?  AND account.removed IS NULL

### Getting Zone Information

2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT data_center.id, data_center.name, data_center.description, data_center.dns1, data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2, data_center.router_mac_address, data_center.guest_network_cidr, data_center.domain_id, data_center.domain, data_center.networktype, data_center.dns_provider, data_center.dhcp_provider, data_center.gateway_provider, data_center.vpn_provider, data_center.userdata_provider, data_center.lb_provider, data_center.firewall_provider, data_center.mac_address, data_center.zone_token, data_center.removed, data_center.allocation_state, data_center.uuid, data_center.is_security_group_enabled, data_center.is_local_storage_enabled, data_center.sort_key FROM data_center WHERE data_center.id = 1  AND data_center.removed IS NULL
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: start requested by: -Transaction.execute:39-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823-NetworkOrchestrator$3.doInTransactionWithoutResult:797-TransactionCallbackWithExceptionNoReturn.doInTransaction:25-TransactionCallbackWithExceptionNoReturn.doInTransaction:21
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: has already been started.
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT data_center.id, data_center.name, data_center.description, data_center.dns1, data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2, data_center.router_mac_address, data_center.guest_network_cidr, data_center.domain_id, data_center.domain, data_center.networktype, data_center.dns_provider, data_center.dhcp_provider, data_center.gateway_provider, data_center.vpn_provider, data_center.userdata_provider, data_center.lb_provider, data_center.firewall_provider, data_center.mac_address, data_center.zone_token, data_center.removed, data_center.allocation_state, data_center.uuid, data_center.is_security_group_enabled, data_center.is_local_storage_enabled, data_center.sort_key FROM data_center WHERE data_center.id = ?  AND data_center.removed IS NULL

### Account Information "system"

2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT account.id, account.account_name, account.type, account.role_id, account.domain_id, account.state, account.created, account.removed, account.cleanup_needed, account.network_domain, account.uuid, account.default_zone_id, account.default FROM account WHERE account.id = 1  AND account.removed IS NULL
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT networks.id, networks.mode, networks.broadcast_domain_type, networks.traffic_type, networks.name, networks.display_text, networks.broadcast_uri, networks.gateway, networks.cidr, networks.network_cidr, networks.network_offering_id, networks.vpc_id, networks.physical_network_id, networks.data_center_id, networks.related, networks.guru_name, networks.state, networks.redundant, networks.dns1, networks.domain_id, networks.account_id, networks.set_fields, networks.guru_data, networks.dns2, networks.network_domain, networks.removed, networks.created, networks.reservation_id, networks.uuid, networks.guest_type, networks.acl_type, networks.restart_required, networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr, networks.display_network, networks.network_acl_id, networks.streched_l2, networks.external_id FROM networks WHERE networks.id = ?  AND networks.removed IS NULL

### Getting Zone Information

2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT data_center.id, data_center.name, data_center.description, data_center.dns1, data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2, data_center.router_mac_address, data_center.guest_network_cidr, data_center.domain_id, data_center.domain, data_center.networktype, data_center.dns_provider, data_center.dhcp_provider, data_center.gateway_provider, data_center.vpn_provider, data_center.userdata_provider, data_center.lb_provider, data_center.firewall_provider, data_center.mac_address, data_center.zone_token, data_center.removed, data_center.allocation_state, data_center.uuid, data_center.is_security_group_enabled, data_center.is_local_storage_enabled, data_center.sort_key FROM data_center WHERE data_center.id = 1  AND data_center.removed IS NULL
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT account_vlan_map.id, account_vlan_map.account_id, account_vlan_map.vlan_db_id FROM account_vlan_map WHERE account_vlan_map.account_id = ?

### Getting network information

2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT networks.id, networks.mode, networks.broadcast_domain_type, networks.traffic_type, networks.name, networks.display_text, networks.broadcast_uri, networks.gateway, networks.cidr, networks.network_cidr, networks.network_offering_id, networks.vpc_id, networks.physical_network_id, networks.data_center_id, networks.related, networks.guru_name, networks.state, networks.redundant, networks.dns1, networks.domain_id, networks.account_id, networks.set_fields, networks.guru_data, networks.dns2, networks.network_domain, networks.removed, networks.created, networks.reservation_id, networks.uuid, networks.guest_type, networks.acl_type, networks.restart_required, networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr, networks.display_network, networks.network_acl_id, networks.streched_l2, networks.external_id FROM networks WHERE networks.id = null  AND networks.removed IS NULL
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT domain_vlan_map.id, domain_vlan_map.domain_id, domain_vlan_map.vlan_db_id FROM domain_vlan_map WHERE domain_vlan_map.domain_id = ?

####   Account_vlan_map - no entry for account "System" nor for Account "Admin"

2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT account_vlan_map.id, account_vlan_map.account_id, account_vlan_map.vlan_db_id FROM account_vlan_map WHERE account_vlan_map.account_id = 1
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT vlan.id, vlan.vlan_id, vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr, vlan.data_center_id, vlan.ip4_range, vlan.ip6_range, vlan.network_id, vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed, vlan.created FROM vlan  LEFT OUTER JOIN domain_vlan_map ON vlan.id=domain_vlan_map.vlan_db_id  LEFT OUTER JOIN account_vlan_map ON vlan.id=account_vlan_map.vlan_db_id WHERE vlan.data_center_id = ?  AND vlan.removed IS NULL  AND  (domain_vlan_map.domain_id IS NULL ) AND (account_vlan_map.account_id IS NULL )

### Checking for availeable VLANS for domain --> VLAN for system VM not shown / or in the result

2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT domain_vlan_map.id, domain_vlan_map.domain_id, domain_vlan_map.vlan_db_id FROM domain_vlan_map WHERE domain_vlan_map.domain_id = 1

### Checking for availeable VLANS wich are NOT assigned to ANY Account NOR to ANY Domain --> Result is the intended vlan with expected IP Range

2022-08-31 18:44:21,214 TRACE [c.c.u.d.GenericDaoBase] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) join search statement is com.mysql.cj.jdbc.ClientPreparedStatement: SELECT vlan.id, vlan.vlan_id, vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr, vlan.data_center_id, vlan.ip4_range, vlan.ip6_range, vlan.network_id, vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed, vlan.created FROM vlan  LEFT OUTER JOIN domain_vlan_map ON vlan.id=domain_vlan_map.vlan_db_id  LEFT OUTER JOIN account_vlan_map ON vlan.id=account_vlan_map.vlan_db_id WHERE vlan.data_center_id = 1  AND vlan.removed IS NULL  AND  (domain_vlan_map.domain_id IS NULL ) AND (account_vlan_map.account_id IS NULL )
2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated FROM configuration WHERE configuration.name = ?

### Getting further information on the vlan --> correct choice of expected VLAN + IP-Range

2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT vlan.id, vlan.vlan_id, vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr, vlan.data_center_id, vlan.ip4_range, vlan.ip6_range, vlan.network_id, vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed, vlan.created FROM vlan  LEFT OUTER JOIN domain_vlan_map ON vlan.id=domain_vlan_map.vlan_db_id  LEFT OUTER JOIN account_vlan_map ON vlan.id=account_vlan_map.vlan_db_id WHERE vlan.data_center_id = 1  AND vlan.removed IS NULL  AND  (domain_vlan_map.domain_id IS NULL ) AND (account_vlan_map.account_id IS NULL )
2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT user_ip_address.id, user_ip_address.account_id, user_ip_address.domain_id, user_ip_address.public_ip_address, user_ip_address.data_center_id, user_ip_address.source_nat, user_ip_address.allocated, user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat, user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address, user_ip_address.source_network_id, user_ip_address.network_id, user_ip_address.uuid, user_ip_address.physical_network_id, user_ip_address.is_system, user_ip_address.vpc_id, user_ip_address.dnat_vmip, user_ip_address.is_portable, user_ip_address.display, user_ip_address.rule_state, user_ip_address.forsystemvms, user_ip_address.removed, user_ip_address.created FROM user_ip_address  INNER JOIN vlan ON user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id = ?  AND user_ip_address.allocated IS NULL  AND user_ip_address.vlan_db_id IN (?,?,?,?)  AND user_ip_address.forsystemvms = ?  AND user_ip_address.removed IS NULL  AND  (vlan.vlan_type = ? ) ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC  LIMIT 0, 1 FOR UPDATE

### Check of setting system.vm.public.ip.reservation.mode.strictness --> value "true"

2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated FROM configuration WHERE configuration.name = x'73797374656D2E766D2E7075626C69632E69702E7265736572766174696F6E2E6D6F64652E7374726963746E657373'
2022-08-31 18:44:21,215 TRACE [c.c.u.d.GenericDaoBase] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) join search statement is com.mysql.cj.jdbc.ClientPreparedStatement: SELECT user_ip_address.id, user_ip_address.account_id, user_ip_address.domain_id, user_ip_address.public_ip_address, user_ip_address.data_center_id, user_ip_address.source_nat, user_ip_address.allocated, user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat, user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address, user_ip_address.source_network_id, user_ip_address.network_id, user_ip_address.uuid, user_ip_address.physical_network_id, user_ip_address.is_system, user_ip_address.vpc_id, user_ip_address.dnat_vmip, user_ip_address.is_portable, user_ip_address.display, user_ip_address.rule_state, user_ip_address.forsystemvms, user_ip_address.removed, user_ip_address.created FROM user_ip_address  INNER JOIN vlan ON user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id = 1  AND user_ip_address.allocated IS NULL  AND user_ip_address.vlan_db_id IN (3,10,13,14)  AND user_ip_address.forsystemvms = 1  AND user_ip_address.removed IS NULL  AND  (vlan.vlan_type = 'VirtualNetwork' ) ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC  LIMIT 0, 1 FOR UPDATE
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT account_details.id, account_details.account_id, account_details.name, account_details.value FROM account_details WHERE account_details.account_id = ?  AND account_details.name = ?  ORDER BY RAND() LIMIT 1

### Trying to fetch IP-Address: No value / ip address returned -  Domain "Root" shall provide ip addresses; thats where the vlanIDs are comming from --> user_ip_address.vlan_db_id IN (3,10,13,14)

2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT user_ip_address.id, user_ip_address.account_id, user_ip_address.domain_id, user_ip_address.public_ip_address, user_ip_address.data_center_id, user_ip_address.source_nat, user_ip_address.allocated, user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat, user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address, user_ip_address.source_network_id, user_ip_address.network_id, user_ip_address.uuid, user_ip_address.physical_network_id, user_ip_address.is_system, user_ip_address.vpc_id, user_ip_address.dnat_vmip, user_ip_address.is_portable, user_ip_address.display, user_ip_address.rule_state, user_ip_address.forsystemvms, user_ip_address.removed, user_ip_address.created FROM user_ip_address  INNER JOIN vlan ON user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id = 1  AND user_ip_address.allocated IS NULL  AND user_ip_address.vlan_db_id IN (3,10,13,14)  AND user_ip_address.forsystemvms = 1  AND user_ip_address.removed IS NULL  AND  (vlan.vlan_type = 'VirtualNetwork' ) ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC  LIMIT 0, 1 FOR UPDATE
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated FROM configuration WHERE configuration.name = ?  ORDER BY RAND() LIMIT 1

### Trying to get information from the table account_detail for the account "system". Table is empty so no result back

2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT account_details.id, account_details.account_id, account_details.name, account_details.value FROM account_details WHERE account_details.account_id = 1  AND account_details.name = x'7573652E73797374656D2E7075626C69632E697073'  ORDER BY RAND() LIMIT 1
2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated FROM configuration WHERE configuration.name = ?

### Check for Configuration enable.account.settings.for.domain --> current value = false

2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated FROM configuration WHERE configuration.name = x'656E61626C652E6163636F756E742E73657474696E67732E666F722E646F6D61696E'  ORDER BY RAND() LIMIT 1
2022-08-31 18:44:21,217 WARN  [c.c.n.IpAddressManagerImpl] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Unable to get ip address in  zone id=1, vlanId id=[3, 10, 13, 14]
2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction: consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB connection
2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT data_center.id, data_center.name, data_center.description, data_center.dns1, data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2, data_center.router_mac_address, data_center.guest_network_cidr, data_center.domain_id, data_center.domain, data_center.networktype, data_center.dns_provider, data_center.dhcp_provider, data_center.gateway_provider, data_center.vpn_provider, data_center.userdata_provider, data_center.lb_provider, data_center.firewall_provider, data_center.mac_address, data_center.zone_token, data_center.removed, data_center.allocation_state, data_center.uuid, data_center.is_security_group_enabled, data_center.is_local_storage_enabled, data_center.sort_key FROM data_center WHERE data_center.id = ?  AND data_center.removed IS NULL

### Check for setting use.system.public.ips --> current value = false

2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated FROM configuration WHERE configuration.name = x'7573652E73797374656D2E7075626C69632E697073'

### getting information for the current data_center / zone

2022-08-31 18:44:21,219 TRACE [c.c.u.d.T.Statement] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT data_center.id, data_center.name, data_center.description, data_center.dns1, data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2, data_center.router_mac_address, data_center.guest_network_cidr, data_center.domain_id, data_center.domain, data_center.networktype, data_center.dns_provider, data_center.dhcp_provider, data_center.gateway_provider, data_center.vpn_provider, data_center.userdata_provider, data_center.lb_provider, data_center.firewall_provider, data_center.mac_address, data_center.zone_token, data_center.removed, data_center.allocation_state, data_center.uuid, data_center.is_security_group_enabled, data_center.is_local_storage_enabled, data_center.sort_key FROM data_center WHERE data_center.id = 1  AND data_center.removed IS NULL
2022-08-31 18:44:21,219 DEBUG [c.c.u.d.T.Transaction] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Rolling back the transaction: Time = 17 Name =  consoleproxy-1; called by -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-Transaction.execute:38-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823
2022-08-31 18:44:21,219 TRACE [c.c.u.d.T.Lock] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) SQL com.mysql.cj.jdbc.ClientPreparedStatement: SELECT user_ip_address.id, user_ip_address.account_id, user_ip_address.domain_id, user_ip_address.public_ip_address, user_ip_address.data_center_id, user_ip_address.source_nat, user_ip_address.allocated, user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat, user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address, user_ip_address.source_network_id, user_ip_address.network_id, user_ip_address.uuid, user_ip_address.physical_network_id, user_ip_address.is_system, user_ip_address.vpc_id, user_ip_address.dnat_vmip, user_ip_address.is_portable, user_ip_address.display, user_ip_address.rule_state, user_ip_address.forsystemvms, user_ip_address.removed, user_ip_address.created FROM user_ip_address  INNER JOIN vlan ON user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id = 1  AND user_ip_address.allocated IS NULL  AND user_ip_address.vlan_db_id IN (3,10,13,14)  AND user_ip_address.forsystemvms = 1  AND user_ip_address.removed IS NULL  AND  (vlan.vlan_type = 'VirtualNetwork' ) ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC  LIMIT 0, 1 FOR UPDATE  took 4
2022-08-31 18:44:21,219 TRACE [c.c.u.d.T.Connection] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing DB connection: dbconn623913035
2022-08-31 18:44:21,220 WARN  [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Unable to allocate proxy [Console VM instance {id: "86", name: "v-86-VM", uuid: "6f77acb4-c65e-49e8-ae03-1175b3974412", type="ConsoleProxy"}] on zone [1] due to [Insufficient address capacity].
com.cloud.exception.InsufficientAddressCapacityException: Insufficient address capacityScope=interface com.cloud.dc.DataCenter; id=1

[shwstppr]

@Hudratronium mine was a new env. Maybe will check with upgrade

[Hudratronium]

@shwstppr

at least it occurs to me that in my current setup the missing entry in the cloud.domain_vlan_map seems to be the crucial thing missing.
I can create a new dedicated range for the domain but it isn't represented in the cloud.domain_vlan_map table. Which seems to be crucial in the overall "search" for a fitting IP-Range. I don't know why this isn't updated / the existing configuration doesn't work out anymore.

Maybe some more infos regarding our global conf:
use.system.public.ips = false
system.vm.public.reservation.mode.strictness = true
From my reading of the trace-log above the process ist:

1. Check if the account has dedicated vlans --> result is "none"
2. Check if the domain has dedicated vlans --> results are all vlan for a domain, but missing the one for use with systemVMs as it isn't present in cloud.domain_vlan_map
3. Check if there are any vlans NOT assigned to a account or domain --> result it the correct VLAN (guess that would be the check if there is a "systempool") --> not choosen as use.system.public.ips = false
4. Looking up setting system.vm.public.reservation.mode.strictness --> if true only domain related VLANs shall be checked / user_ip_address.forsystemvms setting is irrelevant
5. lookup IPs - depenending on assigned vlans of the domain - and check for IPs which are not assigned and user_ip_address.forsystemvms = 1 --> fails as the "correct" vlan isn't in the results of the former querys as it is missing in db table

[shwstppr]

@Hudratronium can you please share logs/API call for adding the new ip range. Also, the cmk output of list vlanipranges
I don't think there should be an entry in cloud.domain_vlan_map when the range is reserved for system vms. Entry in this table is added only when the following code conditions are met for the range

                } else if (domain != null && !forSystemVms) {
                    // This VLAN is domain-wide, so create a DomainVlanMapVO entry
                    final DomainVlanMapVO domainVlanMapVO = new DomainVlanMapVO(domain.getId(), vlan.getId());
                    _domainVlanMapDao.persist(domainVlanMapVO);
                } else if (podId != null) {
                    // This VLAN is pod-wide, so create a PodVlanMapVO entry
                    final PodVlanMapVO podVlanMapVO = new PodVlanMapVO(podId, vlan.getId());
                    podVlanMapDao.persist(podVlanMapVO);
                }

So the range is dedicated to a domain only when it is not marked for systemvms. Error could be due to the fact that the range is not marked for systemvms but assigned to a domain which prevents listing its IPs from user.cloud_ip_address when CPVM is created.

[Hudratronium]

@shwstppr

I don't think there should be an entry in cloud.domain_vlan_map when the range is reserved for system vms.

Takeing a look into your shared code i would think the same.
But the point is, that this is not the thing we want to achieve imho with the configuration - like displayed in the GUI:
We want a 'IP Range (vlan)' which is assigned to the domain 'Root' and dedicated for systemVMs. And simply asking where is the information stored, that this vlan shall be used for systemVM only? I didn't found a table regarding 'vlans' which contained this information. I only stumbled apon the entries for user_cloud_ip_address.forsystemvm = 1.
But this information, looking into the sql queries and there results, will not be fetched during the process. The finding and choosing of vlans to check for IPs is done before the lookup if dedicated IPs are availeable in the gathered set of vlan's.

Creating a vlan (IP range) without the entry to either domain_vlan_map or account_vlan_map might be something like a "systempool" for the Zone which usage is controlled via the setting use.system.public.ips.
Although the nameing is a bit odd, as 'system' is a "Account" itself. And there is no reference for a vlan with usage for systemVMs in the db in account_vlan_map - Even if the GUI would make you think there is one:

This "zone" vlan is checked via the allocator sql-statements:
join search statement is com.mysql.cj.jdbc.ClientPreparedStatement: SELECT vlan.id, vlan.vlan_id, vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr, vlan.data_center_id, vlan.ip4_range, vlan.ip6_range, vlan.network_id, vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed, vlan.created FROM vlan LEFT OUTER JOIN domain_vlan_map ON vlan.id=domain_vlan_map.vlan_db_id LEFT OUTER JOIN account_vlan_map ON vlan.id=account_vlan_map.vlan_db_id WHERE vlan.**data_center_id = 1** AND vlan.removed IS NULL AND (domain_vlan_map.domain_id IS NULL ) AND (account_vlan_map.account_id IS NULL )

And is availeable as a result - however it is not used for allocating ip addresses as, i guess, the current settings avoid it.

Back to your questions regarding logs / outputs:

list vlanipranges

+---------+---------------+--------+--------------------------------------+-------------+--------------+-------------------+------------+--------------------------------------+-----------------+--------------------------------------+--------------------------------------+-------------+-------------+--------------------------------------+
| ACCOUNT |     CIDR      | DOMAIN |               DOMAINID               |    ENDIP    | FORSYSTEMVMS | FORVIRTUALNETWORK |  GATEWAY   |                  ID                  |     NETMASK     |              NETWORKID               |          PHYSICALNETWORKID           |   STARTIP   |    VLAN     |                ZONEID                |
+---------+---------------+--------+--------------------------------------+-------------+--------------+-------------------+------------+--------------------------------------+-----------------+--------------------------------------+--------------------------------------+-------------+-------------+--------------------------------------+
| system  | 10.129.0.0/25 | ROOT   | 0676360f-1d87-11ec-83ce-525400c9c662 | 10.129.0.19 | true         | true              | 10.129.0.1 | adc15744-9b35-4bea-81f0-1fdbd5b544f3 | 255.255.255.128 | 1b0bf965-a391-478d-833d-3e37e4a15eb8 | dec372e0-2fa3-436b-9265-8caba82f0620 | 10.129.0.10 | vlan://1000 | 48e2e928-3300-43b5-8e3a-d9b358022502 |
|         | 10.129.0.0/25 | ROOT   | 0676360f-1d87-11ec-83ce-525400c9c662 | 10.129.0.29 | false        | true              | 10.129.0.1 | c52e9946-a8db-4d66-819f-ee11dfefbb37 | 255.255.255.128 | 1b0bf965-a391-478d-833d-3e37e4a15eb8 | dec372e0-2fa3-436b-9265-8caba82f0620 | 10.129.0.20 | vlan://1000 | 48e2e928-3300-43b5-8e3a-d9b358022502 |
| xxxxxx  | 10.129.0.0/25 | xxxxxx | 8cbb6645-0f0f-4c25-a24d-101d1426bb8f | 10.129.0.49 | false        | true              | 10.129.0.1 | 1031a85e-35d9-4c81-aee2-6d43ed1185ae | 255.255.255.128 | 1b0bf965-a391-478d-833d-3e37e4a15eb8 | dec372e0-2fa3-436b-9265-8caba82f0620 | 10.129.0.30 | vlan://1000 | 48e2e928-3300-43b5-8e3a-d9b358022502 |
+---------+---------------+--------+--------------------------------------+-------------+--------------+-------------------+------------+--------------------------------------+-----------------+--------------------------------------+--------------------------------------+-------------+-------------+--------------------------------------+

This output leads to the question: Where is the reference stored, that the Account 'system' has a mapped vlan?
Currently i would think the only way to get this information is to look into user_cloud_ip_address.forsystemvm = 1. Or there is a table i haven't found yet where this information can be stored...Where ever it is, performing the SQL Queries from the log on my db, these 'value' isn't represented at all.

logs/API call for adding the new ip range

2022-09-05 13:15:12,138 DEBUG [c.c.a.ApiServlet] (qtp961160488-2321:ctx-a8d73839) (logid:e4b1e254) ===START===  192.126.0.168 -- GET  gateway=10.129.0.1&netmask=255.255.255.128&startip=10.129.0.50&endip=10.129.0.59&zoneId=48e2e928-3300-43b5-8e3a-d9b358022502&vlan=1000&forsystemvms=true&forvirtualnetwork=true&command=createVlanIpRange&response=json
2022-09-05 13:15:12,150 DEBUG [c.c.a.ApiServer] (qtp961160488-2321:ctx-a8d73839 ctx-683cd7e0) (logid:e4b1e254) CIDRs from which account 'Acct[60be55c5-1d8a-11ec-83ce-525400c9c662-admin] -- Account {"id": 2, "name": "admin", "uuid": "60be55c5-1d8a-11ec-83ce-525400c9c662"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2022-09-05 13:15:12,164 DEBUG [c.c.c.ConfigurationManagerImpl] (qtp961160488-2321:ctx-a8d73839 ctx-683cd7e0) (logid:e4b1e254) Acquiring lock for committing vlan
2022-09-05 13:15:12,167 DEBUG [c.c.c.ConfigurationManagerImpl] (qtp961160488-2321:ctx-a8d73839 ctx-683cd7e0) (logid:e4b1e254) Access granted to Acct[60be55c5-1d8a-11ec-83ce-525400c9c662-admin] -- Account {"id": 2, "name": "admin", "uuid": "60be55c5-1d8a-11ec-83ce-525400c9c662"} to zone:1 by AffinityGroupAccessChecker
2022-09-05 13:15:12,173 DEBUG [c.c.c.ConfigurationManagerImpl] (qtp961160488-2321:ctx-a8d73839 ctx-683cd7e0) (logid:e4b1e254) Saving vlan range Vlan[vlan://1000|10.129.0.1|255.255.255.128|null|null|10.129.0.50-10.129.0.59|null|200]
2022-09-05 13:15:12,208 DEBUG [c.c.a.ApiServlet] (qtp961160488-2321:ctx-a8d73839 ctx-683cd7e0) (logid:e4b1e254) ===END===  192.126.0.168 -- GET  gateway=10.129.0.1&netmask=255.255.255.128&startip=10.129.0.50&endip=10.129.0.59&zoneId=48e2e928-3300-43b5-8e3a-d9b358022502&vlan=1000&forsystemvms=true&forvirtualnetwork=true&command=createVlanIpRange&response=json

While working on this i stumbled upon the strictness setting for the Management IPs... and i guess one might find the same thing there.

[shwstppr]

I don't think this is critical/blocker. I tried UI. When For system VMs is selected in the UI it correctly marks the IPs from the range for system VMs even when domain is selected (also with API, #6698 (comment)). Traditionally, I don't see any linkage between domain and system VMs. It would be better to discuss and make changes for it if needed in the next release. Moving to the next milestone

[Hudratronium]

@shwstppr
Just to be shure:
And with system.vm.public.reservation.mode.strictness = "true" and use.system.public.ips = "false" the deployment worked for you?
Have you more then one Domain configured in your setting?

As i said i am just curious as this configuration has been working until i upgraded twards 4.17.0.1. Might be that something went wrong during the upgrade of the Management-Server.
However from the logs of the SQL - statements everything is working as one would expected from a logical point.

When For system VMs is selected in the UI it correctly marks the IPs from the range for system VMs even when domain is selected (also with API, #6698 (comment)).

It does indeed - but that doesn't help when during the allocationprocess the group of vlans gathered for a domain isn't containing the vlan-id, which is used to reference to the IP range (IP - addresses) dedicated to systemVMs.
From the GUI point of view literally nothing has changed after the upgrade and everything is shown correctly.

Maybe you could share a log of the deploying sequence with "trace"-logging so that i can look up to it if they differ ?

Traditionally, I don't see any linkage between domain and system VMs.

At least from the overall architecture in my understanding this is going to be relevant if one is working with mulitple zones as well as "Dedicated Zones" for different Domains / Accounts - depending of course on what one wants to achieve.

@nvazquez just mentioning you here as you asked me to raise the issue and i don't know if i informed you about this on the mailinglist

[nvazquez]

Thanks @Hudratronium - I will book some time to try replicating it for an upgraded environment

[Hudratronium]

@nvazquez
Did you found some time to look into this? As we are planning our schedule for the next CS Upgrades, i just wanted to be shure to provide valid information if needed.

[Hudratronium]

@nvazquez
Just asking if you might see any chance to take a look into this topic?
As we are planning the next upgrade, i won't be able to give informations for this CS Version after the end of the month.

[yadvr]

@shwstppr @nvazquez this is critical for 4.17.2.0, it is reproducible?