ipv4_gateway address attached to only a single network_interface when multiple attached to the same r/network

[wsf11]

Terraform Version

Terraform v0.13.2

vSphere Provider Version

1.24.0

Affected Resource(s)

• vsphere_virtual_machine

Terraform Configuration Files

locals {
  resource_pool_name = "${var.host_ip}/Resources"
}

data "vsphere_datacenter" "dc" {
  name = var.datacenter
}

data "vsphere_host" "host" {
  name          = var.host_ip
  datacenter_id = data.vsphere_datacenter.dc.id
}

data "vsphere_datastore" "datastore" {
  name          = var.datastore
  datacenter_id = data.vsphere_datacenter.dc.id
}

data "vsphere_resource_pool" "pool" {
  name          = local.resource_pool_name
  datacenter_id = data.vsphere_datacenter.dc.id
}

# PortGroup one - have ensured these work
data "vsphere_network" "portgroup_one" {
  name          = var.portgroup_one_name
  datacenter_id = data.vsphere_datacenter.dc.id
}

# PortGroup two - have ensured these work
data "vsphere_network" "portgroup_two" {
  name          = var.portgroup_two_name
  datacenter_id = data.vsphere_datacenter.dc.id
}

data "vsphere_virtual_machine" "template" {
  name          = var.template_name
  datacenter_id = data.vsphere_datacenter.dc.id
}

resource "vsphere_virtual_machine" "vm" {
  name             = var.name
  resource_pool_id = data.vsphere_resource_pool.pool.id
  datastore_id     = data.vsphere_datastore.datastore.id
  host_system_id   = data.vsphere_host.host.id
  firmware         = "efi"

  num_cpus               = 8
  cpu_hot_add_enabled    = true
  memory                 = 32768
  memory_hot_add_enabled = true

  guest_id  = data.vsphere_virtual_machine.template.guest_id
  scsi_type = data.vsphere_virtual_machine.template.scsi_type

  # NIC one
  network_interface {
    network_id   = data.vsphere_network.portgroup_one.id
    adapter_type = "vmxnet3"
  }

  # NIC two
  network_interface {
    network_id   = data.vsphere_network.portgroup_two.id
    adapter_type = "vmxnet3"
  }

  disk {
    label            = "disk-${var.name}"
    size             = data.vsphere_virtual_machine.template.disks.0.size
    thin_provisioned = true
  }

  clone {
    template_uuid = data.vsphere_virtual_machine.template.id

    customize {

      windows_options {
        computer_name  = var.name
        admin_password = var.admin_password
      }

      # NIC 1
      network_interface {
        ipv4_address    = var.nic_one_ip_address
        ipv4_netmask    = var.nic_one_ip_mask
        dns_server_list = var.dns_servers
        dns_domain      = var.domain_name
      }

      # NIC 2
      network_interface {
        ipv4_address    = var.nic_two_ip_address
        ipv4_netmask    = var.nic_two_ip_mask
        dns_server_list = var.dns_servers
        dns_domain      = var.domain_name
      }

      ipv4_gateway = var.local_gateway_ip
    }
  }
}

Debug Output

https://gist.github.com/wsf11/7950094d1ec67e91c61d62bab0ca2caf

Completes normally, no errors

Expected Behavior

I expect that two NICs are created, with two separate IP addresses, in the same subnet, both having the default gateway attached to the NIC.

Actual Behavior

Both NICs are created, but only one NIC has the default gateway attached. The gateway appears to be randomly assigned to one NIC or the other. Therefore, I cannot reach the VM through the NIC that does not have the gateway attached.

I remote into the machine and run ipconfig to see that only one NIC, the Ethernet0, has the default gateway IP address correct. It is empty for the other NIC, Ethernet 1

Steps to Reproduce

terraform init & apply

Important Factoids

I have reproduced this in the vSphere client using VM Customization Specifications and it worked fine. The template used was created using two NICs. I do customize the NICs and change the types as part of the deployment. This is a Windows 2019 Image. Also failing for Windows 2016

The terraform state file reports the default_ip_address to be the IP address that doesn't have the gateway attached. I read that the default_ip_address is the first IP address that the gateway can communicate with? Is something being overwritten?

I put the DNS Server IP addresses into the NIC, but do not domain join yet as I domain join later. The IP addresses being used live inside a subnet and are available.

References

• #0000

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[tenthirtyam]

I tested the issue today and can confirm that this is an issue when >1 network_interface is on the same r/network

The issue appears to be within this function:

terraform-provider-vsphere/vsphere/internal/vmworkflow/virtual_machine_customize_subresource.go

Lines 458 to 476 in 82c743a

	// expandSliceOfCustomizationAdapterMapping reads certain ResourceData keys and
	// returns a CustomizationAdapterMapping slice.
	func expandSliceOfCustomizationAdapterMapping(d *schema.ResourceData) []types.CustomizationAdapterMapping {
	s := d.Get(cKeyPrefix + "." + "network_interface").([]interface{})
	if len(s) < 1 {
	return nil
	}
	result := make([]types.CustomizationAdapterMapping, len(s))
	var v4gwFound, v6gwFound bool
	for i := range s {
	var adapter types.CustomizationIPSettings
	adapter, v4gwFound, v6gwFound = expandCustomizationIPSettings(d, i, !v4gwFound, !v6gwFound)
	obj := types.CustomizationAdapterMapping{
	Adapter: adapter,
	}
	result[i] = obj
	}
	return result
	}

It seems that it's not detecting that >1 network_interface are attached to the same network and proceeds to ignore all but the first.

I've updated the issue description and will mark it as confirmed.

Ryan Johnson
Staff II Solutions Architect | VMware, Inc.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.